Examining Temporality in Document Classification

2018

44 citations recorded by the source corpus.

Many corpora span broad periods of time. Language processing models trained during one time period may not work well in future time periods, and the best model may depend on specific times of year (e.g., people might describe hotels differently in reviews during the winter versus the summer). This study investigates how document classifiers trained on documents from certain time intervals perform on documents from other time intervals, considering both seasonal intervals (intervals that repeat across years, e.g., winter) and non-seasonal intervals (e.g., specific years). We show experimentally that classification performance varies over time, and that performance can be improved by using a standard domain adaptation approach to adjust for changes in time.

Subfields: Artificial Intelligence, Sociology and Political Science

Citations in the corpus

• Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Several studies have focused on degradation of models on test data from a different time period than their training data (Huang and Paul, 2018, 2019; Jaidka et al., 2018; Lukes and Søgaard, 2018; Florio et al., 2020).”
• Temporal Effects on Pre-trained Models for Language Processing Tasks (2021, background): “The results are consistent with prior work that uses non-neural models (Fromreide et al., 2014; Lukes and Søgaard, 2018; Huang and Paul, 2018) or evaluates on a single test set (Lukes and Søgaard, 2018; Rijhwani and Preotiuc-Pietro, 2020; Søgaard et al.”
• SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “Recent work studying other NLP tasks have shown that this static paradigm misrepresents how models perform in practical settings (Osborne et al., 2014; Wang et al., 2008; Huang and Paul, 2018; Rijhwani and Preotiuc-Pietro, 2020).”
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, methodology): “We consider another practical fine-tuning scenario in the tweet stream where the model is trained on outdated data and evaluated on the latest data (Rijhwani and Preotiuc-Pietro, 2020; Huang and Paul, 2018), referred to as the temporal generalization ability.”
• Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models (2019, methodology): “We previously showed that out-of-the-box domain adaptation techniques can make n-gram classifiers more robust to temporal shifts (Huang and Paul, 2018).”
• RealTime QA: What's the Answer Right Now? (2022, background): “(Karpukhin et al., 2020); RAG: retrieval-augmented generation (Lewis et al., 2020b).”