The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances

By Anthony J. Bagnall, Jason Lines, Aaron Bostrom, James Large, Eamonn J. Keogh. Data mining and knowledge discovery, 2016

1,466 citations recorded by the source corpus.

In the last 5 years there have been a large number of new time series classification algorithms proposed in the literature. These algorithms have been evaluated on subsets of the 47 data sets in the University of California, Riverside time series classification archive. The archive has recently been expanded to 85 data sets, over half of which have been donated by researchers at the University of East Anglia. Aspects of previous evaluations have made comparisons between algorithms difficult. For example, several different programming languages have been used, experiments involved a single train/test split and some used normalised data whilst others did not. The relaunch of the archive provides a timely opportunity to thoroughly evaluate algorithms on a larger number of datasets. We have implemented 18 recently proposed algorithms in a common Java framework and compared them against two standard benchmark classifiers (and each other) by performing 100 resampling experiments on each of the 85 datasets. We use these results to test several hypotheses relating to whether the algorithms are significantly more accurate than the benchmarks and each other. Our results indicate that only nine of these algorithms are significantly more accurate than both benchmarks and that one classifier, the collective of transformation ensembles, is significantly more accurate than all of the others. All of our experiments and results are reproducible: we release all of our code, results and experimental details and we hope these experiments form the basis for more robust testing of new algorithms in the future.

Subfields: Artificial Intelligence, Economics and Econometrics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/3b421dcebe21bf45c1930b948cfda7ec95b55ac4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Signal Processing

Citations in the corpus

Unsupervised Scalable Representation Learning for Multivariate Time Series (2019, methodology): “…based on neural networks have emerged recently (Cui et al., 2016; Wang et al., 2017) and claim to achieve close to state-of-the-art performance, but, to our knowledge, there was no extensive study similar to Bagnall et al. (2017) comparing them in similar settings and in a reproducible manner.”
Tslearn, A Machine Learning Toolkit for Time Series Data (2020, methodology): “It includes pre-processing routines, feature extractors, and machine learning models for classiﬁcation (Bagnall et al., 2017; Fawaz et al., 2019), regression and clustering (Aghabozorgi et al., 2015).”
Predicting mortgage default using convolutional neural networks (2018, background): “For a review of other state-of-the-art methods, see e.g. Bagnall et al. (2016).”
Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series (2018, positive): “The state-of-the-art approaches are now lead by more complex algorithms [62], that we describe hereafter.”
ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels (2019, methodology): “To this end, Bagnall et al. (2017) used resamples of the datasets to assess performance.”
The UEA multivariate time series classification archive, 2018 (2018, methodology): “It has worked: most recent research into TSC uses all 85 datasets to evaluate algorithmic advances [2].”
A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, methodology): “Regression and classification of time series: Currently, nondeep learning methods such as TS-CHIEF [29], HIVE-COTE [21], and ROCKET [8, 9] constitute the state of the art for time series regression and classification based on evaluations on public benchmarks [2, 30], followed by CNN-based deep archi”
TapNet: Multivariate Time Series Classification with Attentional Prototypical Network (2020, background): “Different from computer vision and nat-ural language processing domains, the availability of labeled data is limited in most of the time series datasets (Bagnall et al. 2018)(Bagnall et al. 2017).”
InceptionTime: Finding AlexNet for time series classification (2019, methodology): “5 illustrates the critical diﬀerence diagram with InceptionTime added to the mix of the current state-of-the-art classiﬁers for time series data, whose results were taken from Bagnall et al. (2017).”
MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification (2020, methodology): “Recent advances in accuracy have largely superseded the most accurate methods originally identified in [3].”
Deep learning for time series classification: a review (2018, methodology): “However, given the small sample size (only 7 datasets), we cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets (Bagnall et al., 2017).”
Mobile sensor data anonymization (2018, methodology): “17k -NN with DTW outperforms other methods in time series classification, except when considerable computation and implementation cost is acceptable for very small improvements [2].”
k-Nearest Neighbour Classifiers - A Tutorial (2020, background): “There is a long history of Machine Learning research on time-series analysis and 1 -NN is the base-line metric for time-series classiﬁcation [5].”
The UCR time series archive (2018, methodology): “We keep the data sets as is if they are from a published papers and are already in a UCR archive preferred format (see guidelines for donating data sets to the archive [46]).”