A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays

By Leonardo Ravaglia, Manuele Rusci, Davide Nadalini, Alessandro Capotondi, Francesco Conti, Luca Benini. 2021

94 citations recorded by the source corpus.

In the last few years, research and development on Deep Learning models & techniques for ultra-low-power devices– in a word, <italic>TinyML</italic> – has mainly focused on a <italic>train-then-deploy</italic> assumption, with static models that cannot be adapted to newly collected data without cloud-based data collection and fine-tuning. Latent Replay-based Continual Learning (CL) techniques (Pellegrini <italic>et al.</italic>, 2020) enable online, serverless adaptation in principle, but so far they have still been too computation- and memory-hungry for ultra-low-power TinyML devices, which are typically based on microcontrollers. In this work, we introduce a HW/SW platform for end-to-end CL based on a 10-core <italic>FP32</italic> -enabled parallel ultra-low-power (PULP) processor. We rethink the baseline Latent Replay CL algorithm, leveraging quantization of the frozen stage of the model and Latent Replays (LRs) to reduce their memory cost with minimal impact on accuracy. In particular, 8-bit compression of the LR memory proves to be almost lossless (−0.26% with 3000LR) compared to the full-precision baseline implementation, but requires <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> less memory, while 7-bit can also be used with an additional minimal accuracy degradation (up to 5%). We also introduce optimized primitives for forward and backward propagation on the PULP processor, together with data tiling strategies to fully exploit its memory hierarchy, while maximizing efficiency. Our results show that by combining these techniques, continual learning can be achieved in practice using less than 64MB of memory – an amount compatible with embedding in TinyML devices. On an advanced 22nm prototype of our platform, called <italic>VEGA</italic>, the proposed solution performs on average <inline-formula> <tex-math notation="LaTeX">$65 \times $ </tex-math></inline-formula> faster than a low-power STM32 L4 microcontroller, being <inline-formula> <tex-math notation="LaTeX">$37\times $ </tex-math></inline-formula> more energy efficient – enough for a lifetime of 535h when learning a new mini-batch of data once every minute.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “Also, iterative retraining to integrate new data is computationally costly and difficult in timeor computation-constrained applications [9, 32].”
Constrained Few-shot Class-incremental Learning (2022, background): “The storage and computational overhead of the rehearsal methods can be reduced by latent replay [34] and its TinyML evolution [38].”
Divide and not forget: Ensemble of selectively trained experts in Continual Learning (2024, background): “However, its application is limited, e.g., due to privacy concerns or in memory-constrained devices (Ravaglia et al., 2021).”