CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery

By Kiwan Maeng, Shivam Bharuka, Isabel Gao, Isabel Gao, M. C. Jeffrey, Vikram Saraph, Bor-Yiing Su, Caroline Trippel, Jiyan Yang, Michael G. Rabbat and 2 more. 2020

47 citations recorded by the source corpus.

The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node fails during training, improving failure-related overheads. The paper is the first to the extent of our knowledge to perform a data-driven, in-depth analysis of applying partial recovery to recommendation models and identified a trade-off between accuracy and performance. Motivated by the analysis, we present CPR, a partial recovery training system that can reduce the training time and maintain the desired level of model accuracy by (1) estimating the benefit of partial recovery, (2) selecting an appropriate checkpoint saving interval, and (3) prioritizing to save updates of more frequently accessed parameters. Two variants of CPR, CPR-MFU and CPR-SSU, reduce the checkpoint-related overhead from 8.2-8.5% to 0.53-0.68% compared to full recovery, on a configuration emulating the failure pattern and overhead of a production-scale cluster. While reducing overhead significantly, CPR achieves model quality on par with the more expensive full recovery scheme, training the state-of-the-art recommendation model using Criteo's Ads CTR dataset. Our preliminary results also suggest that CPR can speed up training on a real production-scale cluster, without notably degrading the accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Disaggregation with well-designed check-pointing support [110], [111] improves training fault tolerance as well.”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “In addition, other prior work proposes to prioritize frequently-accessed embedding rows for model parameter checkpointing [28], in order to improve failure tolerance of DLRM training.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “Finally, making training infrastructures reliable has a profound impact in the training workflow efficiency as well [37], [44], [46].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “To understand and improve fault tolerance training for deep learning recommendations with partial recovery, Maeng et al. [335] optimized CPR, a partial recovery training system for a recommendation model.”