DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction
By Buyun Zhang, Liangchen Luo, Xi Liu, Jay Li, Zeliang Chen, Weilin Zhang, Xiaohan Wei, Yuchen Hao, Michael Tsang, Wen-Jia Wang and 7 more. 2022
17 citations recorded by the source corpus.
Learning feature interactions is important to the model performance of online advertising services. As a result, extensive efforts have been devoted to designing effective architectures to learn feature interactions. However, we observe that the practical performance of those designs can vary from dataset to dataset, even when the order of interactions claimed to be captured is the same. That indicates different designs may have different advantages and the interactions captured by them have non-overlapping information. Motivated by this observation, we propose DHEN - a deep and hierarchical ensemble architecture that can leverage strengths of heterogeneous interaction modules and learn a hierarchy of the interactions under different orders. To overcome the challenge brought by DHEN's deeper and multi-layer structure in training, we propose a novel co-designed training system that can further improve the training efficiency of DHEN. Experiments of DHEN on large-scale dataset from CTR prediction tasks attained 0.27\% improvement on the Normalized Entropy (NE) of prediction and 1.2x better training throughput than state-of-the-art baseline, demonstrating their effectiveness in practice.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sam Shleifer
- Chien-chin Huang
- Min Xu
- Alban Desmaison
- Myle Ott
- Liangchen Luo
- Luke S. Zettlemoyer
- Michael Auli
In Information Systems
In Natural Language Processing
- Sam Shleifer
- Chien-chin Huang
- Myle Ott
- Liangchen Luo
- Michael Auli
- Luke S. Zettlemoyer
- Graham Neubig
- Alexei Baevski
Citations in the corpus
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, methodology): “In these experiments, we conducted evaluations on the HuggingFace T5-11B transformer [25], minGPT-175B transformer [2], and DHEN recommendation model [32].”