BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

By Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang. International Conference on Information and Knowledge Management, 2019

3,307 citations recorded by the source corpus.

Modeling users' dynamic preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks to encode users' historical interactions from left to right into hidden representations for making recommendations. Despite their effectiveness, we argue that such left-to-right unidirectional models are sub-optimal due to the limitations including: \begin enumerate* [label=series\itshape\alph*\upshape)] \item unidirectional architectures restrict the power of hidden representation in users' behavior sequences; \item they often assume a rigidly ordered sequence which is not always practical. \end enumerate* To address these limitations, we proposed a sequential recommendation model called BERT4Rec, which employs the deep bidirectional self-attention to model user behavior sequences. To avoid the information leakage and efficiently train the bidirectional model, we adopt the Cloze objective to sequential recommendation, predicting the random masked items in the sequence by jointly conditioning on their left and right context. In this way, we learn a bidirectional representation model to make recommendations by allowing each item in user historical behaviors to fuse information from both left and right sides. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/690edf44e8739fd80bdfb76f40c9a4a222f3bba8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Wayne Xin Zhao
3. Ji-rong Wen
4. Yongfeng Zhang
5. Yi Tay
6. Jiliang Tang
7. Hongzhi Yin
8. Huan Liu

In Information Systems

1. Xiangnan He
2. Wayne Xin Zhao
3. Hongzhi Yin
4. Yongfeng Zhang
5. Ji-rong Wen
6. Yupeng Hou
7. Tong Chen
8. Jun Xu

Citations in the corpus

• Text Is All You Need: Learning Language Representations for Sequential Recommendation (2023, methodology): “• BERT4Rec [27]employsabi-directionalself-attentivemodel withtheclozeobjectiveformodelinguserbehaviorsequences.”
• Is ChatGPT a Good Recommender? A Preliminary Study (2023, methodology): “For sequential recommendation, we adopt Caser [51], HGN [35], GRU4Rec [21], BERT4Rec [49], FDSA [64], SASRec [24] and S 3 -Rec [68] as baselines for comparison.”
• Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, methodology): “BERT4Rec [60] mimics the BERT-style masked language modeling and learns a bidirectional representation for sequential recommendation.”
• AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “, ATRank [82], SASRec [32], and BERT4Rec [63] introduce self-attention layers and point-wise feedforward layers to capture long-term semantics, which are the same as the stack of multi-head attention layers and 1× 1 convolutions in the choice blocks.”
• Next-item Recommendation with Sequential Hypergraphs (2020, methodology): “In the experiments, we formulate the next-item recommendation problem under leave-one-out setting as in previous works [18, 31] and split the train-test data following the real-world scenario as in [35, 39].”
• Improving Transformer Optimization Through Better Initialization (2020, background): “…et al., 2017; Edunov et al., 2018), language modeling (Devlin et al., 2019; Yang et al., 2019; Lan et al., 2020; Beltagy et al., 2020), video cap-tioning and summarization (Bilkhu et al., 2019; Fan et al., 2018; Chen et al., 2018a) and recommender systems (Chen et al., 2019; Sun et al., 2019).”
• Transformer Network for Remaining Useful Life Prediction of Lithium-Ion Batteries (2022, neutral): “Transformers, which perform well in encoding text, have been explored for various applications, such as recommendation systems [54], [55] and CV [56], [57].”
• Contrastive Learning for Sequential Recommendation (2022, methodology): “The framework is illustrated on the left side of Figure 1. encoder [42] to encode the user representation, which has shown promising results in recent works [23, 40].”
• Graph Neural Networks in Recommender Systems: A Survey (2020, background): “When the timestamps of the user’s historical behavior are known or the historical behavior is organized in chronological order, the user representations can be enhanced via exploring the sequential patterns in her/his historical interactions [52, 60, 69, 84, 96, 118, 130, 135, 149].”
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation (2021, methodology): “Besides, attention mechanism is widely used in recent models such as SASRec [11] and BERT4Rec [23].”
• Self-Supervised Learning for Recommender Systems: A Survey (2022, methodology): “The recommendation community soon embraced SSL and started to work on pre-training recommendation models with the Clozelike tasks on sequential data [24], [25], [26].”
• Deep Learning for Matching in Search and Recommendation (2018, methodology): “To handle user behavior sequence data, self-attention and BERT are also adopted, which demonstrates promising results in sequential recommendation (Sun et al., 2019; Yuan et al., 2020).”
• Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction (2020, background): “Recently, a series of works, which we called User Interest Model, focus on learning the representation of latent user interest from historical behaviors, using different neural network architecture such as CNN[14, 17], RNN[5, 19], Transformer[3, 13] and Capsule[6], etc.”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “…new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and files in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history [52], agent workflow with long horizon [61]).”
• S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization (2020, methodology): “(7) BERT4Rec [23] uses a Cloze objective loss for sequential recommendation by the bidirectional self-attention mechanism.”