Towards Universal Sequence Representation Learning for Recommender Systems

By Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, Ji-rong Wen. Knowledge Discovery and Data Mining, 2022

355 citations recorded by the source corpus.

In order to develop effective sequential recommenders, a series of sequence representation learning (SRL) methods are proposed to model historical user behaviors. Most existing SRL methods rely on explicit item IDs for developing the sequence models to better capture user preference. Though effective to some extent, these methods are difficult to be transferred to new recommendation scenarios, due to the limitation by explicitly modeling item IDs. To tackle this issue, we present a novel universal sequence representation learning approach, named UniSRec. The proposed approach utilizes the associated description text of items to learn transferable representations across different recommendation scenarios. For learning universal item representations, we design a lightweight item encoding architecture based on parametric whitening and mixture-of-experts enhanced adaptor. For learning universal sequence representations, we introduce two contrastive pre-training tasks by sampling multi-domain negatives. With the pre-trained universal sequence representation model, our approach can be effectively transferred to new recommendation domains or platforms in a parameter-efficient way, under either inductive or transductive settings. Extensive experiments conducted on real-world datasets demonstrate the effectiveness of the proposed approach. Especially, our approach also leads to a performance improvement in a cross-platform setting, showing the strong transferability of the proposed universal SRL method. The code and pre-trained model are available at: https://github.com/RUCAIBox/UniSRec.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/e7f3f2b77994c2cabad612ba0881e63763ec2dad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wayne Xin Zhao
2. Xiangnan He
3. Ji-rong Wen
4. Yupeng Hou
5. Fuli Feng
6. Hongyu Lu
7. Fajie Yuan
8. Jiacheng Li

In Information Systems

1. Wayne Xin Zhao
2. Xiangnan He
3. Yupeng Hou
4. Ji-rong Wen
5. Fuli Feng
6. Hongyu Lu
7. Jiacheng Li
8. Fajie Yuan

Citations in the corpus

• Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “For instance, Unisec [65] is one such approach that takes advantage of item descriptions to learn transferable representations from various recommendation scenarios.”
• Large Language Models are Zero-Shot Rankers for Recommender Systems (2023, methodology): “They aim to explicitly capture user preference in natural language [10] or transfer rich world knowledge from text corpora [18, 16].”
• Self-Supervised Learning for Recommender Systems: A Survey (2022, neutral): “Althoughtherehavebeeneffortstodevelopgeneral-purposerec-ommendationmodels[54],[114],[147],[148],theyaremostly trainedinaBERT-likefashionwithsimilararchitectures.”
• Uncovering ChatGPT’s Capabilities in Recommender Systems (2023, methodology): “Existing works can be categorized into two types: (i) utilizing LMs training strategies to reformulate and model recommendation tasks, such as BERT4Rec (masked language modeling) [36], UnisRec (pre-train and finetune)[13], P5 (pre-train and prompting) [9] and (ii) using LMs to obtain better represen”
• Text Is All You Need: Learning Language Representations for Sequential Recommendation (2023, methodology): “Unlike previous sequential recommenders [12, 37] using both text and item IDs, in this study, we use only text for the sequential recommendation.”
• Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach (2023, positive): “Following previous work [18], we filter unpopular users and items with fewer than five interactions for all datasets.”
• Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited (2023, background): “In the past two years, some works have begun to explore E2E-based MoRec, however, most of them focus on text recommendation [24, 55, 67, 72, 73, 75].”
• Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, methodology): “Furthermore, the inherent natural language characteristics of the item title and description have motivated numerous researchers to explore the utilization of pre-trained language models (PLMs) in recommender systems [12]–[15].”
• TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation (2023, background): “Furthermore, there have been attempts to utilize LMs to encode the natural language information of users/items, such as reviews, and incorporate them as part of the embedding of users/items [17].”