Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
By Ruihong Qiu, Zi-Liang Huang, Hongzhi Yin, Zijian Wang. Web Search and Data Mining, 2021
572 citations recorded by the source corpus.
Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.
Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research
View Paper: https://www.semanticscholar.org/paper/06e9bd1b8bb3f9274e52af5b1c89fe5f4f07bfc1
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hongzhi Yin
- Tong Chen
- Xin Xia
- Junliang Yu
- Li-zhen Cui
- Zi-Liang Huang
- Jundong Li
- Nguyen Quoc Viet Hung
In Information Systems
- Hongzhi Yin
- Tong Chen
- Xin Xia
- Junliang Yu
- Li-zhen Cui
- Zi-Liang Huang
- Jundong Li
- Nguyen Quoc Viet Hung
In Management Science and Operations Research
Citations in the corpus
- Self-Supervised Learning for Recommender Systems: A Survey (2022, methodology): “DuoRec [73] applies different dropout masks to a Transformer-based backbone for two model-level representation augmentations, maximizing the mutual information be-tween the two representations.”
- Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation (2021, background): “, eu and ei aggregate information from each other in the graph convolution) and causes the representation degeneration [21].”
- XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation (2022, background): “Qiu et al. proposed DuoRec [61] which adopts a model-level augmentation by conducting dropout on the encoder.”