AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks

By Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, Jian Tang. 2018

1,089 citations recorded by the source corpus.

Click-through rate (CTR) prediction, which aims to predict the probability of a user clicking on an ad or an item, is critical to many online applications such as online advertising and recommender systems. The problem is very challenging since (1) the input features (e.g., the user id, user age, item id, item category) are usually sparse and high-dimensional, and (2) an effective prediction relies on high-order combinatorial features (a.k.a. cross features), which are very time-consuming to hand-craft by domain experts and are impossible to be enumerated. Therefore, there have been efforts in finding low-dimensional representations of the sparse and high-dimensional raw features and their meaningful combinations. In this paper, we propose an effective and efficient method called the AutoInt to automatically learn the high-order feature interactions of input features. Our proposed algorithm is very general, which can be applied to both numerical and categorical input features. Specifically, we map both the numerical and categorical features into the same low-dimensional space. Afterwards, a multi-head self-attentive neural network with residual connections is proposed to explicitly model the feature interactions in the low-dimensional space. With different layers of the multi-head self-attentive neural networks, different orders of feature combinations of input features can be modeled. The whole model can be efficiently fit on large-scale raw data in an end-to-end fashion. Experimental results on four real-world datasets show that our proposed approach not only outperforms existing state-of-the-art approaches for prediction but also offers good explainability. Code is available at: \urlhttps://github.com/DeepGraphLearning/RecommenderSystems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Others Time series forecasting [248], [249], [250], Link prediction [251], Recommendation system [77], [252], [253], [254] Graph classification [121], Document classification [156], [255], [256], [257], Stereo confidence estimation [258]”
Revisiting Deep Learning Models for Tabular Data (2021, methodology): “The results averaged over 15 runs are reported in Table 5 and demonstrate both the superiority of the Transformer’s backbone to that of AutoInt and the necessity of feature biases.”
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, methodology): “Multiple studies [6, 13, 18, 20, 22, 37, 39] have shown that a better design of the interaction part can generate significant lift on the prediction accuracy, which motivates a variety of designs from different perspectives.”
Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, methodology): “We compare our proposed approach with the following baseline methods: (1) PopRec ranks items according to the popularity measured by the number of interactions; (2) FM [41] characterizes the pairwise interactions between variables using factorized model; (3) AutoInt [46] utilizes the multihead self-”
Hopular: Modern Hopfield Networks for Tabular Data (2022, methodology): “A multi-head self-attentive neural network for modeling feature-feature interactions was also used in AutoInt (Song et al., 2019).”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “Tabular data is the most common data type in many realworld applications such as recommender systems (Cheng et al. 2016), online advertising (Song et al. 2019), and portfolio optimization (Ban, El Karoui, and Lim 2018).”
S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization (2020, methodology): “alized method that ranks items according to popularity measured by the number of interactions. (2) FM[20] characterizes the pairwise interactions between variables using factorized model. (3) AutoInt [22] utilizes the multi-head self-attentive neural network to learn the feature interaction. (4) GRU”
RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms (2020, background): “hou et al. (2018) DSSM Huang et al. (2013) FM Rendle (2010) DeepFM Guo et al. (2017) xDeepFM Lian et al. (2018) Wide&Deep Cheng et al. (2016) NFM He and Chua (2017) AFM Xiao et al. (2017) AutoInt Song et al. (2019) DCN Wang et al. (2017) FNN(DNN) Zhang et al. (2016b) PNN Qu et al. (2016) FFM Jua”
A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, methodology): “These feature embeddings are directly concatenated and fed into DNN [28, 57] or specifically designed models [66, 148, 156] in an unstructured way to estimate the ranking score.”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “Unlike vision tasks that rely heavily on spatial priors such as translation invariance, tabular learning requires modeling implicit and high-order interactions among heterogeneous features (Song et al., 2019; Xie et al., 2021).”