Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks

By Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, Tat-seng Chua. International Joint Conference on Artificial Intelligence, 2017

1,012 citations recorded by the source corpus.

Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Despite effectiveness, FM can be hindered by its modelling of all feature interactions with the same weight, as not all feature interactions are equally useful and predictive. For example, the interactions with useless features may even introduce noises and adversely degrade the performance. In this work, we improve FM by discriminating the importance of different feature interactions. We propose a novel model named Attentional Factorization Machine (AFM), which learns the importance of each feature interaction from data via a neural attention network. Extensive experiments on two real-world datasets demonstrate the effectiveness of AFM. Empirically, it is shown on regression task AFM betters FM with a $8.6\%$ relative improvement, and consistently outperforms the state-of-the-art deep learning methods Wide&Deep and DeepCross with a much simpler structure and fewer model parameters. Our implementation of AFM is publicly available at: this https URL

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/f0af0029293dc8f242894f113baf15d68228ec4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Self-Attentive Sequential Recommendation (2018, background): “For example, Attentional Factorization Machines (AFM) [30] learn the importance of each feature interaction for content-aware recommendation.”
Neural Attentional Rating Regression with Review-level Explanations (2018, methodology): “Attention mechanism has been widely used in many tasks, such as information retrieval[43], recommendation[4, 42], computer vision[5], and machine translation[8].”
AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “We use a to denote the vector of weighted scores of interactions and embeddings computed by cross network [70], compressed interaction network (CIN) [39], or attention module [41, 75].”
Deep Learning Based Recommender System (2017, background): “Currently, attentional models [127, 147, 179] have more or less eased the noninterpretable concerns of neural models.”
Cross-domain Recommendation Without Sharing User-relevant Data (2019, positive): “…we apply a non-uniform coefficient when fusing the transferred item embeddings: To model the various item importance in a user-sensitive fashion, here we introduce attention mechanism, which has achieved great success in recommendation tasks [3, 13, 40] and natural language processing [20, 37].”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “For deep models designed specifically for tabular data, there are deep versions of factorization machines (Guo et al. 2018; Xiao et al. 2017), Transformers-based methods (Song et al.”
FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction (2019, methodology): “The shallow baseline models include LR(logistic regression)[17], FM[19, 20], FFM[9, 10], AFM[23], and the deep baseline models include FNN[25], DCN[22], DeepFM[4], XDeepFM[15].”
A3T-GCN: Attention Temporal Graph Convolutional Network for Traffic Forecasting (2020, background): “Nowadays, attention models are widely applied in image caption generation [33], recommendation system [32], and document classiﬁcation [24].”
Sequential Recommender System based on Hierarchical Attention Networks (2018, methodology): “It has been shown that nonlinear models can better model the user-item interaction in user activities [He and Chua, 2017; Xiao et al. , 2017; Cheng et al. , 2016].”
AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction (2020, methodology): “Note that HOFM [2] is not compared with because it is shown to be inferior to AFM in [34].”
xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems (2018, background): “ Kingdom J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, and G. Sun major downside is that, [2] proposes to model all feature interactions, including both useful and useless combinations. As revealed in [43], the interactions with useless features may introduce noises and degrade the performance. In re”
NAIS: Neural Attentive Item Similarity Model for Recommendation (2018, methodology): “cess of using neural networks to model the attention weight [15], [16], we similarly use a Multi-Layer Perception (MLP) to parameterize the attention function f .”
Time Interval Aware Self-Attention for Sequential Recommendation (2020, background): “Recently, attention mechanisms have been incorporated into recommender systems [5, 28, 30].”
TEM: Tree-enhanced Embedding Model for Explainable Recommendation (2018, methodology): “Inspired by the recent success [4, 9, 46] that uses multi-layer perceptrons (MLPs) to learn the attentive weights, we similarly use a MLP to parameterizewuil .”
Multi-Pointer Co-Attention Networks for Recommendation (2018, background): “, 23] were popular standard machine learning baselines for interaction-based recommendation. Today, we see a shift into deep learning, in which neural models are claiming state-of-the-art performance [10, 27, 36, 38]. He et al. [10] proposed a neural framework that combines a generalized matrix fact”
AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks (2018, background): “GBFM [6] and AFM [36] considered the importance of different second-order feature interactions.”