Neural Factorization Machines for Sparse Predictive Analytics

By Xiangnan He, Tat-seng Chua. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017

1,428 citations recorded by the source corpus.

Many predictive tasks of web applications need to model categorical variables, such as user IDs and demographics like genders and occupations. To apply standard machine learning techniques, these categorical predictors are always converted to a set of binary features via one-hot encoding, making the resultant feature vector highly sparse. To learn from such sparse data effectively, it is crucial to account for the interactions between features. Factorization Machines (FMs) are a popular solution for efficiently using the second-order feature interactions. However, FM models feature interactions in a linear way, which can be insufficient for capturing the non-linear and complex inherent structure of real-world data. While deep neural networks have recently been applied to learn non-linear feature interactions in industry, such as the Wide&Deep by Google and DeepCross by Microsoft, the deep structure meanwhile makes them difficult to train. In this paper, we propose a novel model Neural Factorization Machine (NFM) for prediction under sparse settings. NFM seamlessly combines the linearity of FM in modelling second-order feature interactions and the non-linearity of neural network in modelling higher-order feature interactions. Conceptually, NFM is more expressive than FM since FM can be seen as a special case of NFM without hidden layers. Empirical results on two regression tasks show that with one hidden layer only, NFM significantly outperforms FM with a 7.3% relative improvement. Compared to the recent deep learning methods Wide&Deep and DeepCross, our NFM uses a shallower structure but offers better performance, being much easier to train and tune in practice.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/3d1f9a530e710fdd4e2313bda4c8a1f574e60ab6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Yongfeng Zhang
3. Tat-seng Chua
4. Xiang Wang
5. Xu Chen
6. Xing Xie
7. Wayne Xin Zhao
8. Liqiang Nie

In Information Systems

1. Xiangnan He
2. Yongfeng Zhang
3. Xiang Wang
4. Tat-seng Chua
5. Xu Chen
6. Xing Xie
7. Meng Wang
8. Fuli Feng

Citations in the corpus

• Deep Learning for Matching in Search and Recommendation (2018, methodology): “In recommendation, FNNs like Factorization Machine (FM) and Multi-Layer Perceptron (MLP) have been used as the matching function to learn second-order and higher-order feature interactions [3, 8, 9].”
• Advances and Challenges in Conversational Recommender Systems: A Survey (2021, neutral): “Recently, more complicated but powerful neural networks have been developed, including Wide&Deep [28], neural collaborative filtering (NCF) [58], deep interest network (DIN) [218], tree-based deep model (TDM) [227], and graph convolutional networks [199, 59].”
• Personalized Fashion Recommendation with Visual Explanations based on Multimodal Attention Network: Towards Visually Explainable Recommendation (2019, background): “• Encouragingly, we find that our VECF model was better than NFM++ across different datasets in most cases.”
• DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, background): “Similarly like PNN and FNN, NFM stacks deep neural networks (DNNs) on top of the second-order feature interactions to model higher-order features.”
• An Adversarial Feature Distillation Method for Audio Classification (2019, background): “Generative adversarial networks (GAN) [28]–[30] produces fairly good output throughmutual confrontation learning of two modules in the framework: the generative model and the discriminative model.”
• DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020, background): “Neural FM (NFM) [16] extends FM by replacing the inner-product with a Hadamard product; DLRM [34] follows FM to compute the feature crosses through inner products; These models can only create up to 2nd-order explicit crosses.”
• Multi-modal Knowledge Graphs for Recommender Systems (2020, methodology): “• NFM: Neural Factorization Machines (NFM) [7]is a state-ofthe-art Factorization Machines (FM), which subsumes FM under neural network.”
• TEM: Tree-enhanced Embedding Model for Explainable Recommendation (2018, methodology): “Specially, we employed one hidden layers for NFM as suggested in [20].”
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems (2018, background): “res, researchers apply DNNs on field embedding, thus patterns from categorical feature interactions can be learned automatically. Representative models include FNN [46], PNN [31], DeepCross [37], NFM [12], DCN [40], Wide&Deep [5], and DeepFM [9]. These models are highly related to our proposed x”
• Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation (2020, result): “For cold-start user recommendation, we compare it with two powerful baseline NeuFM [12] and DeepFM [9].”
• One Person, One Model, One World: Learning Continual User Representation without Forgetting (2020, background): “Among them, deep structured semantic models (DSSM) [14], neural networkbased collaborative filtering (NCF) [30], deep or neural factorization machines (DeepFM/NFM) [7, 10] are some of the representative works based on supervised representation learning.”
• Neural Attentive Session-based Recommendation (2017, background): “Deep models can be trained to learn discriminative representations from unstructured data [9, 11, 19].”
• Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising (2018, background): “Recently, deep learning for CTR prediction also attracted much research attention [5, 9, 10, 17, 22, 24, 26].”
• Controllable Multi-Interest Framework for Recommendation (2020, background): “NFM [19] seamlessly combines the linearity of FMs in modeling second-order feature interactions and the non-linearity of neural networks in modeling higher-order feature interactions.”
• Latent Cross: Making Use of Context in Recurrent Recommender Systems (2018, background): “Cross RNN [22, 34] [29, 32, 48] [17, 37, 46] [21, 43] [11] [20] [19, 33] [41] [49]”
• Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences (2019, methodology): “With the surge of neural network (NN) models, a host of methods extend similarity-based methods with NN and present a more effective mechanism in automatically extracting latent features of users and items for recommendation [7, 12–14].”
• Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation (2019, background): “3371790 have attracted most of the attention, in which users and items are represented by latent vectors, such as the matrix factorization (MF) models [16], factorization machines (FM) [27], and their recent neural extensions [5, 9, 10].”
• Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems (2020, background): “Neural FM [16] and DeepFM [15] have improved FM’s representation ability with deep neural networks.”
• AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks (2018, methodology): “Recently, many works [7, 10, 12, 34] based on deep neural networks have been proposed to model the high-order feature interactions.”