Product-Based Neural Networks for User Response Prediction

By Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, Jun Wang. 2016

790 citations recorded by the source corpus.

Predicting user responses, such as clicks and conversions, is of great importance and has found its usage inmany Web applications including recommender systems, websearch and online advertising. The data in those applicationsis mostly categorical and contains multiple fields, a typicalrepresentation is to transform it into a high-dimensional sparsebinary feature representation via one-hot encoding. Facing withthe extreme sparsity, traditional models may limit their capacityof mining shallow patterns from the data, i.e. low-order featurecombinations. Deep models like deep neural networks, on theother hand, cannot be directly applied for the high-dimensionalinput because of the huge feature space. In this paper, we proposea Product-based Neural Networks (PNN) with an embeddinglayer to learn a distributed representation of the categorical data, a product layer to capture interactive patterns between interfieldcategories, and further fully connected layers to explorehigh-order feature interactions. Our experimental results on twolarge-scale real-world ad click datasets demonstrate that PNNsconsistently outperform the state-of-the-art models on various metrics.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Xing Xie
3. Xiang Wang
4. Xia Hu
5. Meng Wang
6. Le Wu
7. Qingquan Song
8. Tat-seng Chua

In Information Systems

1. Xiangnan He
2. Xing Xie
3. Xiang Wang
4. Meng Wang
5. Le Wu
6. Kun Zhang
7. Tat-seng Chua
8. Fuzheng Zhang

Citations in the corpus

• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “Compared to hand-crafted features, well-learned embeddings have been demonstrated to significantly enhance the recommendation performance [5, 8, 21, 24, 38].”
• A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation (2021, methodology): “MLP based higher order Design better initialization techniques to facilitate MLP modeling NFM [57], FNN [58], PNN [59], DeepCrossing [60], [61]”
• AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “DeepFM [18], DCN [70] and xDeepFM [39] enumerate interactions of bounded degrees and combine the feature interactions with MLP via shortcut connection while IPNN [51] and FibiNet [31] directly apply interactions to MLP.”
• Deep Interest Evolution Network for Click-Through Rate Prediction (2018, methodology): “From Table 2, we can find Wide & Deep depends on the quality of manually designed features heavily and performs not well, while automative interaction between features ( PNN ) can improve the performance of BaseModel .”
• Latent Cross: Making Use of Context in Recurrent Recommender Systems (2018, background): “Cross RNN [22, 34] [29, 32, 48] [17, 37, 46] [21, 43] [11] [20] [19, 33] [41] [49]”
• AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks (2018, methodology): “Similarly, PNN [25], FNN [41], DeepCrossing [32], Wide&Deep [8] and DeepFM [11] utilized feed-forward neural networks to model high-order feature interactions.”
• How Can Recommender Systems Benefit from Large Language Models: A Survey (2023, positive): “…the permission of LLM, and proposes a roadmap for building an integrated conversational recommender system, where LLM is able to manage the dialogue, understand user preference, arrange the ranking stage, and even provide a controllable LLM-based user simulator to generate synthetic conversations.”
• Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction (2020, methodology): “ore than seven blocks, the space would still contain over 1011 distinct architectures. Moreover, it could cover multiple representative human-crafted architectures such as deepFM [9], DLRM [21], IPNN [23], and Wide&Deep [7]. Figure 2: An illustration of the AutoCTR search loop 4 MULTI-OBJECTIVE ”
• Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction (2020, background): “In early age, most pioneer works[1, 4, 7, 9, 15] use a deep neural network to capture interactions between features from different fields so that engineers could get rid of boring feature engineering works.”
• DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, background): “Similarly like PNN and FNN, NFM stacks deep neural networks (DNNs) on top of the second-order feature interactions to model higher-order features.”
• Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems (2020, background): “, 2018) design special fully-connected layers for corresponding tasks to capture the latent interactions among features; Product-based Neural Network (PNN) (Qu et al., 2016) employs a product layer to capture high-order feature interactions; DeepFM (Guo et al.”
• Learning Tree-based Deep Model for Recommender Systems (2018, background): “thatuserinterestsarediverse,andanattentionlikenetworkstructurecangenerate varying user vectors according to different candidate items. Beyond the above works, other methods like product neural network [27] have also proven the effectiveness of advanced neural networks. However, as these kinds of model”
• AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction (2020, methodology): “The revenue of an online advertising system depends critically on the performance of CTR prediction [9, 26] because candidate advertisements are ranked by CTR × bid, where bid is the price to pay once the advertisement is clicked [9, 39, 40].”
• Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising (2018, methodology): “Recently, deep learning for CTR prediction also attracted much research attention [5, 9, 10, 17, 22, 24, 26].”
• DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020, background): “Many recent works [1, 6, 13, 26, 34, 35, 46, 50] tried to tackle this challenge.”