Deep Learning over Multi-field Categorical Data - - A Case Study on User Response Prediction

By Weinan Zhang, Tianming Du, Jun Wang. European Conference on Information Retrieval, 2016

540 citations recorded by the source corpus.

Predicting user responses, such as click-through rate and conversion rate, are critical in many web applications including web search, personalised recommendation, and online advertising. Different from continuous raw features that we usually found in the image and audio domains, the input features in web space are always of multi-field and are mostly discrete and categorical while their dependencies are little known. Major user response prediction models have to either limit themselves to linear models or require manually building up high-order combination features. The former loses the ability of exploring feature interactions, while the latter results in a heavy computation in the large feature space. To tackle the issue, we propose two novel models using deep neural networks (DNNs) to automatically learn effective patterns from categorical feature interactions and make predictions of users' ad clicks. To get our DNNs efficiently work, we propose to leverage three feature transformation methods, i.e., factorisation machines (FMs), restricted Boltzmann machines (RBMs) and denoising auto-encoders (DAEs). This paper presents the structure of our models and their efficient training algorithms. The large-scale experiments with real-world data demonstrate that our methods work better than major state-of-the-art models.

Subfields: Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/1bc072002d97808340b312b69427baf2dc9fcb8e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Neural information retrieval: at the end of the early years (2017, methodology): “…(2016) 9.4.1 Content-based recommendation Aggregate Explicit Manotumruksa et al. (2016) 9.4.2 Learn Learn to match Gao et al. (2014) 10 Non-textual/Behavioural Borisov et al. (2016b), Zhang et al. (2016a) Documents are ranked by the cosine similarity of their embedding vector to the query vector.”
AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks (2018, methodology): “Similarly, PNN [22], FNN [37], DeepCrossing [29], Wide&Deep [7] and DeepFM [10] utilized feed-forward neural networks to model high-order feature interactions.”
Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting (2016, methodology): “RTB Glossary A summary of the terminology used in the monograph, which is also presented in [Zhang, 2016].”
Deep learning in business analytics and operations research: Models, applications and managerial implications (2018, unknown): “In addition, our experiments build upon common techniques for preprocessing categorical variables from prior literature, namely a factorization machine (Zhang et al., 2016) and deep neural networks with näıve embedding layers (Jia et al.”
Product-Based Neural Networks for User Response Prediction (2016, methodology): “Factorization-machine supported neural networks (FNN) was proposed in [12] to learn embedding vectors of categorical data via pre-trained FM.”
A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation (2021, methodology): “MLP based higher order Design better initialization techniques to facilitate MLP modeling NFM [57], FNN [58], PNN [59], DeepCrossing [60], [61]”
Neural Factorization Machines for Sparse Predictive Analytics (2017, methodology): “Inspired by FNN [44], we further explore the use of feature embeddings learned by FM to initialize DNNs, which can be seen as a pre-training step.”
AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction (2020, methodology): “FNN [37] uses FM to pre-train feature embeddings and then feeds them into an MLP. Wide & Deep Learning [5] jointly trains a wide model for artificial features and a deep model for raw features.”
A review on deep learning for recommender systems: challenges and remedies (2018, background): “…et al. (2017) Unger et al. (2016) Ying et al. (2016) Zuo et al. (2016) Wei et al. (2017) Ko et al. (2016) Ouyang et al. (2014) Wang et al. (2015a) Zhang et al. (2016a) Hu et al. (2014) Huang et al. (2015) Paradarami et al. (2017) Zheng et al. (2016a) Kyo-Joong et al. (2014) Yang et al. (2017) Du…”
FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction (2019, methodology): “If we remove the SENET layer and Bilinear-Interaction layer, our shallow FiBiNET and deep FiBiNET will downgrade to FM and FNN.”
Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction (2020, background): “with some analysis in existing works [35]: diamond networks are preferable to the increasing/decreasing width networks (triangular networks) and the constant width networks (rectangular networks).”
Dynamic Attention Deep Model for Article Recommendation by Learning Human Editors' Demonstration (2017, background): “(ii) The meta-data is another issue since its sparse property and categorical data type [35, 52].”
xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems (2018, methodology): “[44] proposes a Factorisation-machine supported Neural Network (FNN) to learn high-order feature interactions.”
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, methodology): “This has motivated a variety of studies that evolves from capturing low-order interactions towards high-order ones [7, 11, 13, 18–20, 22, 23, 29, 30, 34, 40, 43, 43, 48, 51].”
AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction (2020, background): “In the meantime, Deep Neural Network (DNN) models [6, 30] are proposed.”