The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models

By Shiliang Zhang, Hui Jiang, Mingbin Xu, Junfeng Hou, Lirong Dai. 2015

85 citations recorded by the source corpus.

In this paper, we propose the new fixedsize ordinally-forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence of words into a fixed-size representation. FOFE can model the word order in a sequence using a simple ordinally-forgetting mechanism according to the positions of words. In this work, we have applied FOFE to feedforward neural network language models (FNN-LMs). Experimental results have shown that without using any recurrent feedbacks, FOFE based FNNLMs can significantly outperform not only the standard fixed-input FNN-LMs but also the popular recurrent neural network (RNN) LMs.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhenhua Ling
2. Xiao-Dan Zhu
3. Si Wei
4. Hui Jiang
5. QUAN LIU
6. Yu Hu
7. Yang Peng
8. Qian Chen

Citations in the corpus

• Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks (2015, background): “For examples, the conventional structured feed-forward network [38] is a model contains multiple fully connected layers of perceptrons.”
• FreebaseQA: A New Factoid QA Data Set Matching Trivia-Style Question-Answer Pairs with Freebase (2019, methodology): “Finally, we use FOFE-net (Zhang et al., 2015; Xu et al., 2017) to build a baseline KBQA system on FreebaseQA, which consists of subject detection, entity linking and relation detection in the pipeline.”
• Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems (2016, methodology): “For the composition function, we design to use the recently proposed approach, named fixed-size ordinally-forgetting encoding (FOFE) (Zhang et al. 2015).”
• Higher Order Recurrent Neural Networks (2016, methodology): “…Penn Treebank (PTB) and English text8 sets, to compare our proposed HORNNs with traditional n-gram LMs, RNN-based LMs and the state-of-the-art performance obtained (Graves, 2013; Mikolov et al., 2014), FOFE based feedforward NNs (Zhang et al., 2015) and memory networks (Sukhbaatar et al., 2015).”
• Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge (2016, methodology): “For one sentence, the function FOFE works as follows.”