Modeling Intra-Relation in Math Word Problems with Different Functional Multi-Head Attentions

By Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, Dongxiang Zhang. 2019

109 citations recorded by the source corpus.

Several deep learning models have been proposed for solving math word problems (MWPs) automatically. Although these models have the ability to capture features without manual efforts, their approaches to capturing features are not specifically designed for MWPs. To utilize the merits of deep learning models with simultaneous consideration of MWPs’ specific features, we propose a group attention mechanism to extract global features, quantity-related features, quantity-pair features and question-related features in MWPs respectively. The experimental results show that the proposed approach performs significantly better than previous state-of-the-art methods, and boost performance from 66.9% to 69.5% on Math23K with training-test split, from 65.8% to 66.9% on Math23K with 5-fold cross-validation and from 69.2% to 76.1% on MAWPS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangliang Zhang
2. Zhenwen Liang
3. Jipeng Zhang
4. Wei Dang Lu
5. Nakul Verma
6. Iddo Drori
7. Lei Wang
8. Yibin Shen

In Computer Vision and Pattern Recognition

1. Xiangliang Zhang
2. Zhenwen Liang
3. Jun Zhao
4. Kang Liu
5. Ji-rong Wen
6. Kun Zhou
7. Rodrigo Nogueira
8. Wei Dang Lu

In Natural Language Processing

1. Xiangliang Zhang
2. Zhenwen Liang
3. Iddo Drori
4. Nakul Verma
5. Jipeng Zhang
6. Wei Dang Lu
7. Jierui Li
8. Lei Wang

In Model Architectures

1. Zhenwen Liang
2. Xiangliang Zhang
3. Jun Zhao
4. Rodrigo Nogueira
5. Wei Dang Lu
6. Kun Zhou
7. Lei Wang
8. Liang Wang

Citations in the corpus

• MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, methodology): “The models include Seq2seq, Seq2tree, Graph2tree, and commonly-used non-pretrained (AttSeq2Seq (Bahdanau et al., 2014), LSTMVAE (Zhang et al., 2016), and Trans-former (Vaswani et al., 2017)) and pretrained models (GPT-2 (Radford et al., 2019), BERTGen (De-vlin et al., 2018), and RoBERTaGen (Liu et a”
• Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “GroupAttn (Li et al., 2019) designed several types of attention mechanisms such as question or quantity related attentions in the seq2seq model.”
• Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, methodology): “Numerous deep learning based methods have been proposed to tackle the MWP task, including Seq2Seq (Chiang and Chen, 2019; Li et al., 2019), Seq2Tree (Wang et al., 2019; Qin et al., 2020), and Pre-trained Language Models (Kim et al., 2020; Liang et al., 2021).”
• Graph-to-Tree Learning for Solving Math Word Problems (2020, methodology): “GROUPATT (Li et al., 2019) borrows the idea of multihead attentions from Transformer (Vaswani et al.”
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “Similarly, better numerical skills can be achieved by augmenting input texts with pre-computed numerical computations (Andor et al., 2019) or by explicitly inferring mathematical equations from natural language text (Zou & Lu, 2019a;b; Li et al., 2019; Liu et al., 2019; Shi, 2020).”
• Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems (2020, methodology): “…et al., 2017) 64.7 StackDecoder (Chiang and Chen, 2018) 65.8 Seq2SeqET (Wang et al., 2018) 66.7* T-RNN (Wang et al., 2019) 66.9* Group Attention (Li et al., 2019) 66.9 TreeDecoder (Liu et al., 2019) 69.5 GTS-model (Xie and Sun, 2019) 74.3 Our Approach 77.5 Table 6: The performance of models on…”
• Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Similarly, better numerical skills can be achieved by augmenting input texts with pre-computed numerical computations (Andor et al., 2019) or by explicitly inferring mathematical equations from natural language text (Zou & Lu, 2019a;b; Li et al., 2019; Liu et al., 2019; Shi, 2020).”
• Solving Linear Algebra by Program Synthesis (2021, methodology): “Techniques have included sequence-tosequence and graph-to-tree Transformers which achieve around 80% on Math23k and MAWPS datasets (Koncel-Kedziorski et al., 2016; Li et al., 2019; Wang et al., 2019; Zhang et al., 2020; Li et al., 2020; Wu et al., 2020; Qin et al., 2020; Lan et al., 2021).”
• Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems (2021, unknown): “In addition to comparing with BERT, we also investigate the following approaches: GroupAttention 2 (Li et al., 2019) develop an attention mechanism to capture the quantity-related and question-related information.”
• Math Word Problem Solving with Explicit Numerical Values (2021, background): “Li et al. (2019) propose a group attention mechanism to extract quantity-related features and quantity-pair features.”
• Large Language Models are Better Reasoners with Self-Verification (2022, methodology): “Early work attempted to solve complex reasoning tasks using Seq2Seq models (Wang et al., 2018; Li et al., 2019).”
• Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem (2020, background): “As shown in Table 5, our Graph2Tree model consistently outperforms other state-of-the-art baselines by a large margin up to 10 points absolute accuracy except Group-Att baseline.”
• Solving Math Word Problems with Multi-Encoders and Multi-Decoders (2020, background): “(Li et al., 2019) added different functional multihead attentions to the Seq2Seq framework.”
• Solving Math Word Problems with Teacher Supervision (2021, methodology): “, 2017], multi-head attention mechanism is also applied in MWP solver [Li et al. , 2019].”
• HMS: A Hierarchical Solver with Dependency-Enhanced Understanding for Math Word Problem (2021, methodology): “There is much effort in solving math word problems from early rule-based approaches (Fletcher 1985), statistical machine learning (Hosseini et al. 2014), semantic parsing (Koncel-Kedziorski et al. 2015) to recent sequence-to-sequence (seq2seq) methods (Wang et al. 2019; Xie and Sun 2019).”
• MWP-BERT: A Strong Baseline for Math Word Problems (2021, methodology): “…has been formulated as a pipeline of translating language descriptions to mathematical equations with neural network based encoder-decoder framework (Wang et al., 2018, 2019; Chiang and Chen, 2018; Liu et al., 2019; Li et al., 2019; Xie and Sun, 2019; Zhang et al., 2020b; Shen and Jin, 2020).”
• A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “Various attention mechanisms have been studied to extract better representations, such as Group-ATT (Li et al., 2019) which uses different multi-head attention to extract various types of MWP features, and graph attention which is applied to extract knowledge-aware information in (Wu et al., 2020).”