Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT
By Han He, Jinho D. Choi. 2019
51 citations recorded by the source corpus.
This paper presents new state-of-the-art models for three tasks, part-of-speech tagging, syntactic parsing, and semantic parsing, using the cutting-edge contextualized embedding framework known as BERT. For each task, we first replicate and simplify the current state-of-the-art approach to enhance its model efficiency. We then evaluate our simplified approaches on those three tasks using token embeddings generated by BERT. 12 datasets in both English and Chinese are used for our experiments. The BERT models outperform the previously best-performing models by 2.5% on average (7.5% for the most significant case). Moreover, an in-depth analysis on the impact of BERT embeddings is provided using self-attention, which helps understanding in this rich yet representation. All models and source codes are available in public so that researchers can improve upon and utilize them to establish strong baselines for the next decade.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Kyle Richardson
- Swarnadeep Saha
- Peter Clark
- Oyvind Tafjord
- Ashish Sabharwal
- Hannaneh Hajishirzi
- Shashank Srivastava
In Computer Vision and Pattern Recognition
- Mohit Bansal
- Peter Clark
- Kyle Richardson
- Swarnadeep Saha
- Oyvind Tafjord
- Sayan Ghosh
- Shashank Srivastava
- Hannaneh Hajishirzi
In Natural Language Processing
- Mohit Bansal
- Kyle Richardson
- Swarnadeep Saha
- Peter Clark
- Oyvind Tafjord
- Ashish Sabharwal
- Shashank Srivastava
- Sayan Ghosh
Citations in the corpus
- Transformers as Soft Reasoners over Language (2020, background): “…that transformers either already know [Talmor et al. , 2019] or can learn to emulate other algorithms, including for semantic parsing [He and Choi, 2019], machine translation [Wang et al. , 2019], symbolic integration [Lample and Charton, 2019], and mathematics [Saxton et al. , 2019].”
- PRover: Proof Generation for Interpretable Reasoning over Rules (2020, background): “, 2019) or can be fine-tuned for tasks like semantic parsing (He and Choi, 2020), translation (Wang et al.”
- Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, unknown): “For POS experiments, the representation of a tokenized word is its last wordpiece (Liu et al., 2019a; He and Choi, 2019).”