HUBERT Untangles BERT to Improve Transfer across NLP Tasks
By Mehrad Moradshahi, Hamid Palangi, Monica S. Lam, Paul Smolensky, Jianfeng Gao. 2019
19 citations recorded by the source corpus.
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tal Linzen
- R. Thomas McCoy
- Yonatan Belinkov
- Junghyun Min
- Dipanjan Das
- Emily Pitler
- Lawrence Carin
- Yoav Goldberg
In Computer Vision and Pattern Recognition
- Tal Linzen
- Lawrence Carin
- R. Thomas McCoy
- Dipanjan Das
- Jianqiao Li
- Sijia Wang
- Yulai Cong
- Junghyun Min
Citations in the corpus
- GAN Memory with No Forgetting (2020, background): “Transfer learning Being general and effective, transfer learning has attracted increasing attention recently in various research fields, with a focus on discriminative tasks like classification [6, 70, 90, 93, 16, 24, 44, 46, 74, 92] and those in natural language processing [3, 63, 53, 52, 2].”
- Syntactic Data Augmentation Increases Robustness to Inference Heuristics (2020, methodology): “These include syntax-aware models (Moradshahi et al., 2019; Pang et al., 2019), auxiliary models designed to capture pre-defined shallow heuristics so that the main model can focus on robust strategies (Clark et al., 2019; He et al., 2019; Mahabadi and Henderson, 2019), and methods to up-weight…”