Image-to-Markup Generation with Coarse-to-Fine Attention

By Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush. 2016

276 citations recorded by the source corpus.

We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, background): “2 Coarse-to-ﬁne attention (Ling and Rush, 2017; Deng et al., 2017) was proposed for the same motivation, but they resort to reinforcement learning which is hard to train, and the performance is sacriﬁced for efﬁciency.”
Learning to Infer Graphics Programs from Hand-Drawn Images (2017, background): “IM2LATEX [17] is a recent work that derenders LATEX equations, recovering a markup language representation.”
Image-based table recognition: data, model, and evaluation (2019, background): “Camelot9, PDFPlumber10, and Adobe Acrobat R © Pro11) and the WYGIWYS model12 [34].”
Semantic Search in Millions of Equations (2020, background): “While one could argue that this is an unsuitable choice, the multitude of machine-learning or computervision approaches that successfully transform images of typeset [4] or hand-written [1, 15] math back to tree-based representations suggests that bitmap representations preserve all required informa”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “Im2Tex (Deng et al., 2017) is conceptually the most relevant in showing that a pixel-only parser can be learned from freely-available pairs of markup and renders, but doesn’t focus on transferring this signal to wider applications.”
Latent Alignment and Variational Attention (2018, background): “These models have proven much more difﬁcult to train, and existing works typically treat hard attention as a black-box reinforcement learning problem with log-likelihood as the reward [80, 3, 53, 26, 19].”