Resources and End-to-End Neural Network Models for Arabic Image Captioning
By Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem M. Hajj, Daniel C. Asmar. 2020
41 citations recorded by the source corpus.
: Image Captioning (IC) is the process of automatically augmenting an image with semantically-laden descriptive text. While English IC has made remarkable strides forward in the past decade, very little work exists on IC for other languages. One possible solution to this problem is to boostrap off of existing English IC systems for image understanding, and then translate the outcome to the required language. Unfortunately, as this paper will show, translated IC is lacking due to the error accumulation of the two tasks; IC and translation. In this paper, we address the problem of image captioning in Arabic. We propose an end-to-end model that directly transcribes images into Arabic text. Due to the lack of Arabic resources, we develop an annotated dataset for Arabic image captioning (AIC). We also develop a base model for AIC that relies on text translation from English image captions. The two models are evaluated with the new dataset, and the results show the superiority of our end-to-end model.
Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Natural Language Processing
In Multimodal Learning
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Data for specialized domains like medical report generation [15,40] and low-resource language captioning [18,74] cannot be easily scaled.”