RSVQA: Visual Question Answering for Remote Sensing Data

By Sylvain Lobry, Diego Marcos, Jesse Murray, Devis Tuia. IEEE Transactions on Geoscience and Remote Sensing, 2020

385 citations recorded by the source corpus.

This article introduces the task of visual question answering for remote sensing data (RSVQA). Remote sensing images contain a wealth of information, which can be useful for a wide range of tasks, including land cover classification, object counting, or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With RSVQA, we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high-level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method introduced in this article, we built two data sets (using low- and high-resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM). The data sets can be used to train (when using supervised methods) and evaluate models to solve the RSVQA task. We report the results obtained by applying a model based on convolutional neural networks (CNNs) for the visual part and a recurrent neural network (RNN) for the natural language part of this task. The model is trained on the two data sets, yielding promising results in both cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1015e77ecc84237515cccf9fc583796c38584c28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “S. Lobry et al. [90] proposed a VQA framework for remote sensing data, which can be useful for land cover classification tasks.”
GeoChat:Grounded Large Vision-Language Model for Remote Sensing (2023, methodology): “RSVQA-LR [20] is made up of 772 low-resolution images and 77,232 question-answer pairs, with 77.8%, 11.1%, and 11.1% used for training, validation, and testing, respectively.”
RSGPT: A Remote Sensing Vision Language Model and Benchmark (2023, methodology): “RSIEval provides the most diverse set of questions and answers to date, compared to existing RSVQA datasets such as RSVQA [33], RSIVQA [82], and RSVQA × BEN [32].”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “3) VQA-to-Instruction: For VQA datasets including Flood-net [74], RSVQA-LR [45], RSIVQA [75], and CRSVQA [16], the instruction “Answer the question using a single word or phrase” is added after the original question to control the output format of the answer.”
The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot (2023, background): “This is achieved through "prompt engineering," where prompts are carefully designed to guide the model toward generating the desired output [29, 48].”