Assessing the Type Annotation Burden

By J. Ore, Sebastian G. Elbaum, Carrick Detweiler, Lambros Karkazis. 2018

26 citations recorded by the source corpus.

Type annotations provide a link between program variables and domain-specific types. When combined with a type system, these annotations can enable early fault detection. For type annotations to be cost-effective in practice, they need to be both accurate and affordable for developers. We lack, however, an understanding of how burdensome type annotation is for developers. Hence, this work explores three fundamental questions: 1) how accurately do developers make type annotations; 2) how long does a single annotation take; and, 3) if a system could automatically suggest a type annotation, how beneficial to accuracy are correct suggestions and how detrimental are incorrect suggestions? We present results of a study of 71 programmers using 20 random code artifacts that contain variables with physical unit types that must be annotated. Subjects choose a correct type annotation only 51% of the time and take an average of 136 seconds to make a single correct annotation. Our qualitative analysis reveals that variable names and reasoning over mathematical operations are the leading clues for type selection. We find that suggesting the correct type boosts accuracy to 73%, while making a poor suggestion decreases accuracy to 28%. We also explore what state-of-the-art automated type annotation systems can and cannot do to help developers with type annotations, and identify implications for tool developers.

Subfields: Artificial Intelligence, Information Systems, Software, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Software Engineering

Citations in the corpus

TypeWriter: neural type prediction with search-based validation (2019, methodology): “As manually annotating code with types is time-consuming and error-prone [26], developers must resort to automated methods.”
Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python (2021, methodology): “Recent studies show that developer-provided types rarely type-check and Python projects may contain type-related defects [21, 46, 52].”
NL2Type: Inferring JavaScript Function Types from Natural Language Information (2019, background): “The main bottleneck of these existing solutions is that they rely on developers to provide type annotations, which remains a manual and time-consuming [4] task.”