TypeWriter: neural type prediction with search-based validation

By Michael Pradel, Georgios Gousios, Jason Liu, Satish Chandra. 2019

135 citations recorded by the source corpus.

Maintaining large code bases written in dynamically typed languages, such as JavaScript or Python, can be challenging due to the absence of type annotations: simple data compatibility errors proliferate, IDE support is limited, and APIs are hard to comprehend. Recent work attempts to address those issues through either static type inference or probabilistic type prediction. Unfortunately, static type inference for dynamic languages is inherently limited, while probabilistic approaches suffer from imprecision. This paper presents TypeWriter, the first combination of probabilistic type prediction with search-based refinement of predicted types. TypeWriter’s predictor learns to infer the return and argument types for functions from partially annotated code bases by combining the natural language properties of code with programming language-level information. To validate predicted types, TypeWriter invokes a gradual type checker with different combinations of the predicted types, while navigating the space of possible type combinations in a feedback-directed manner. We implement the TypeWriter approach for Python and evaluate it on two code corpora: a multi-million line code base at Facebook and a collection of 1,137 popular open-source projects. We show that TypeWriter’s type predictor achieves an F1 score of 0.64 (0.79) in the top-1 (top-5) predictions for return types, and 0.57 (0.80) for argument types, which clearly outperforms prior type prediction models. By combining predictions with search-based validation, TypeWriter can fully annotate between 14% to 44% of the files in a randomly selected corpus, while ensuring type correctness. A comparison with a static type inference tool shows that TypeWriter adds many more non-trivial types. TypeWriter currently suggests types to developers at Facebook and several thousands of types have already been accepted with minimal changes.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis, Program Analysis & Verification

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

In Program Analysis & Verification

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We compare our zero-shot model to one such approach for Python, TypeWriter [50], which combines a neural architecture for type hint prediction with a search-based incremental type validation procedure.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
SantaCoder: don't reach for the stars! (2023, neutral): “Evaluating other tasks Code generation models have also been used to solve a variety of tasks (Tufano et al., 2020; Feng et al., 2020b; Ahmed & Devanbu, 2022; Hellendoorn et al., 2018; Pradel et al., 2020).”
Learning to Predict User-Defined Types (2023, methodology): “Unknown : In previous works with ﬁxed type vocabularies [9], [10], [11], [12], [13] type inference models would predict UNK if the type exceeded its classiﬁcation capabilities.”
A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages (2022, background): “…have been applied to a variety of tasks, including test generation (Tufano et al. 2020), doc-string generation (Lu et al. 2021), code search (Feng et al. 2020; Ahmed and Devanbu 2022), type inference (Wei et al. 2020; Hellendoorn et al. 2018; Pradel et al. 2020), and more (Drori et al. 2022).”
Sound, heuristic type annotation inference for Ruby (2020, methodology): “TypeWriter [28] uses a neural model to predict Python types based on natural language and code context information, and uses a gradual type checker to rule out incorrect types.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, methodology): “Code generation models have been applied to a variety of tasks, including test generation [19], docstring generation [20], code search [17, 21], type inference [22, 23, 24], and more [25].”
Contrastive Code Representation Learning (2020, background): “, 2020) and types (Pradel et al., 2019; Pandi et al., 2020; Wei et al., 2020; Allamanis et al., 2020; Bielik and Vechev, 2020; Allamanis et al., 2018) for various languages.”
Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python (2021, methodology): “[64] PGM DeepTyper [26] 10K+ biRNN NL2Type [41] 1K LSTM TypeWriter [51] 1K HNN (3x RNNs) LAMBDANET [62] 100a GNN OptTyper [47] 100 LSTM Typilus [12] Unlimited GNN TypeBert [29] 40K BERT”
Learning type annotation: is big data enough? (2021, background): “Pradel [20] combines a probabilistic guessing component with a typechecker that verifies the proposed annotations.”
SimTyper: sound type inference for Ruby using type equality prediction (2021, methodology): “TypeWriter [Pradel et al. 2019] trains a neural model to predict types based on identifier names, comments, and source code from Python programs.”