Deep learning type inference

By Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, Miltiadis Allamanis. 2018

222 citations recorded by the source corpus.

Dynamically typed languages such as JavaScript and Python are increasingly popular, yet static typing has not been totally eclipsed: Python now supports type annotations and languages like TypeScript offer a middle-ground for JavaScript: a strict superset of JavaScript, to which it transpiles, coupled with a type system that permits partially typed programs. However, static typing has a cost: adding annotations, reading the added syntax, and wrestling with the type system to fix type errors. Type inference can ease the transition to more statically typed code and unlock the benefits of richer compile-time information, but is limited in languages like JavaScript as it cannot soundly handle duck-typing or runtime evaluation via eval. We propose DeepTyper, a deep learning model that understands which types naturally occur in certain contexts and relations and can provide type suggestions, which can often be verified by the type checker, even if it could not infer the type initially. DeepTyper, leverages an automatically aligned corpus of tokens and types to accurately predict thousands of variable and function type annotations. Furthermore, we demonstrate that context is key in accurately assigning these types and introduce a technique to reduce overfitting on local cues while highlighting the need for further improvements. Finally, we show that our model can interact with a compiler to provide more than 4,000 additional type annotations with over 95% precision that could not be inferred without the aid of DeepTyper.

Subfields: Information Systems, Software, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

In Signal Processing

Citations in the corpus

The adverse effects of code duplication in machine learning models of code (2018, background): “• Type prediction models [13, 23] where the goal is to infer (or provide probabilistic hints for) the types of new, previously untyped, programs (e.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically ge”
Do Machine Learning Models Produce TypeScript Types that Type Check? (2023, methodology): “LambdaNet [50] and DeepTyper [23] are two different approaches for predicting types for TypeScript and JavaScript programs.”
SantaCoder: don't reach for the stars! (2023, neutral): “Evaluating other tasks Code generation models have also been used to solve a variety of tasks (Tufano et al., 2020; Feng et al., 2020b; Ahmed & Devanbu, 2022; Hellendoorn et al., 2018; Pradel et al., 2020).”
TypeWriter: neural type prediction with search-based validation (2019, methodology): “As all existing type prediction models [15, 21, 29] target JavaScript code, it is difficult to draw conclusions as to whether the TypeWriter architecture is the best for the task.”
LambdaNet: Probabilistic Type Inference using Graph Neural Networks (2020, methodology): “In this experiment, we compare L AMBDA N ET ’s performance with DeepTyper (Hellendoorn et al., 2018), which treats programs as sequences of tokens and uses a bidirectional RNN to make type predictions.”
Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python (2021, methodology): “To address these limitations of static type inference methods, researchers have recently employed Machine Learning (ML) techniques for type prediction in dynamic languages [2, 16, 31, 51].”
Learning type annotation: is big data enough? (2021, methodology): “LambdaNet [25] (like other recent works) used sophisticated inductive biases [3, 4, 10] to achieve state-of-the-art (SOTA) type inference, improving substantially upon earlier approaches like Hellendoorn [11] and Malik et al. [17].”
Backdoors in Neural Models of Source Code (2020, methodology): “The insights we obtain from our proposed defense for the task of code summarization, have potential applications to other tasks in NLP. Deep learning for source code: See [26] for a survey on machine learning for code.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, methodology): “Code generation models have been applied to a variety of tasks, including test generation [19], docstring generation [20], code search [17, 21], type inference [22, 23, 24], and more [25].”
The growing cost of deep learning for source code (2021, background): “…that their future scales align as well; as the AI+SE field has evolved, the application domain of AI models has grown from relatively simple objectives, like token-level code completion, 9,11 to more semantically challenging tasks, such as type inference, 8 bug detection, 21 and program repair.”
Learning to Predict User-Defined Types (2023, methodology): “TypeBert, like other ﬁxed type vocabulary models [8], [9], [11], [12], ignores new types.”
Contrastive Code Representation Learning (2020, methodology): “While improvements on adversarial benchmarks would not be expected to translate to real code, we ﬁnd signiﬁcant improvements in extreme code summarization (Allamanis et al., 2016) and type inference (Hellendoorn et al., 2018) tasks.”
A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages (2022, background): “…have been applied to a variety of tasks, including test generation (Tufano et al. 2020), doc-string generation (Lu et al. 2021), code search (Feng et al. 2020; Ahmed and Devanbu 2022), type inference (Wei et al. 2020; Hellendoorn et al. 2018; Pradel et al. 2020), and more (Drori et al. 2022).”
Sound, heuristic type annotation inference for Ruby (2020, methodology): “DeepTyper [16] uses bidirectional RNNs trained on JavaScript source code to infer types from over 11,000 types in its training dataset, while NL2Type [22] trains an RNN exclusively on JavaScript programs’ natural language information, such as comments and identifier names, to predict from a set of 1”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, background): “[16] to extract type annotations of the identifiers.”
SimTyper: sound type inference for Ruby using type equality prediction (2021, background): “DeepTyper [Hellendoorn et al. 2018] trains a bidirectional recurrent neural network (RNN) on JavaScript source code to predict type annotations from over 11,000 types seen in its training data.”