Investigating statistical machine learning as a tool for software development

By Kayur Patel, James A. Fogarty, James A. Landay, Beverly L. Harrison. 2008

144 citations recorded by the source corpus.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition, Information Systems and Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Information Systems and Management

Citations in the corpus

Anchors: High-Precision Model-Agnostic Explanations (2018, background): “Relying only on validation accuracy has many well studied problems, as practitioners consistently overestimate their model’s accuracy (Patel et al. 2008), propagate feedback loops (Sculley et al. 2015), or fail to notice data leaks (Kaufman, Rosset, and Perlich 2011).”
Software Engineering for Machine Learning: A Case Study (2019, methodology): “One commonly used machine learning workflow at Microsoft has been depicted in various forms across industry and research [1], [9], [10], [11].”
FeatureInsight: Visual support for error-driven feature ideation in text classification (2015, background): “Practitioners often begin by searching for existing features from related problems, using their own domain knowledge to think of relevant features, or trying to algorithmically generate features for the given dataset [1], [7].”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Prior research in HCI has drawn particular attention to work practices and challenges faced by practitioners in working with data [48, 65, 86, 93, 96].”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, background): “This user need corroborates results from [24], which suggests that tools for ML practitioners should support (1) an iterative and exploratory process; (2) comprehension of the relationships between data and models; and (3) evaluation of model performance in the context of an application.”
Toolkit to support intelligibility in context-aware applications (2010, background): “Although important [33], the toolkit does not currently capture and explain these preFigure 7: IM Autostatus.”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “Patel et al. (2008) suggest there is value in integrating data collection and feature creation tools.”
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models (2017, background): “In developing interpretation tools, revealing relationships between data and models is one of the the most important design goals [29, 30].”
How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation (2019, result): “Patel et al. showed that data science workers engage in creating features as inputs to formal modeling software [72], in contrast to the more conventional, “given” data concept [10] or of extracted features [48], [85].”
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers (2018, unknown): “Building machine learning models is an iterative design process [71], [72], and developing deep neural networks is no different.”
Human-AI Collaboration in Data Science (2019, background): “explored analogous human-constructive activities during feature engineering [55].”
Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, background): “While held-out accuracy is often useful, it is not sufficient: practitioners consistently overestimate their model’s generalization (Patel et al., 2008) since test data is usually gathered in the same manner as training and validation.”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, background): “Patel et al. [53] and Zhu et al. [69] provide domain-speciﬁc challenges to designing iML systems for software developers and game designers, respectively.”
Flock: Hybrid Crowd-Machine Learning Classifiers (2015, background): “Flock is a hybrid crowd-machine learning platform that capi-talizes on analogical encoding to guide crowds to nominate effective features, then uses machine learning techniques to aggregate their labels. after many iterations [36].”
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020, background): “, 2018), such that real-world performance may be overestimated (Patel et al., 2008; Recht et al., 2019).”