The MADlib Analytics Library or MAD Skills, the SQL

By Joseph M Hellerstein, Christopher R'e, Florian Schoppmann, Daisy Zhe Wang, Eugene Fratkin, Aleksander Gorajek, Kee Siong Ng, Caleb Welton, Xixuan Feng, Kun Li and 1 more. 2012

449 citations recorded by the source corpus.

MADlib is a free, open source library of in-database analytic methods. It provides an evolving suite of SQL-based algorithms for machine learning, data mining and statistics that run at scale within a database engine, with no need for data import/export to other tools. The goal is for MADlib to eventually serve a role for scalable database systems that is similar to the CRAN library for R: a community repository of statistical methods, this time written with scale and parallelism in mind. In this paper we introduce the MADlib project, including the background that led to its beginnings, and the motivation for its open source nature. We provide an overview of the library's architecture and design patterns, and provide a description of various statistical methods in that context. We include performance and speedup results of a core design pattern from one of those methods over the Greenplum parallel DBMS on a modest-sized test cluster. We then report on two initial efforts at incorporating academic research into MADlib, which is one of the project's goals. MADlib is freely available at this http URL, and the project is open for contributions of both new methods, and ports to additional database platforms.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

Citations in the corpus

BoostClean: Automated Error Detection and Repair for Machine Learning (2017, methodology): “The database community has built systems to support almost every stage of the development process including featurization [1,53], distributed model training [2,17,19,24], and model de-”
MLbase: A Distributed Machine-learning System (2013, unknown): “[4] Weka. http://www.cs.waikato.ac.nz/ml/weka/.”
Towards Linear Algebra over Normalized Data (2016, unknown): “The data management industry and academia are working intensively on tools to integrate machine learning (ML) algorithms and frameworks such as R with data systems [1,9,10, 18,20,23,33].”
Opaque: An Oblivious and Encrypted Distributed Analytics Platform (2017, background): “variety of big data workloads, including complex graph analytics [14] and machine learning [17].”
TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries (2015, methodology): “…techniques, bandit resource allocation, and batching optimizations, T U PAQ identiﬁes a suitable model to satisfy a user’s high-level declarative query, thus differing from previous systems [29, 20, 49, 53] focused on predictive analytics, which force users to perform model search themselves.”
Are Key-Foreign Key Joins Safe to Avoid when Learning High-Capacity Classifiers? (2017, unknown): “The data management community has long studied how to integrate ML with data systems [19, 12, 52]), how to scale ML [6, 31], and how to use database ideas to improve ML tasks [25, 26].”
PALM: Machine Learning Explanations For Iterative Debugging (2017, methodology): “While the data-base community has made substantial contributions in other areas of machine learning (e.g., training scalability and model speciﬁca-tion [1, 2, 5, 8, 10]), the opportunity is ripe for developing scalable tools to debug modern machine learning models.”
Materialization optimizations for feature selection workloads (2014, methodology): “These include data mining toolkits from major RDBMS vendors, which integrate spe-ciﬁc algorithms with an RDBMS [3,23].”
Profiling relational data: a survey (2015, methodology): “Data proﬁling functionality is also included in the MADLib toolkit for scalable in-database analytics [71], including column statistics, such as count, count distinct, Recent data quality tools are dependency-driven: Classical dependencies, such as Fd s and Ind s, as well as their conditional…”
Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics (2016, background): “The past decade has seen signiﬁcant interest from both the data management industry and academia in integrating machine learning (ML) algorithms into scalable data processing systems such as RDBMSs [23, 19], Hadoop [1], and Spark [2].”
SampleClean: Fast and Reliable Analytics on Dirty Data (2015, background): “5 ActiveClean: Machine Learning on Dirty Data [31] Analytics is moving beyond SQL, and the growing popularity of predictive models [1, 7, 17, 28] leads to additional challenges in managing dirty data.”
Heterogeneity-aware Distributed Parameter Servers (2017, methodology): “The database community has developed a lot of ML systems, such as SystemML [20], SimSQL [10], Vertica-R [33], FactorizedLearning [26], and MADlib [22], in which ML analytics is expressed in a higher-level language and is compiled and executed in a MapReduce environment.”
SystemML: Declarative Machine Learning on Spark (2016, background): “Examples include Mahout [4], Spark MLlib [25], MADlib [17], Vowpal Wabbit [21], Revolution R ScaleR [28], SkyTree ML software [29], and H2O [16].”
NeurIDA: Dynamic Modeling for Effective In-Database Analytics (2025, unknown): “DFS [21] automatically engineers aggregation features from joined data.”