Learning Bayesian networks: The combination of knowledge and statistical data

By David E. Heckerman, Dan Geiger, David Maxwell Chickering. 1994

We describe a Bayesian approach for learning Bayesian networks from a combination of prior knowledge and statistical data. First and foremost, we develop a methodology for assessing informative priors needed for learning. Our approach is derived from a set of assumptions made previously as well as the assumption oflikelihood equivalence, which says that data should not help to discriminate network structures that represent the same assertions of conditional independence. We show that likelihood equivalence when combined with previously made assumptions implies that the user's priors for network parameters can be encoded in a single Bayesian network for the next case to be seen—aprior network—and a single measure of confidence for that network. Second, using these priors, we show how to compute the relative posterior probabilities of network structures given data. Third, we describe search methods for identifying network structures with high posterior probabilities. We describe polynomial algorithms for finding the highest-scoring network structures in the special case where every node has at mostk=1 parent. For the general case (k>1), which is NP-hard, we review heuristic search algorithms including local search, iterative local search, and simulated annealing. Finally, we describe a methodology for evaluating Bayesian-network learning algorithms, and apply this approach to a comparison of various approaches.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Bayesian Structure Learning with Generative Flow Networks (2022, methodology): “Examples of modular scores include the BDe score (Heckerman et al., 1995) and the BGe score (Geiger and Heckerman, 1994; Kuipers et al., 2014).”
An Analysis of the Adaptation Speed of Causal Models (2020, neutral): “In Bayesian Networks literature, this is known as the Bayesian Dirichlet equivalent uniform prior (Heckerman et al., 1995).”
Learning Neural Causal Models from Unknown Interventions (2019, background): “…possible directed acyclic graphs (DAGs) representing the causal structure based on some form of scoring function for network structures (Chickering, 2002; Tsamardinos et al., 2006; Goudet et al., 2017; Hauser & Bühlmann, 2012; Heckerman et al., 1995; Cooper & Yoo, 1999; Hauser & Bühlmann, 2012).”
Causal Discovery from Heterogeneous/Nonstationary Data (2019, background): “The so-called score-based approach (see, e.g., (Chickering, 2003; Heckerman et al., 1995)) searches for the equivalence class which gives the highest score under some scoring criteria, such as the Bayesian Information Criterion (BIC), the posterior of the graph given the data (Heckerman et al.,…”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, methodology): “Approaches for Bayesian network structure learning based on discrete search over model structures and simulated annealing are reviewed in Heckerman et al. (1995).”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, methodology): “Given the above assumptions, the marginal likelihood can be expressed as (Heckerman, Geiger, and Chickering 1995): P (D|M ) = B b=1 Γ( N B ) Γ(N b + N B ) Γ(m b + α b ) Γ(α b ) Γ(n b + β b ) Γ(β b ) , where Γ is the gamma function and N b is the total number of training instances located in the…”
DAGs with NO TEARS: Continuous Optimization for Structure Learning (2018, methodology): “However it is also interesting to consider non-smooth, even discrete scores such as BDe (Heckerman et al., 1995).”
Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm (2005, background): “The notion of d-separation can be defined via moral graphs; details are described in Lauritzen (1996, Prop. 3.25). We remark here that faithfulness is ruling out some classes of probability distributions. An example of a non-faithful distribution is given in Spirtes et al. (2000, Chapter 3.5.2). On ”
Order-independent constraint-based causal structure learning (2012, methodology): “ constraint-based method, as it relies on conditional independence constraints. Alternative approaches include score-based and Bayesian methods (e.g., Chickering (2002); Cooper and Herskovits (1992); Heckerman et al. (1995); Spiegelhalter et al. (1993)), as well as hybrids of the different approache”
Learning with Submodular Functions: A Convex Optimization Perspective (2011, background): “In the context of probabilistic graphical models, entropies occur in particular in algorithms for structure learning: indeed, for directed graphical models, given the directed acyclic graph, the minimum KullbackLeibler divergence between a given distribution and a distribution that factorizes into t”
DAG-GNN: DAG Structure Learning with Graph Neural Networks (2019, methodology): “Score-based methods generally use a score to measure the goodness of fit of different graphs over data; and then use a search procedure—such as hillclimbing (Heckerman et al., 1995; Tsamardinos et al., 2006; Gmez et al., 2011), forward-backward search (Chickering, 2002), dynamic programming (Singh &”
Discriminative Training of Markov Logic Networks (2005, methodology): “In addition, we compared MLN learning with a pure ILP approach (CLAUDIEN (De Raedt & Dehaspe 1997) (CL)), a pure knowledge-based approach (the hand-coded KB (KB)), and two pure probabilistic approaches: naive Bayes (NB) (Domingos & Pazzani 1997) and Bayesian networks (BN) ( Heckerman, Geiger, & Chic”
On Structure Priors for Learning Bayesian Networks (2019, background): “The marginal likelihood is an average over a parameter prior, different choices of which result in different scoring functions, such as the Bayesian Dirichlet (BD) family for categorical data (Heckerman et al. 1995).”
Estimating Causal Direction and Confounding of Two Discrete Variables (2016, background): “In a classic work on Bayesian Network learning (then called Belief Net learning), Heckerman and Chick-ering (Heckerman et al., 1995; Chickering, 2002a,b) developed a Bayesian scoring criterion that allowed them to deﬁne the posterior probability of each possible Bayesian network given a dataset.”
Learning Object Affordances: From Sensory--Motor Coordination to Imitation (2008, methodology): “We used the K2 algorithm 3 to find the maximum likelihood graph with a random starting point and BDeu prior s [27] to give uniform priors to different equivalence classe .”
Abstraction Levels for Robotic Imitation: Overview and Computational Approaches (2010, background): “Based on such prior experience, structure learning [60] can be used to distinguish all such relations.”