“Dark Knowledge”

By Geoffrey E. Hinton. 2020

This paper is about the dark realm of knowledge that is nuclear science. I discuss the uses and abuses of this knowledge over the last century, and an installation project I developed that seeks to manifest such dark knowledge, titled Demon Core (2019). To enframe this discussion, I shall start with some quotes from a somewhat dark materialist philosopher Martin Heidegger and his key text, The Question Concerning Technology (1977). Heidegger says that technology is no mere means. It's a way of “ revealing ” , which is a sort of knowing in the wider material sense - through our interactions with matter, working with matter, making matter do things, one gains material knowledge. Technology, the tools and techniques we use to work with matter, is a mode of “ revealing ” aspects or attributes of nature. Heidegger asks ‘ What is modern technology? It too is a revealing. …And yet … The revealing that rules in modern technology is a challenging’ .[1] Con-temporary technology is different to traditional technologies, as it's based on modern science and engineering. Technology challenges nature and drives it to do certain things, such as taking the energy that naturally exists within a river and using it through the technology of a dam. Technology based upon modern physics takes this to a much more fundamental level: ‘ The Earth is now set upon to yield uranium. Uranium is set upon to yield Atomic Energy, which can be released either for destruction or for peaceful use … . This setting upon that challenges for the energies of nature is an expediting in two ways. It unlocks and exposes … That challenging happens in that the energy concealed in nature is unlocked, what is unlocked is trans-formed…’ .[2] Through such challenging and transforming of materials and energies, technology enframes nature, challenges it, summoning it forth and forcing it to work for us, in ways against its innate tendencies . A darkly archetypal example of this is the atom bomb. The 20th and

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

How to Represent Part-Whole Hierarchies in a Neural Network (2021, methodology): “Co-distillation was initially based on an analogy with how scientists in a community learn from each other (Hinton, 2014), but the same mechanism could be used in a community of columns.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Compared to ER [86], DER further distills previously learned knowledge with the help of Dark Knowledge [219] and regularizes the model based on the network’s logits so that consistent performance can be expected through the sequential stream of data without a clear task boundary.”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, unknown): “We use the conventional dark-knowledge-based distillation [42].”
Online Continual Learning through Mutual Information Maximization (2022, background): “DER++ (Buzzega et al., 2020) utilizes the dark knowledge (Hinton et al., 2014) and knowledge distillation.”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “Compression Wide, over-parametrized, layers have been shown to be important for learning (Neyshabur et al., 2018), but it is often possible to reduce the layer size a posteriori without losing signiﬁcant accuracy (Hinton et al., 2014; LeCun et al., 1990).”
Consistency by Agreement in Zero-Shot Neural Machine Translation (2019, background): “Another family of approaches is based on distillation (Hinton et al., 2014; Kim and Rush, 2016).”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, unknown): “Though sharing the similarity in expanding the task-relevant data, the two methods are different in the augmented knowledge: ( i ) For an image, the supervision signal in self-training is based on the teacher model’s internal “dark knowl-edge” [28], which is limited in a ﬁxed prediction space.”