Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2

By Gregor Betz, Kyle Richardson, Christian Voigt. 2021

34 citations recorded by the source corpus.

Thinking aloud is an effective meta-cognitive strategy human reasoners apply to solve difficult problems. We suggest to improve the reasoning ability of pre-trained neural language models in a similar way, namely by expanding a task's context with problem elaborations that are dynamically generated by the language model itself. Our main result is that dynamic problem elaboration significantly improves the zero-shot performance of GPT-2 in a deductive reasoning and natural language inference task: While the model uses a syntactic heuristic for predicting an answer, it is capable (to some degree) of generating reasoned additional context which facilitates the successful application of its heuristic. We explore different ways of generating elaborations, including fewshot learning, and find that their relative performance varies with the specific problem characteristics (such as problem difficulty). Moreover, the effectiveness of an elaboration can be explained in terms of the degree to which the elaboration semantically coheres with the corresponding problem. In particular, elaborations that are most faithful to the original problem description may boost accuracy by up to 24%.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Large Language Models Are Human-Level Prompt Engineers (2022, background): “Early works (Nye et al., 2021; Betz et al., 2021; Wei et al., 2022b) on chain-of-thought used fine-tuning or in-context learning to get LLMs to show their work for such problems.”
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “, asking the model to elaborate on the problem) [9, 55] attempt to seed LLMs to generate more effective prompts before solving the task.”
Flexible Generation of Natural Language Deductions (2021, methodology): “This baseline, inspired by the zero-shot premise elaboration employed by Betz et al. (2021), is intended to assess the likelihood of making consistent deductions under a general model of language with no logical specialization.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Shwartz et al. (2020); Betz et al. (2021); Liu et al. (2022) use a pretrained causal LM to generate context to improve performance on multiple choice commonsense tasks.”
AI literacy and its implications for prompt engineering strategies (2024, methodology): “Using the LLM itself to elaborate on problems is another method for improving prompt design (Betz, Richardson, & Voigt, 2021); this method is comparable to the human practice of “ thinking aloud".”