InternLM2 Technical Report

By Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu and 90 more. 2024

416 citations recorded by the source corpus.

The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack"test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.

Subfields: Artificial Intelligence, Health Informatics, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Lu
2. Hongxu Yin
3. Zhaoyang Liu
4. Jan Kautz
5. Ruihan Yang
6. Weijie Su
7. Dahua Lin
8. Yinan He

In Large Language Models

1. Dahua Lin
2. Yinan He
3. Conghui He
4. Jan Kautz
5. Xizhou Zhu

Citations in the corpus

• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “Compared (Cai et al., 2024) Discriminative 90.2 98.9 76.5 89.9 95.8 Llama-3-OffsetBias-RM-8B (Park et al., 2024) Discriminative 89.4 97.2 81.8 86.8 91.9 gemini-1.”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “This method serves as a baseline to gauge how much spatial reasoning can be derived from purely existing world knowledge.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “This pipeline utilizes state-of-the-art open-source LLMs [6, 13, 125] or GPT-3.”
• Visual-RFT: Visual Reinforcement Fine-Tuning (2025, neutral): “Studies have explored improving LLMs’ performance in reasoning tasks such as solving mathematical problems [2, 22, 33, 43, 45] and coding [6, 8, 51, 54].”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…enhancing model architectures [220, 232, 5, 172, 157, 210], scaling vision encoders [252, 66, 36, 293, 185] and language models [231, 235, 64, 19, 229, 221, 62], incorporating more diverse and high-quality datasets [124, 234, 25, 155], and refining the test-time scaling process [215, 249,…”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “(2) Text-only data: this part of data is constructed based on the training corpora of InternLM series [9, 124] and is further augmented with open-source datasets [6, 73, 75, 94].”
• LLM-confidence reranker: A training-free approach for enhancing retrieval-augmented generation systems (2026, unknown): “5-7B-Chat (InternLM7B) (Cai et al., 2024).”