mannix / alchemistcoder-7b

摘要：开源大型语言模型（LLMs）及其专用变体，特别是代码LLMs，近期表现出令人印象深刻的性能。然而，之前的代码LLMs通常在单源数据上微调，数据质量和多样性有限，这可能无法充分挖掘预训练代码LLMs的潜力。在本文中，我们介绍了AlchemistCoder，这是一系列在多源数据上微调以增强代码生成和泛化能力的代码LLMs。为实现此目标，我们首先揭露了多源代码语料库中各种风格和质量的固有冲突，并引入具有事后重新标记的数据特定提示（称为AlchemistPrompts），以协调不同数据源和指令-响应对。此外，我们将数据构建过程纳入微调数据作为代码理解任务，包括指令演变、数据过滤和代码审查。大量的实验表明，AlchemistCoder在所有相同大小（6.7B/7B）的模型中均具有明显优势，并在6个代码基准测试中与更大的模型（15B/33B/70B）相媲美甚至超越，展示了我们方法在提高指令遵循能力和推进代码智能边界方面的有效性。

AlchemistPrompts：设计为协调多源数据中的固有冲突并在精细级别上缓解指令/响应错位的数据特定提示。

代码理解任务：来源为数据构建过程，包括指令演变、数据过滤和代码审查。

协调的多源数据：200M标记的指令调整，包括6种高质量数据。

卓越的模型性能：超越所有同一大小（6.7/7B）的开源模型，并在6个代码基准测试中与更大的模型（15B/33B/70B/ChatGPT）相媲美甚至 beat

高级通用能力：通过在MMLU、BBH和GSM8K上的显著改进展示。

**Model Summary:**
AlchemistCoder is a series of coding models by InternLM.
This model is tuned from Llama 2, and should excel at all coding related tasks.

**Highlights**

**Abstract**: Open-source Large Language Models (LLMs) and their specialized variants, particularly Code LLMs, have recently delivered impressive performance. However, previous Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained Code LLMs. In this paper, we present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-response pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B) and rivals or even surpasses larger models (15B/33B/70B), showcasing the efficacy of our method in refining instruction-following capabilities and advancing the boundaries of code intelligence.

AlchemistPrompts: Designed as data-specific prompts for harmonizing inherent conflicts in multi-source data and mitigating the instruction/response misalignment at a fined-grained level.

Code Comprehension Tasks: Sourced from the process of data construction, consisting of instruction evolution, data filtering, and code review.

Harmonized Multi-source Data: Instruction tuned on 200M tokens, including 6 types of high-quality data.

Superior Model Performance: Surpassing all the open-source models of the same size (6.7/7B), and rivaling or even beating larger models (15B/33B/70B/ChatGPT) on 6 code benchmarks.

Advanced generic capabilities: Demonstrated by the significant improvements on MMLU, BBH, and GSM8K.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)