mannix / alchemistcoder-7b

摘要：开源大规模语言模型（LLMs）及其专用变体，特别是代码LLMs，最近展现出令人印象深刻的性能。然而，以前的代码LLMs通常在单一来源数据上微调，这些数据的质量和多样性有限，可能无法充分调动预训练代码LLMs的潜力。在这篇论文中，我们介绍了AlchemistCoder，这是一个系列代码LLMs，通过在多云集数据上微调，提高了代码生成和泛化能力。为此，我们率先揭示多云集代码语料库中各种风格和质量的内在冲突，并引入了具有事后标注的数据特定提示，称为AlchemistPrompts，以协调不同的数据源和指令-响应对。此外，我们将数据构建过程纳入微调数据作为代码理解任务，包括指令进化、数据过滤和代码审查。大量实验表明，AlchemistCoder在各种相同大小（6.7B/7B）的模型中占据明显优势，并与更大模型（15B/33B/70B）相当甚至超越，展示了我们在改善指令跟随能力和推进代码智能边界方面的方法有效性。

AlchemistPrompts：设计为特定于数据提示，以协调多云集数据中的内在冲突并在细粒度级别减轻指令/响应不匹配。

代码理解任务：来自数据构建流程，包括指令进化、数据过滤和代码审查。

协调多云集数据：在包含6种高质量数据的200M令牌上微调的指令。

卓越的模型性能：超越所有相同大小（6.7/7B）的开源模型，并在6个代码基准测试中与更大模型（15B/33B/70B/ChatGPT）相当甚至击败。

高级通用能力：通过在MMLU、BBH和GSM8K上的显著改进而得到证明。

**Model Summary:**
AlchemistCoder is a series of coding models by InternLM.
This model is tuned from Llama 2, and should excel at all coding related tasks.

**Highlights**

**Abstract**: Open-source Large Language Models (LLMs) and their specialized variants, particularly Code LLMs, have recently delivered impressive performance. However, previous Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained Code LLMs. In this paper, we present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-response pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B) and rivals or even surpasses larger models (15B/33B/70B), showcasing the efficacy of our method in refining instruction-following capabilities and advancing the boundaries of code intelligence.

AlchemistPrompts: Designed as data-specific prompts for harmonizing inherent conflicts in multi-source data and mitigating the instruction/response misalignment at a fined-grained level.

Code Comprehension Tasks: Sourced from the process of data construction, consisting of instruction evolution, data filtering, and code review.

Harmonized Multi-source Data: Instruction tuned on 200M tokens, including 6 types of high-quality data.

Superior Model Performance: Surpassing all the open-source models of the same size (6.7/7B), and rivaling or even beating larger models (15B/33B/70B/ChatGPT) on 6 code benchmarks.

Advanced generic capabilities: Demonstrated by the significant improvements on MMLU, BBH, and GSM8K.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)