opencoder

OpenCoder 是一个开放且可复现的代码 LLM 系列，包含 1.5B 和 8B 模型，支持英语和中文。 OpenCoder 从零开始，在 2.5 万亿 tokens 上进行预训练，其中 90% 为原始代码，10% 为代码相关的网络数据，并在超过 450 万个高质量 SFT 示例上进行监督微调，最终达到顶级的代码 LLM 性能。我们不仅提供模型权重和推理代码，还提供可复现的训练数据、完整的数据处理管道、严谨的实验消融结果和详细的训练协议。 OpenCoder 赋能研究人员进行构建和创新，是您推进代码 AI 的开放基础。

完全开源：OpenCoder 通过发布模型权重和即将发布的推理代码，以及完整的训练数据清理代码，确保完全透明。此版本包括高质量的合成数据、大量的检查点以及超过 450 万个监督微调 (SFT) 条目的数据集，使 OpenCoder 成为最全面的开源模型之一。
全面的实验分析：OpenCoder 通过对各种数据清理策略和训练过程（包括文件级和存储库级去重实验）进行广泛的消融研究，进行了严格的测试，确保对模型性能进行彻底的探索和验证。
高质量的合成数据：OpenCoder 提供了完全开发的合成数据生成流程和超过 450 万个 SFT 数据条目，为模型训练和评估奠定了强大的数据基础。
卓越的性能：OpenCoder 在多个语言模型基准测试中取得了高性能，使其在代码领域领先的开源模型中占有一席之地。

参考

GitHub

论文

Hugging Face

**OpenCoder** is an open and reproducible code LLM family which includes 1.5B and 8B  models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is pretrained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, and supervised finetuned on over 4.5M high-quality SFT examples, finally reaching the performance of top-tier code LLMs. We provide not only model weights and inference code, but also the reproducible training data, the complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols. Empowering researchers to build and innovate, OpenCoder is your open foundation for advancing code AI.

- **Complete Open Source**: OpenCoder ensures full transparency by releasing not only the model weights and forthcoming inference code but also the complete data-cleaning code for training. This release includes high-quality synthetic data, an extensive set of checkpoints, and a dataset of over 4.5 million supervised fine-tuning (SFT) entries, making OpenCoder one of the most comprehensively open-sourced models available.
- **Comprehensive Experimental Analysis**: OpenCoder is rigorously tested through extensive ablation studies on various data-cleaning strategies and training processes, including file-level and repository-level deduplication experiments, ensuring thorough exploration and validation of the model’s performance.
- **High-Quality Synthetic Data**: OpenCoder provides a fully developed synthetic data generation process and over 4.5 million SFT data entries, establishing a robust data foundation for model training and evaluation.
- **Exceptional Performance**: OpenCoder achieves high performance across multiple language model benchmarks, positioning it among the leading open-source models for code.

## References

[GitHub](https://github.com/OpenCoder-llm/OpenCoder-llm)

[Paper](https://arxiv.org/pdf/2411.04905)

[Hugging Face](https://hugging-face.cn/collections/infly/opencoder-672cec44bbb86c39910fb55e)

粘贴、拖放或点击上传图片 (.png, .jpeg, .jpg, .svg, .gif)