mannix / phi3-mini-cpo-simpo

Phi-3-mini-4K instructors配合 CPO-SimPO

25 Pulls 更新于6周前

更新于6周前

6周前

c7ac3e0408ff · 2.6GB

读我

Phi-3-mini-4K instructors配合 CPO-SimPO

使用 i-matrix 校准的量子化
将 Saftensors 转换为 fp32

此存储库包含使用 CPO-SimPO 技术增强的 Phi-3-mini-4K-instruct 模型。CPO-SimPO 结合了对比偏好优化 (CPO) 和简单偏好优化 (SimPO)。

介绍

Phi-3-mini-4K-instruct 是针对基于指令的任务进行优化的模型。这种方法在关键基准测试中表现出显著改进，推动了 AI 偏好学习的发展边界。

什么是 CPO-SimPO？

CPO-SimPO 是一种新技术，它结合了 CPO 和 SimPO 的元素

对比偏好优化 (CPO): 添加行为克隆正则化器，以确保模型保持接近理想数据分布。
简单偏好优化 (SimPO): 集成了长度归一化和目标奖励范围，以防止生成长但质量低下的序列。

Github

CPO-SIMPO

模型性能

即将推出！

主要改进

增强模型性能: 显著提升了分数，特别是在 GSM8K（提升8.49分！）和 TruthfulQA（提升2.07分）。
质量控制: 通过长度归一化和奖励范围改进了高质序列的生成。
平衡优化: BC正则化器帮助维持学习偏好的完整性，同时不离谱于理想数据分布。

# Phi-3-mini-4K-instruct with CPO-SimPO

- Quantizations with i-matrix `calibration_datav3.txt`
- Saftensors converted to fp32

This repository contains the Phi-3-mini-4K-instruct model enhanced with the CPO-SimPO technique. CPO-SimPO combines Contrastive Preference Optimization (CPO) and Simple Preference Optimization (SimPO).

## Introduction

Phi-3-mini-4K-instruct is a model optimized for instruction-based tasks. This approach has demonstrated notable improvements in key benchmarks, pushing the boundaries of AI preference learning.

### What is CPO-SimPO?

CPO-SimPO is a novel technique, which combines elements from CPO and SimPO:

- **Contrastive Preference Optimization (CPO):** Adds a behavior cloning regularizer to ensure the model remains close to the preferred data distribution.
- **Simple Preference Optimization (SimPO):** Incorporates length normalization and target reward margins to prevent the generation of long but low-quality sequences.

### Github

**[CPO-SIMPO](https://github.com/fe1ixxu/CPO_SIMPO)**

## Model Performance

COMING SOON!

### Key Improvements:
- **Enhanced Model Performance:** Significant score improvements, particularly in GSM8K (up by 8.49 points!) and TruthfulQA (up by 2.07 points).
- **Quality Control:** Improved generation of high-quality sequences through length normalization and reward margins.
- **Balanced Optimization:** The BC regularizer helps maintain the integrity of learned preferences without deviating from the preferred data distribution.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)