Hugging Face Blog·· 2024-08-21AI 评分40
Hugging Face 用 Flash Attention 2 与打包训练将吞吐提升至 2 倍
Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2
AI 导读
Hugging Face Transformers 新增 DataCollatorWithFlattening，使无填充的打包指令微调与 Flash Attention 2 兼容，训练吞吐最高提升 2 倍且不损失收敛质量。
来源：Hugging Face Blog · huggingface.co