IDEA-CCNL MindBot 封神榜大模型

由国内IDEA研究院CCNL技术团队创建封神榜大模型，用人类的语言跟机器的智能直接对话

通过大规模预训练、海量任务有监督微调和人类反馈学习，聚合海量知识和Al能力，让Al透过自然语言触达每个人。

为了让大家更便利、高效地运用“封神榜”系列大模型，参与继续训练和下游应用，近日，IDEA研究院CCNL封神榜团队开源了封神框架（FengShen）。

FengShen可以应用在基于海量数据的大模型训练，以及各种下游任务微调，其中数据规模支持TB级别，模型参数可达百亿。用户通过自主配置的方式进行分布式训练，不仅节省显存，还可以让个人精力更加聚焦在模型实现和创新。同时，为了方便领域间的模型迁移，FengShen也支持用户直接使用Huggingface/Transformers中的模型结构继续训练。

换言之，通过FengShen框架，你可以享受：

1.比原生torch更强的性能，训练速度提升300%

2.支持更大模型，支持百亿级别内模型训练与微调

3.支持万亿级以上数据集，在家用主机上感受预训练模型带来的效果提升

4.丰富的预训练、下游任务示例

5.适应各种设备环境，支持在CPU、GPU、TPU等不同设备上运行

6.集成主流的分布式训练逻辑，无需修改代码即可支持DDP、Zero Optimizer等分布式优化技术

目前“封神榜”上32个的预训练模型，涵盖了中文自然语言推理、情感分类、文本相似度计算、语义纠错、文本摘要、续写、问答等多个任务，包括BERT、GPT、T5、BART、Transformer-XL等经典模型结构的模型，最大的模型规模达到了35亿参数。

CCNL technology team of the domestic IDEA Research Institute creates a large model of the Pantheon, and uses human language to talk directly with machine intelligence

Through large-scale pre-training, mass tasks, supervision and fine-tuning, and human feedback learning, mass knowledge and Al ability are aggregated, so that Al can reach everyone through natural language.

In order to make everyone more convenient and efficient use of the “Sacred List” series of large models, participate in continued training and downstream application, recently, IDEA Research Institute CCNL Sacred List team open source framework (FengShen).

FengShen can be applied in large model training based on massive data, as well as fine-tuning of various downstream tasks. The data scale supports TB level, and the model parameters can reach tens of billions. Users conduct distributed training through self-configuration, which not only saves video memory, but also allows them to focus more on model implementation and innovation. At the same time, in order to facilitate model migration between fields, FengShen also supports users to directly use the model structure in Huggingface/Transformers to continue training.

In other words, with the FengShen frame, you can enjoy:

1. Stronger performance than the original torch, training speed increased by 300%
2. Support larger models, support model training and fine tuning within 10 billion level
3. Support trillion-level data sets, feel the effect of pre-training model on the home host
4. Rich examples of pre-training and downstream tasks
5. Adapt to various device environments and support different devices such as cpus, Gpus, and Tpus
6. Integrate mainstream distributed training logic and support DDP, Zero Optimizer and other distributed optimization technologies without modifying the code

At present, 32 pre-training models on the “Canonism list” cover multiple tasks such as Chinese natural language reasoning, emotion classification, text similarity calculation, semantic error correction, text summary, continuation, question and answer, including BERT, GPT, T5, BART, Transformer-XL and other classic model structure models. The largest model size reached 3.5 billion parameters.