Merge branch 'main' into release/1.1

* main: fix internlm-20b-chat model_id (#88) Add internlm agent (#86) add mm_notebook (#85) Replace with loralib and add unload lora interface (#83) Add sphinx doc builder (#77) add notebook (#81) update feat: merge lora (#82) fix unpatch_lora (#80) Fix bug: LoRA not work with diffusers>0.20.0 (#73) update readme (#76) Feat 0919 (#78) fix bug: internlm-20b (#75) Fix bug 0917 (#74) update publish workflow bump version to 1.1.0
modelscope · Sep 22, 2023 · f82100d · f82100d
2 parents 331e2c5 + 11383fe
commit f82100d
Show file tree

Hide file tree

Showing 108 changed files with 2,750 additions and 1,710 deletions.
diff --git a/.dev_scripts/build_docs.sh b/.dev_scripts/build_docs.sh
@@ -0,0 +1,8 @@
+pip install -r requirements/docs.txt
+cd docs
+rm -rf build
+
+# update api rst
+#rm -rf source/api/
+#sphinx-apidoc --module-first -o source/api/ ../modelscope/
+make html
diff --git a/.gitignore b/.gitignore
@@ -124,7 +124,7 @@ replace.sh
 result.png
 result.jpg
 result.mp4
-runs/
+output/
 *.out
 
 # Pytorch

diff --git a/README.md b/README.md
@@ -14,7 +14,7 @@
 
 # Introduction
 
-SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning. It integrates implementations for various efficient fine-tuning methods,  by embracing approaches that is parameter-efficient, memory-efficient, and time-efficient. SWIFT integrates seamlessly into ModelScope ecosystem and offers the capabilities to finetune various modles, with a primary emphasis on LLMs and vision models. Additionally, SWIFT is fully compatible with [Peft](https://github.com/huggingface/peft), enabling users to  leverage the familiar Peft interface to finetune ModelScope models.
+SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference. It integrates implementations for various efficient fine-tuning methods,  by embracing approaches that is parameter-efficient, memory-efficient, and time-efficient. SWIFT integrates seamlessly into ModelScope ecosystem and offers the capabilities to finetune various models, with a primary emphasis on LLMs and vision models. Additionally, SWIFT is fully compatible with [PEFT](https://github.com/huggingface/peft), enabling users to  leverage the familiar Peft interface to finetune ModelScope models.
 
 Currently supported approches (and counting):
 
@@ -23,77 +23,74 @@ Currently supported approches (and counting):
 3. Prompt Tuning: [Visual Prompt Tuning](https://arxiv.org/abs/2203.12119)
 4. Side: [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://arxiv.org/abs/1912.13503)
 5. ResTuning-Bypass
-7. All tuners offered on [Peft](https://github.com/huggingface/peft)
+7. All tuners offered on [PEFT](https://github.com/huggingface/peft)
 
 Key features:
 
 1. By integrating the ModelScope library, models can be readily obatined via a model-id.
-2. Tuners provided by SWIFT be combined together to allow exploration of multiple tuners on a model for best result.
-3. Support calling `activate_adapter`或`deactivate_adapter` to activate/deactivate a single tuner. User can use one model with multiple tuners in different threads.
+2. Tuners provided by SWIFT can be combined together to allow exploration of multiple tuners on a model for best result.
+3. Support calling `activate_adapter` or `deactivate_adapter` or `set_active_adapters`  to activate/deactivate tuners. User can inference with one model and multiple tuners in different threads independently.
 
-Users can check the [documentation of Swift](./docs/Get Started/1.Introduction.md) to get detail tutorials.
+Users can check the [documentation of Swift](docs/source/GetStarted/Introduction.md) to get detail tutorials.
 
 ## LLM SFT Example
 [code link](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm)
 
-1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning)
+1. supported SFT methods: [LoRA](https://arxiv.org/abs/2106.09685), [QLoRA](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning)
 2. supported models:
    1. qwen series: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B)
    2. qwen-vl series: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL)
-   3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat
-   4. chatglm2 series: chatglm2-6b, chatglm2-6b-32k
-   5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat
-   6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b
-   7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k
-   8. other: polylm-13b, seqgpt-560m
+   3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, [baichuan2-7b-chat](https://modelscope.cn/models/baichuan-inc/Baichuan2-7B-Chat/summary), baichuan2-13b, baichuan2-13b-chat
+   4. chatglm2 series: [chatglm2-6b](https://modelscope.cn/models/ZhipuAI/chatglm2-6b/summary), chatglm2-6b-32k
+   5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, [llama2-70b-chat](https://modelscope.cn/models/modelscope/Llama-2-70b-chat-ms/summary)
+   6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, [openbuddy-llama2-70b](https://modelscope.cn/models/OpenBuddy/openbuddy-llama2-70b-v10.1-bf16/summary)
+   7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k, [internlm-20b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-20b/summary), [internlm-20b-chat](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-chat-20b/summary)
+   8. other: [polylm-13b](https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation/summary), [seqgpt-560m](https://modelscope.cn/models/damo/nlp_seqgpt-560m/summary)
 3. supported features: quantization, DDP, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ...
 4. supported datasets:
-   1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh, jd-zh, dureader-robust-zh, medical-en, medical-zh, medical-mini-zh, sharegpt-en, sharegpt-zh
+   1. NLP: [alpaca-en](https://modelscope.cn/datasets/AI-ModelScope/alpaca-gpt4-data-en/summary)(gpt4), [alpaca-zh](https://modelscope.cn/datasets/AI-ModelScope/alpaca-gpt4-data-zh/summary)(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh, [jd-zh](https://modelscope.cn/datasets/DAMO_NLP/jd/summary), [dureader-robust-zh](https://modelscope.cn/datasets/modelscope/DuReader_robust-QG/summary), medical-en, medical-zh, medical-mini-zh, sharegpt-en, sharegpt-zh, [code-python-zh](https://modelscope.cn/datasets/codefuse-ai/CodeExercise-Python-27k/summary), [advertise-gen](https://modelscope.cn/datasets/lvjianjin/AdvertiseGen/summary)
    2. agent: [damo-agent-zh](https://modelscope.cn/datasets/damo/MSAgent-Bench/summary), damo-agent-mini-zh
-   3. multi-modal: coco-en
-   4. other: cls-fudan-news-zh, ner-jave-zh
+   3. multi-modal: [coco-en](https://modelscope.cn/datasets/modelscope/coco_2014_caption/summary)
+   4. other: [cls-fudan-news-zh](https://modelscope.cn/datasets/damo/zh_cls_fudan-news/files), [ner-jave-zh](https://modelscope.cn/datasets/damo/zh_ner-JAVE/summary)
 5. supported templates: chatml(qwen), baichuan, chatglm2, llama, openbuddy-llama, default, default-generation
 
 # Installation
 
 SWIFT is running in Python environment. Please make sure your python version is higher than 3.8.
 
-Please install SWIFT by the `pip` command:
+- Install SWIFT by the `pip` command:
 
 ```shell
 pip install ms-swift -U
 ```
 
-If you want to install SWIFT by source code, please run:
+- Install SWIFT by source code(for running sft/infer examples), please run:
 
 ```shell
 git clone https://github.com/modelscope/swift.git
 cd swift
 pip install -e .
 ```
 
-If you are using source code, please remember install requirements by:
-```shell
-pip install -r requirements/framework.txt
-```
-
 SWIFT requires torch>=1.13.
 
-We also recommend to use SWIFT in our docker image:
+- Use SWIFT in our docker image:
+
 ```shell
-docker pull registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.7.1-py38-torch2.0.1-tf1.15.5-1.8.0
+docker pull registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.8.0-py38-torch2.0.1-tf2.13.0-1.9.1
 ```
 
 # Getting Started
 
-SWIFT supports multiple tuners, as well as tuners provided by [Peft](https://github.com/huggingface/peft). To use the these tuners, simply call:
+SWIFT supports multiple tuners, as well as tuners provided by [PEFT](https://github.com/huggingface/peft). To use these tuners, simply call:
 
 ```python
-from swift import Swift
+from swift import Swift, LoRAConfig
+config = LoRAConfig(...)
 model = Swift.prepare_model(model, config, extra_state_keys=['...'])
 ```
 
-The code snippet above initialized the tuner randomly. The input model is an instance of `torch.nn.Module`, config is a subclass instance of `SwiftConfig` or `PeftConfig`. extra_state_keys is
+The code snippet above initialized the tuner randomly. The input model is an instance of `torch.nn.Module`, the config is a subclass instance of `SwiftConfig` or `PeftConfig`. extra_state_keys is
 the extra module weights(like the linear head) to be trained and stored in the output dir.
 
 You may combine multiple tuners by:
@@ -103,7 +100,7 @@ from swift import Swift, LoRAConfig, PromptConfig
 model = Swift.prepare_model(model, {'lora': LoRAConfig(...), 'prompt': PromptConfig(...)})
 ```
 
-You can all `save_pretrained` and `push_to_hub` after finetuning:
+Call `save_pretrained` and `push_to_hub` after finetuning:
 
 ```python
 from swift import push_to_hub
@@ -199,20 +196,8 @@ model_wrapped = Swift.prepare_model(model, lora_config)
 model_wrapped = Swift.from_pretrained(model, 'some-id-in-the-modelscope-modelhub')
 ```
 
-or:
-
-```python
-from swift import LoraConfig, get_peft_model, PeftModel
-from peft import TaskType
-lora_config = LoraConfig(target_modules=['query', 'key', 'value'], task_type=TaskType.CAUSAL_LM)
-model_wrapped = get_peft_model(model, lora_config)
-
-# or call from_pretrained to load weights in the modelhub
-model_wrapped = PeftModel.from_pretrained(model, 'some-id-in-the-modelscope-modelhub')
-```
-
 
-The saving strategy between Swift tuners and Peft tuners are slightly different. You can name a tuner of a SWIFT by:
+The saving strategy between Swift tuners and Peft tuners are slightly different. You can name a tuner by:
 
 ```python
 model = Swift.prepare_model(model, {'default': LoRAConfig(...)})
@@ -230,7 +215,7 @@ output
     |-- adapter_model.bin
 ```
 
-The config/weights stored in the output dir is the config of `extra_state_keys` and the weights of it. This is different from Peft, which stores the weights and config of the `default` tuner.
+The config/weights stored in the output dir is the config of `extra_state_keys` and the weights of it. This is different from PEFT, which stores the weights and config of the `default` tuner.
 
 
 # Learn More