About Lora finetuning

[cpystan]

您好，当我对vit加入lora，微调模型，训练会报如下错误：
File "Monkey-main/monkey_model/modeling_qwen.py", line 816, in forward
hidden_states[i][a + 1 : b] = images[idx]
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

可以提供一些思路和建议吗
谢谢

[echo840]

您好，您是使用file 进行训练吗？

[cpystan]

是的

[echo840]

Hello, here is my environment. You might need to check whether the versions of packages like DeepSpeed are correct.

Package                       Version
----------------------------- ------------
absl-py                       2.0.0
accelerate                    0.23.0
aiofiles                      23.2.1
altair                        5.1.2
annotated-types               0.6.0
anyio                         3.7.1
asttokens                     2.4.0
attrs                         23.1.0
backcall                      0.2.0
cachetools                    5.3.1
certifi                       2023.7.22
charset-normalizer            3.3.0
click                         8.1.7
colorama                      0.4.6
contourpy                     1.1.1
cycler                        0.12.1
decorator                     5.1.1
deepspeed                     0.11.1
einops                        0.7.0
einops-exts                   0.0.4
et-xmlfile                    1.1.0
exceptiongroup                1.1.3
executing                     2.0.0
fastapi                       0.104.1
ffmpy                         0.3.1
filelock                      3.12.4
flash-attn                    2.3.3
fonttools                     4.43.1
fsspec                        2023.9.2
google-auth                   2.23.2
google-auth-oauthlib          1.0.0
gradio                        4.1.2
gradio_client                 0.7.0
grpcio                        1.59.0
h11                           0.14.0
hjson                         3.1.0
httpcore                      1.0.1
httpx                         0.25.1
huggingface-hub               0.17.3
idna                          3.4
imagesize                     1.4.1
importlib-resources           6.1.1
inflect                       7.0.0
ipdb                          0.13.13
ipython                       8.16.1
jedi                          0.19.1
Jinja2                        3.1.2
joblib                        1.3.2
jsonschema                    4.19.2
jsonschema-specifications     2023.7.1
kiwisolver                    1.4.5
Markdown                      3.5
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
matplotlib                    3.8.0
matplotlib-inline             0.1.6
mdurl                         0.1.2
mpmath                        1.3.0
networkx                      3.1
ninja                         1.11.1.1
nltk                          3.8.1
numpy                         1.26.0
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu12              2.18.1
nvidia-nvjitlink-cu12         12.2.140
nvidia-nvtx-cu12              12.1.105
oauthlib                      3.2.2
openpyxl                      3.1.2
orjson                        3.9.10
packaging                     23.2
pandas                        2.1.2
parso                         0.8.3
peft                          0.5.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        10.0.1
pip                           23.2.1
prompt-toolkit                3.0.39
protobuf                      4.24.4
psutil                        5.9.5
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyasn1                        0.5.0
pyasn1-modules                0.3.0
pycocoevalcap                 1.2
pycocotools                   2.0
pydantic                      1.10.13
pydantic_core                 2.10.1
pydub                         0.25.1
Pygments                      2.16.1
pyparsing                     3.1.1
python-dateutil               2.8.2
python-multipart              0.0.6
pytz                          2023.3.post1
PyYAML                        6.0.1
referencing                   0.30.2
regex                         2023.10.3
requests                      2.31.0
requests-oauthlib             1.3.1
rich                          13.6.0
rpds-py                       0.12.0
rsa                           4.9
safetensors                   0.4.0
scikit-learn                  1.3.2
scipy                         1.11.3
seaborn                       0.13.2
semantic-version              2.10.0
setuptools                    68.0.0
shellingham                   1.5.4
six                           1.16.0
sklearn                       0.0.post11
sniffio                       1.3.0
stack-data                    0.6.3
starlette                     0.27.0
sty                           1.0.6
sympy                         1.12
tabulate                      0.9.0
tensorboard                   2.14.1
tensorboard-data-server       0.7.1
textdistance                  4.6.1
threadpoolctl                 3.2.0
tiktoken                      0.5.1
timm                          0.9.8
tokenizers                    0.13.3
tomli                         2.0.1
tomlkit                       0.12.0
toolz                         0.12.0
torch                         2.1.0
torchvision                   0.16.0
tqdm                          4.66.1
traitlets                     5.11.2
transformers                  4.32.0
transformers-stream-generator 0.0.4
triton                        2.1.0
typer                         0.9.0
typing_extensions             4.8.0
tzdata                        2023.3
urllib3                       2.0.6
uvicorn                       0.24.0.post1
wcwidth                       0.2.8
websockets                    11.0.3
Werkzeug                      3.0.0
wheel                         0.41.2

[cpystan]

您好 我的deepspeed版本是0.14.0。 另外我注意到之前的issue里也有人遇到了和我一样的问题：#83