Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Major refactor to support new architectures #261

Draft
wants to merge 53 commits into
base: main
Choose a base branch
from
Draft

Major refactor to support new architectures #261

wants to merge 53 commits into from

Commits on Aug 25, 2023

  1. deepspeed running

    @anas-awadalla
    anas-awadalla committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    e19133d View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2023

  1. more progress

    @anas-awadalla
    anas-awadalla committed Aug 26, 2023
    Configuration menu
    Copy the full SHA
    870f20c View commit details
    Browse the repository at this point in the history

  2. added ds checkpointing

    @anas-awadalla
    anas-awadalla committed Aug 26, 2023
    Configuration menu
    Copy the full SHA
    f9162a0 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. more progress

    @anas-awadalla
    anas-awadalla committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    ded3485 View commit details
    Browse the repository at this point in the history

  2. mllm

    Anas Awadalla committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    3672042 View commit details
    Browse the repository at this point in the history

  3. merge deepspeed

    @anas-awadalla
    anas-awadalla committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    99c350f View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. rewrite src: add VLM, Kosmos, Flamingo

    @i-gao
    i-gao committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    2f634f0 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. fix kosmos models

    @i-gao
    i-gao committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    7261639 View commit details
    Browse the repository at this point in the history

  2. cosmetic: num_params helper fn

    @i-gao
    i-gao committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    09977ba View commit details
    Browse the repository at this point in the history

  3. revert to deepspeed branch code for train/

    @i-gao
    i-gao committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    6bb9071 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. add BLIP

    @i-gao
    i-gao committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    7984adb View commit details
    Browse the repository at this point in the history

  2. minor train script fixes

    @i-gao
    i-gao committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    7eab26a View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. fix vocab len issues

    @i-gao
    i-gao committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    aed0f21 View commit details
    Browse the repository at this point in the history

  2. fixes

    @i-gao
    i-gao committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    47c8e19 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. big refactor of training code

    @i-gao
    i-gao committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    11ab894 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2023

  1. many fixes + rewrite FSDP for torch nightly

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    cd4f3aa View commit details
    Browse the repository at this point in the history

  2. fixes

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    74686a7 View commit details
    Browse the repository at this point in the history

  3. fixes

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    61f5a3d View commit details
    Browse the repository at this point in the history

  4. run linter & fix gradient ckpting

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    ccfcb0f View commit details
    Browse the repository at this point in the history

  5. no need to untie embeddings for fsdp

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    303e707 View commit details
    Browse the repository at this point in the history

  6. add in missing kwarg

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    fc660e7 View commit details
    Browse the repository at this point in the history

  7. Merge branch deepspeed: eval code only

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    be9a4dd View commit details
    Browse the repository at this point in the history

  8. update eval code to match new src args

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    b0ff9a4 View commit details
    Browse the repository at this point in the history

  9. update documentation and example scripts

    @i-gao
    i-gao committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    92bc4b7 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2023

  1. fix deepspeed train script

    @anas-awadalla
    anas-awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    60a82d7 View commit details
    Browse the repository at this point in the history

  2. removed non default loss scale window

    @anas-awadalla
    anas-awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    82d1c69 View commit details
    Browse the repository at this point in the history

  3. init flamingo embeds new weights

    @anas-awadalla
    anas-awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    4875822 View commit details
    Browse the repository at this point in the history

  4. init flamingo embeds new weights

    @anas-awadalla
    anas-awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    8f2f040 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into mllm

    @anas-awadalla
    anas-awadalla authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    beba4d2 View commit details
    Browse the repository at this point in the history

  6. fix mmc4 sim threshold arg

    @anas-awadalla
    anas-awadalla authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    b81379f View commit details
    Browse the repository at this point in the history

  7. add z-loss

    @anas-awadalla
    anas-awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    f91c14a View commit details
    Browse the repository at this point in the history

  8. Merge pull request #262 from mlfoundations/add-z-loss 

    add z-loss
    @anas-awadalla
    anas-awadalla authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    df96979 View commit details
    Browse the repository at this point in the history

  9. Update eval README.md

    @i-gao
    i-gao authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    bcc5a8f View commit details
    Browse the repository at this point in the history

  10. have a default stdev for init

    Anas Awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    770e653 View commit details
    Browse the repository at this point in the history

  11. Update run_train_deepspeed.sh

    @anas-awadalla
    anas-awadalla authored Sep 17, 2023
    Configuration menu
    Copy the full SHA
    ef268be View commit details
    Browse the repository at this point in the history

  12. fix loss impl and model vocab size

    Anas Awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    da07e35 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'mllm' of https://github.com/mlfoundations/open_flamingo … 

    …into mllm
    Anas Awadalla committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    3fcda82 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. remove ds act checkpointing exception

    Anas Awadalla committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    bcd2cf5 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. fixes from PR review

    @i-gao
    i-gao committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    9b1a764 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mllm' of github.com:mlfoundations/open_flamingo into mllm

    @i-gao
    i-gao committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    866a780 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. add weight/bias init to decouple linear

    @anas-awadalla
    anas-awadalla committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    5ad05c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Language stream changes (#264) 

    * fix padding side when generating

* clean up language stream forward pass (less for looping)

* expose BLIP model

* fixes for forward pass without images

* restore for looping
    @anas-awadalla
    anas-awadalla authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    939d460 View commit details
    Browse the repository at this point in the history

  2. grad checkpointing + ds saving patch (we should find a cleaner solution)

    @anas-awadalla
    anas-awadalla committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    ae76178 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. Update run_train_deepspeed.sh

    @anas-awadalla
    anas-awadalla authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    d29c8b8 View commit details
    Browse the repository at this point in the history

  2. clearer parameter count logging

    @anas-awadalla
    anas-awadalla authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    b7af1d6 View commit details
    Browse the repository at this point in the history

  3. Fix model vocab size (now it is len of tokenizer)

    @anas-awadalla
    anas-awadalla authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    43ac961 View commit details
    Browse the repository at this point in the history

  4. Update code example

    @anas-awadalla
    anas-awadalla authored Oct 18, 2023
    Configuration menu
    Copy the full SHA
    e7684b5 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. fix LR schedule

    @anas-awadalla
    anas-awadalla authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    735a880 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. fix var naming in load_deepspeed_checkpoint

    @anas-awadalla
    anas-awadalla authored Oct 24, 2023
    Configuration menu
    Copy the full SHA
    496e656 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. Update losses.py

    @anas-awadalla
    anas-awadalla authored Nov 30, 2023
    Configuration menu
    Copy the full SHA
    c5feb97 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2023

  1. train_utils media token fix

    Oscar Lo committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    dbb1ad8 View commit details
    Browse the repository at this point in the history

  2. remove unnecessary model unwrap lines

    Oscar Lo committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    fa6af69 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #283 from mlfoundations/media_token_fix 

    train_utils media token fix
    @anas-awadalla
    anas-awadalla authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    eb6b8aa View commit details
    Browse the repository at this point in the history