Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Algorithm] RLHF end-to-end, clean #1597

Merged
merged 52 commits into from
Oct 5, 2023
Merged

[Algorithm] RLHF end-to-end, clean #1597

merged 52 commits into from
Oct 5, 2023

Commits on Jun 27, 2023

  1. RLHF end2end example

    @apbard
    apbard committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    f24281c View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. add VmapModule and from_lmhead_model method

    @apbard
    apbard committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    ef3f76f View commit details
    Browse the repository at this point in the history

  2. Update examples/rlhf/train_rlhf.py 

    Co-authored-by: Vincent Moens <[email protected]>
    @apbard @vmoens
    apbard and vmoens authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    02a909b View commit details
    Browse the repository at this point in the history

  3. addressing comments

    @apbard
    apbard committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    953e4af View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into rlhf-networks

    @vmoens
    vmoens committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    ffb8661 View commit details
    Browse the repository at this point in the history

  5. Update torchrl/modules/tensordict_module/common.py

    @vmoens
    vmoens authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    f43faea View commit details
    Browse the repository at this point in the history

  6. Update torchrl/modules/tensordict_module/actors.py

    @vmoens
    vmoens authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    69b0588 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. Add RolloutFromModel class

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    b6fecbb View commit details
    Browse the repository at this point in the history

  2. Add rollout tests

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    bd8fbb6 View commit details
    Browse the repository at this point in the history

  3. Apply suggestions from code review 

    Co-authored-by: Alessandro Pietro Bardelli <[email protected]>
    @tcbegley @apbard
    tcbegley and apbard committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    6fbb603 View commit details
    Browse the repository at this point in the history

  4. Address comments

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    3e80a55 View commit details
    Browse the repository at this point in the history

  5. Docstring lint

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    385ac90 View commit details
    Browse the repository at this point in the history

  6. Apply suggestions from code review 

    Co-authored-by: Vincent Moens <[email protected]>
    @tcbegley @vmoens
    tcbegley and vmoens committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    8d0a152 View commit details
    Browse the repository at this point in the history

  7. Address comments

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    fcddc97 View commit details
    Browse the repository at this point in the history

  8. Fix tests

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    5c7c72e View commit details
    Browse the repository at this point in the history

  9. Handle missing transformers import

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    92d5757 View commit details
    Browse the repository at this point in the history

  10. Import transformers locally

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    eec0eaf View commit details
    Browse the repository at this point in the history

  11. lint

    @vmoens @tcbegley
    vmoens authored and tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    87501ea View commit details
    Browse the repository at this point in the history

  12. Merge branch 'rlhf-rollout' into rlhf-example

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    043fcf6 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'rlhf-networks' into rlhf-example

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    3f53046 View commit details
    Browse the repository at this point in the history

  14. lint

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    8b69e41 View commit details
    Browse the repository at this point in the history

  15. Example bugfixes

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    24eaa3a View commit details
    Browse the repository at this point in the history

  16. Move KL controller logic

    @tcbegley
    tcbegley committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    fba43a1 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Merge branch 'main' into rlhf-example 

    # Conflicts:
#	test/test_rlhf.py
#	torchrl/data/rlhf/utils.py
#	torchrl/modules/tensordict_module/actors.py
#	torchrl/modules/tensordict_module/common.py
    @vmoens
    vmoens committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    20fa920 View commit details
    Browse the repository at this point in the history

  2. amend

    @vmoens
    vmoens committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    c07ac93 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. addressing comments about klcontroller

    @apbard
    apbard committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    f463e0e View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. Merge remote-tracking branch 'origin/main' into rlhf-example

    @vmoens
    vmoens committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    eac5374 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2023

  1. Merge remote-tracking branch 'origin/main' into rlhf-example

    @vmoens
    vmoens committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    8d2dde7 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. Merge branch 'main' into rlhf-example

    @vmoens
    vmoens committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    a2ba045 View commit details
    Browse the repository at this point in the history

  2. amend

    @vmoens
    vmoens committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    a9b94f0 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. init

    @vmoens
    vmoens committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    d983ebd View commit details
    Browse the repository at this point in the history

  2. readme

    @vmoens
    vmoens committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    097c443 View commit details
    Browse the repository at this point in the history

  3. amend

    @vmoens
    vmoens committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    0efd93a View commit details
    Browse the repository at this point in the history

  4. amend

    @vmoens
    vmoens committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    fba9f03 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. amend

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    cc535e5 View commit details
    Browse the repository at this point in the history

  2. amend

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    28c116f View commit details
    Browse the repository at this point in the history

  3. amend

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    0f128a6 View commit details
    Browse the repository at this point in the history

  4. amend

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    e0ad043 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/main' into rlhf-example-refactor

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    e8cad9b View commit details
    Browse the repository at this point in the history

  6. amend

    @vmoens
    vmoens committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    c93c134 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2023

  1. init

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    56f7597 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refactor_ddpg_loss' into rlhf-example-refactor

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    3fa6ea5 View commit details
    Browse the repository at this point in the history

  3. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    c1c41dc View commit details
    Browse the repository at this point in the history

  4. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    880e5b4 View commit details
    Browse the repository at this point in the history

  5. Update run_test.sh

    @vmoens
    vmoens authored Oct 5, 2023
    Configuration menu
    Copy the full SHA
    d36ce77 View commit details
    Browse the repository at this point in the history

  6. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    942b311 View commit details
    Browse the repository at this point in the history

  7. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    fca9f7b View commit details
    Browse the repository at this point in the history

  8. lint

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    6362715 View commit details
    Browse the repository at this point in the history

  9. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    e3b2d4f View commit details
    Browse the repository at this point in the history

  10. amend

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    7918f86 View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'origin/main' into rlhf-example-refactor

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    9658a44 View commit details
    Browse the repository at this point in the history

  12. lint

    @vmoens
    vmoens committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    eb041a4 View commit details
    Browse the repository at this point in the history