Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Sophia optimizer #852

Open
PeanutButterRat opened this issue Nov 11, 2024 · 0 comments · May be fixed by #844
Open

Add Sophia optimizer #852

PeanutButterRat opened this issue Nov 11, 2024 · 0 comments · May be fixed by #844
Labels
enhancement New feature or request

Comments

@PeanutButterRat
Copy link
Contributor

Is your feature request related to a problem? Please describe:
Sophia is a fairly new optimization algorithm for training language models and boasts some substantial improvements (2x speed up over Adam in wall-clock time in GPT-2 tests). Sophia seems like a nice addition to Fairseq2.

Describe the solution you would like:
It would be nice to offer Sophia as a finetuning recipe invoked with something like...

fairseq2 lm instruction_finetune --preset llama3_2_1b_instruct_sophiag

Describe the alternatives you have considered:
AdamW already exists as part of Fairseq2 as the default optimization algorithm.

Additional Context:
None

@PeanutButterRat PeanutButterRat added the enhancement New feature or request label Nov 11, 2024
@PeanutButterRat PeanutButterRat linked a pull request Nov 11, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant