You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe: Sophia is a fairly new optimization algorithm for training language models and boasts some substantial improvements (2x speed up over Adam in wall-clock time in GPT-2 tests). Sophia seems like a nice addition to Fairseq2.
Describe the solution you would like:
It would be nice to offer Sophia as a finetuning recipe invoked with something like...
Is your feature request related to a problem? Please describe:
Sophia is a fairly new optimization algorithm for training language models and boasts some substantial improvements (2x speed up over Adam in wall-clock time in GPT-2 tests). Sophia seems like a nice addition to Fairseq2.
Describe the solution you would like:
It would be nice to offer Sophia as a finetuning recipe invoked with something like...
Describe the alternatives you have considered:
AdamW already exists as part of Fairseq2 as the default optimization algorithm.
Additional Context:
None
The text was updated successfully, but these errors were encountered: