This repository has been archived by the owner on Nov 21, 2023. It is now read-only.
Fixes all current issues with FLORES V1 #40
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This command breaks without the above fix
Solved: Couldn't find file
facebook-github-bot
added
the
CLA Signed
min-lr no longer part of inverse_sqrt_lr_scheduler
update loaded fairseq model to use cfg instead of args convert corresponding args (data) into cfg (task.data)
some of the monolingual data is > 1024 tokens in length ignore that data when generating BT
mnoukhov
changed the title
guzmanhe
approved these changes
Apr 14, 2022
floresv1/download-data.sh
Outdated
| @@ -204,7 +206,7 @@ REMOVE_FILE_PATHS+=( $NE_DICT dictionaries ) | |||
|
|
|||
|
|
|||
| # Download test sets | |||
| download_data $DATA/wikipedia_en_ne_si_test_sets.tgz "https://github.com/facebookresearch/flores/raw/master/data/wikipedia_en_ne_si_test_sets.tgz" | |||
| download_data $DATA/wikipedia_en_ne_si_test_sets.tgz "https://github.com/facebookresearch/flores/raw/main/data/wikipedia_en_ne_si_test_sets.tgz" | |||
There was a problem hiding this comment.
this was fixed in a previous PR, it might cause merge conflicts.
| @@ -11,7 +11,7 @@ ROOT=$(dirname "$0") | |||
| INDICNLP=$ROOT/indic_nlp_library | |||
| if [ ! -e $INDICNLP ]; then | |||
| echo "Cloning Indic NLP Library..." | |||
| git -C $ROOT clone https://github.com/anoopkunchukuttan/indic_nlp_library.git | |||
| git clone https://github.com/anoopkunchukuttan/indic_nlp_library.git $INDICNLP | |||
There was a problem hiding this comment.
already fixed in a previous PR
|
@guzmanhe thanks! I rebased + merged the two previous PRs so if they are accepted I should have no merge conflicts but let me know if there are issues |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A working branch of FLORES v1
Tested
reproduce.sh neenand got 12.5 BLEU on devtest after 2 iterations of BT (compared to README's 15.9). I used 1 RTX8000 and the full pipeline ran in ~100 hours (after adjustingmax_tokensto 16000 to eliminate unnecessaryupdate_freq)Download Issues:
download_indic.shgit command FIx download_indic.sh (outdated git command) #22en-hidataset as it is now manual download only (https://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html)Other issues:
fairseq-trainargs e.g.min-lrargsto omegaconfcfg