Improved Documentation, Rust Tokenize Shuffle #96
Conversation
| ``` | ||
|
|
||
| ### Training | ||
| data-config comes from the json created (manually for rust code, automatically for ray) after tokenize shuffle. |
There was a problem hiding this comment.
Is it possible to incorporate the manifest creation in the above?
There was a problem hiding this comment.
We would have to write a separate python script and have a bash script that runs the two together.
| --tokenizer "EleutherAI/gpt-neox-20b" \ | ||
| --seqlen 2049 \ | ||
| --wds-chunk-size 8192 \ | ||
| --num-local-cells 512 |
There was a problem hiding this comment.
Should --num-local-cells be equal to the number of available cores (in which case we should probably decrease the default / add a comment)
There was a problem hiding this comment.
No. There is an explanation for this in the rust tokenize shuffle readme, but this default is reasonable.
README.md
Outdated
| ### Training | ||
| data-config comes from the json created (manually for rust code, automatically for ray) after tokenize shuffle. | ||
| ```bash | ||
| torchrun --nproc-per-node 8 -m training.train --scale 1b_1x_fast --data-config exp_data/datasets/tokenized/dclm_gs3_ls1_rs_tokshuf.json --logs dclm_rs_tokshuf_training_local_logs --torchcompile |
There was a problem hiding this comment.
We should change this to torchrun --nproc-per-node 8 -m training.train -- --scale 1b_1x_fast --data-config exp_data/datasets/tokenized/dclm_gs3_ls1_rs_tokshuf.json --logs dclm_rs_tokshuf_training_local_logs --attn-name torch_attn --torchcompile because it seems that new version of torchrun have an issue with the --logs option (so adding -- makes it clear that this is an option to training.train, not torchrun)
edit: added --attn-name torch_attn as well
Added rust tokenize shuffle + commands
Explained exp_data
Fixed commands for training and evaluation
Included examples in the above sections
Combined above examples with some additional commands to have a brief list of commands to download DCLM-baseline, tokenize shuffle, train, and evaluate a 1B_1x model.