…ting_target (mosaicml#977)

…al when the checkpoint is ready (mosaicml#813)

* working without sharded checkpointing..

* add more debugs

* try this

* more debugging

* yikes dumb bug

* add notes

* fixes

* remove prints

* small updates

* fix typo

* refactor

* fix docstring formatting

* fighting with docstrings

* try this

* add unit tests

* point to composer update

* values -> items

* serialize time

* fix merge

* nits

* warning, small comment update

* add error

---------

Co-authored-by: Daniel King <[email protected]>

Co-authored-by: Daniel King <[email protected]>

* if condition in tie weights added

* unit test for tie weights

* add oom observer callback

* fix format

This reverts commit e3f214e.

Co-authored-by: Irene Dea <[email protected]>

…g; Make ComposerHFT5 experimental (mosaicml#1007)

* Deprecate features and mark experimental

* fix typo

---------

Co-authored-by: Daniel King <[email protected]>

* add magic filename for sharded state dicts

* Update scripts/train/train.py

Co-authored-by: Daniel King <[email protected]>

* oops forgot to push this

* no shard if no fsdp

* default to full on foundry

---------

Co-authored-by: Daniel King <[email protected]>

* fix bug on metrics

* lint

* lint

* add unit test

* lint

Red button because CI running jobs it doesn't need. Tests passed on main.

…l#1003)

Co-authored-by: Daniel King <[email protected]>

… finetuning (mosaicml#985)

The main purpose of this PR is to support training on non-terminal responses in multi-round chats. This is achieved by tokenizing at the level of conversation "turns" and exposing some options for what turns are used as training targets (i.e. generate loss). This also adds support for treating prompt tokens as loss-generating.

The script for converting a finetuning dataset to streaming has also been updated (with some bug fixes).

Co-authored-by: Max Marion <[email protected]>

* Fix typo in monolithic chkpt callback docs

* reorder to match function signature

Reverts part of the change made in
https://github.com/mosaicml/llm-foundry/pull/1000/files#diff-4a2765c2cfcbd3804a66aab805cb92ddda74de1730923cc5bf53671d0beccf06L11

Fixed script path

* bump composer

* accounted for bug fix

* changed fsdp_elastic_sharded_enabled to fsdp_sharded_state_dict_enabled in the AsyncEvalCallback

---------

Co-authored-by: Daniel King <[email protected]>

* flagged areas to throw ft errors + added custom exception

* added exceptions for all files

* fixed type of invalid type errors

* fixed merge

* added exceptions for all files

* fixed merge

* added a catch for missing hf url

* expain dataloader catch to all error types

* added tests for custom exceptions

* fixed a couple tests

* fixed some more tests

* addressed comments

* fixed formatting, updated split correction, moved logger setup into helper

* added check for repeating roles

* more strict checking for prompt response type

* removed some cases we don't need

* updated name of unknown conversation type error

* formatting

* Fix multi model eval (mosaicml#1055)

* resolved merge conflict

* formatted

* added back two imports

* formatting changes

* formatted again

* sorted imports

* diable yapf for exceptions import

* disabled yapf on test dataloader

---------

Co-authored-by: Daniel King <[email protected]>

* Bump version

* Remove triton (mosaicml#1062)

* Remove github action workflows for version bumps

* Fix cpu test issues

* code quality

* Fix gpu tests

* Fix gpu tests nicely

* Remove z-loss (mosaicml#1064)

* Remove prefix lm and denoising (mosaicml#1065)

* Remove hf_prefix_lm

* Remove prefix_lm from mpt modeling

* Remove bidirectional mask

* Remove text denoising dataloading

* Remove adapt tokenizer

* Remove llama attention patch (mosaicml#1066)

* Remove bidirectional mask in tests

* Fix test_hf_config_override with patch

* Fix comparison in train for hf_t5

* Add back context printing in dataloader.py

* Skip flaky lion8b test (mosaicml#598)

* relax atol and add retries to reduce flakiness in lion8b timing test

* add eval output logging

* add back tasks

* foo

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* fix prompt

* fix prompt

* modify mcli

* test

* test

* fix

* fix merge

* wip

* merge

* reset files, wip commit

* rm small changes

* reduce changes

* reduce changes

* .

* wip

* rm batch keys

* revert init device

* linting

* add import

* fix import

* add eval_output_logging to registry

* readd import

* pyright + linting

---------

Co-authored-by: dblalock <[email protected]>
Co-authored-by: Jeremy Dohmann <[email protected]>

…/ bos token id (mosaicml#1039)

* lint

* lint

* added warning and error message instead of setting the eos and bos token ids

* Update text_data.py

Adding info about the override flags in the error message.

* Update llmfoundry/data/text_data.py

Co-authored-by: Vitaliy Chiley <[email protected]>

* Update llmfoundry/data/text_data.py

Co-authored-by: Vitaliy Chiley <[email protected]>

* adding warning if user does not provide eos or bos token id

* adding warning if user does not provide eos or bos token id

---------

Co-authored-by: Vitaliy Chiley <[email protected]>

* Triton RMS Norm

* fix

* refactor

* Fix test

* no bias

* Update tests/models/test_model.py

Co-authored-by: Daniel King <[email protected]>

* fixes

* Unittest

* fix

* fix2

* fix3

* fix3

---------

Co-authored-by: Daniel King <[email protected]>

* fix vocab size bug

* more tests

…mosaicml#1079)

* setting loss_fn reduction to None

* fixing a unit test

* add error message

* adding test to check reduction

* adding test to check reduction

* Update llmfoundry/models/mpt/modeling_mpt.py

Co-authored-by: Vitaliy Chiley <[email protected]>

* preserving batch dimension of targets

* minor change

---------

Co-authored-by: Vitaliy Chiley <[email protected]>
Co-authored-by: Daniel King <[email protected]>

)

* log packing ratio progress

* use tqdm to show progress of packing ratio profiling

* fix pbar

* Update packing.py

* added log to search for

* fix

* good old fashioned progress logging

* Update packing.py

Co-authored-by: Daniel King <[email protected]>

* updating the streaming version in setup.py

* updating constructor call to StreamingDataset

* making allow_unsafe_types and replication configurable through dataset configs

* adding docstring

* [Stage] Megablocks release (mosaicml#241)

* V1 of MegaBlocks
---------

* fix hf ckptr

* rename

* lint

* lint

---------

Co-authored-by: Abhinav Venigalla <[email protected]>
Co-authored-by: Sasha Doubov <[email protected]>
Co-authored-by: Cheng Li <[email protected]>
Co-authored-by: Ning Wang <[email protected]>
Co-authored-by: Irene Dea <[email protected]>
Co-authored-by: Shashank Rajput <[email protected]>
Co-authored-by: Chuck Tang <[email protected]>
Co-authored-by: Jose Javier <[email protected]>
Co-authored-by: Angel Ruiz <[email protected]>
Co-authored-by: Denny Lee <[email protected]>
Co-authored-by: Jane Zhang <[email protected]>
Co-authored-by: Daniel King <[email protected]>
Co-authored-by: Chuck Tang <[email protected]>
Co-authored-by: Vitaliy Chiley <[email protected]>

https://databricks.atlassian.net/browse/GRT-2812

see [here](https://github.com/mosaicml/llm-foundry-private/pull/245#issuecomment-2046306845)
Ran 500 steps, new version did marginally better

…#1106)

This adds support for the other common chat format. We just remap keys and add a new role.