[Text Generation][V2] `LinearRouter` to accept SPLIT/JOIN #1434

dbogunowicz · 2023-11-28T08:04:47Z

It seems that fundamentally at the Pipeline level, there is an assumption that ops is a list, not a dictionary.

To reproduce:

from deepsparse.v2.text_generation import TextGenerationPipelineNoCache

prompt = ["Some funny prompt"]

pipeline = TextGenerationPipelineNoCache(model_path="hf:mgoin/TinyStories-1M-ds",
                                         onnx_model_name="model-orig.onnx",
                                         sequence_length=20)

out = pipeline(prompt=prompt)

Traceback (most recent call last):
  File "/home/ubuntu/.cache/JetBrains/RemoteDev/dist/67886da002816_pycharm-professional-231.9225.5/plugins/python/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/ubuntu/.cache/JetBrains/RemoteDev/dist/67886da002816_pycharm-professional-231.9225.5/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/ubuntu/damian/deepsparse/hehe2.py", line 10, in <module>
    out = pipeline(prompt=prompt,
  File "/home/ubuntu/damian/deepsparse/src/deepsparse/v2/pipeline.py", line 265, in __call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/damian/deepsparse/src/deepsparse/v2/text_generation/pipeline_no_kv_cache.py", line 123, in run
    return super().run(*args, **kwargs)
  File "/home/ubuntu/damian/deepsparse/src/deepsparse/v2/pipeline.py", line 217, in run
    operator=self.ops[next_step],
KeyError: 0

… router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix

* [v2] EngineOperator updates to make continuous batching easier * test fixes

…ity (#1348) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings

…generation functionality (#1356) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * fix capacity settting again * typo fixes

…e code to remove repeat code, update map function

…eature/damian/v2/factor_out_transformation_utils

) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class

…scope of fixtures module to help with speed

…eature/damian/v2/factor_out_transformation_utils

* unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * fix name

…ng and prioritization (#1373) * [Continuous Batching] Queue Implementation to support batching grouping and prioritization * has_key method * thread safety * add blocking option for pop_batch * update docstring * allow mutex to be shared across continuous batching objects * revert last commit

…#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack

* [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition

* [continuous batching] singleton pattern for scheduler * catch from review

…ating engine_inputs (#1364) * rebasing off my initial commit * cleanups * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py --------- Co-authored-by: Dipika Sikka <[email protected]>

…functions (#1394) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * pipeline runs, but incorrectly * Revert "pipeline runs, but incorrectly" This reverts commit 51c4ee6. * PR review comments --------- Co-authored-by: Dipika Sikka <[email protected]>

* initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review

…#1409) * update split/join * use map * update * run end-to-end * clean-up * fix bug with batch size, introduce SplitRoute dataclass * update tests to use new inputs/outputs * use the normal scheduler for internal kv_cache * add pipeline inpuits * clean-up * change engine type, update docstrings, update override function to be more generic * move subgraph functionality to its own function; clean-up cont batching in text gen pipeline * update linear pathway to also use subgraph execution * rebase fix * fix tests

* initial registry functionality * use sparsezoo mixin

…outer now

bfineran and others added 30 commits October 26, 2023 13:22

Pipelines Refactor - Initial Impl (#1287)

3e00175

[v2] EngineOperator updates to make continuous batching easier (#1371)

58b0758

* [v2] EngineOperator updates to make continuous batching easier * test fixes

add split/join functionality

f18d5f3

update router to include split/join in parent class, refactor pipelin…

2c4d231

…e code to remove repeat code, update map function

process multiple generations

672ca20

initial commit

304eb35

fix error

71515ac

Merge remote-tracking branch 'origin/features/v2/run_multiple' into f…

6f1b175

…eature/damian/v2/factor_out_transformation_utils

[Pipeline Refactor] Split/Join Functionality for multiple prompts (#1384

041174b

) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class

unit testing for text generation operators

a508342

additional changes

cbb0e86

unit testing completion

2541581

remove debug

8c8989d

fix

f8d75e3

add todo

fd1e466

more clean-up

64c0552

fix test

913665a

add docstrings/comments

e15521f

break out tests to individual unit test files; add conftest and make …

379481e

…scope of fixtures module to help with speed

Merge remote-tracking branch 'origin/features/v2/unit_testing' into f…

a90a20a

…eature/damian/v2/factor_out_transformation_utils

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

c0c4240

Delete tests/deepsparse/v2/unit/text_generation/test_msic.py

4f248dd

[Continuous Batching] Executor thread for running continuous batching (…

d81012d

…#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack

[ContinuousBatching] ContinuousBatchingScheduler Implementation (#1375)

5c48505

* [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition

[continuous batching] singleton pattern for scheduler (#1391)

e1b7f37

* [continuous batching] singleton pattern for scheduler * catch from review

dbogunowicz and others added 20 commits November 14, 2023 10:46

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

98f7a6d

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

d1683b4

pipeline runs, but incorrectly

51c4ee6

it works for a single sequence

fa96efb

cleanup. now lets figure out how to run multiple sequences

e41ddf8

[Text Generation][V2] End-to-end tests (#1402)

1b9238a

* initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review

Merge remote-tracking branch 'origin/v2' into feature/damian/no_kv_cache

89f11e5

integration tests pass

9b441f5

[Pipeline Refactor] Operator Registry (#1420)

bb3ff41

* initial registry functionality * use sparsezoo mixin

Merge remote-tracking branch 'origin/v2' into feature/damian/no_kv_cache

19434e7

fix tricky rebase

90de2b3

one more cleanup

66ca295

got tests to work after rebase. implementing SPLIT and JOIN in linear…

dcded1d

…outer now

pipeline working, with GraphRouter. Needs some more testing

127aa00

ready for review

af57698

cleanup

4397c80

initial commit

6f1214e

dbogunowicz changed the base branch from main to feature/damian/no_kv_cache November 28, 2023 08:05

dbogunowicz mentioned this pull request Nov 28, 2023

[Text Generation][V2] NonKVCachePipeline #1417

Closed

dbogunowicz force-pushed the feature/damian/no_kv_cache branch from e0a9dee to 7f3eb12 Compare December 18, 2023 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Text Generation][V2] `LinearRouter` to accept SPLIT/JOIN #1434

[Text Generation][V2] `LinearRouter` to accept SPLIT/JOIN #1434

dbogunowicz commented Nov 28, 2023

[Text Generation][V2] LinearRouter to accept SPLIT/JOIN #1434

Are you sure you want to change the base?

[Text Generation][V2] LinearRouter to accept SPLIT/JOIN #1434

Conversation

dbogunowicz commented Nov 28, 2023

[Text Generation][V2] `LinearRouter` to accept SPLIT/JOIN #1434

[Text Generation][V2] `LinearRouter` to accept SPLIT/JOIN #1434