Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Move] Common Evaluation Modules to Sparsezoo` #1520

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

rahul-tuli
Copy link
Member

@rahul-tuli rahul-tuli commented Jan 9, 2024

Evaluator Move

This PR moves a few modules from deepsparse.evaluation to
sparsezoo.evaluation

Motivation and Context

The moved modules provide a common interface for evaluating models.
This interface can be leveraged and used by both SparseML
and DeepSparse to evaluate models.

Since SparseZoo is a common dependency for both repositories, it
makes sense to move these modules to SparseZoo.

How Has This Been Tested?

The tests for moved modules in deepsparse.evaluation are moved to sparsezoo.evaluation and other deepsparse specific deepsparse.evaluation tests were updated to use the moved modules.

Both deepsparse and sparsezoo tests were run to ensure that the move did not break anything.

Also ran the following eval command:

$ deepsparse.eval --target zoo:mpt-7b-mpt_pretrain-base_quantized \                                                             (move-evaluator-to-sparsezoo|●1✚2…4)
                --dataset hellaswag \
                --dataset gsm8k \
                --integration lm-evaluation-harness \
                --limit 2

Note: This diff relies on sparsezoo diff neuralmagic/sparsezoo#411

The failing tests should pass after sparsezoo side changes are merged!

rather than overriding `load_from_registry`
Copy link

@Satrat Satrat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall, see my comment on the DeepSparseEvaluationRegistry, was wondering if any of that can be moved to the SparseZoo base class

src/deepsparse/evaluation/registry.py Show resolved Hide resolved
src/deepsparse/evaluation/cli.py Show resolved Hide resolved
tests/deepsparse/evaluation/test_registry.py Outdated Show resolved Hide resolved
src/deepsparse/evaluation/registry.py Show resolved Hide resolved
@bfineran
Copy link
Contributor

bfineran commented Feb 2, 2024

base tests failing - LGTM otherwise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants