Add CLIP model to enable test_clip.py #1500
Conversation
There was a problem hiding this comment.
Will this model work just for classification or for captioning as well? From the OpenCLIP models available, only a small fraction worked for captioning.
Can we add a description of the available models on sparsezoo somewhere? Either in a read me or docstring? Should make it clear how/if this differs from OpenCLIP
|
@dsikka it is just made for image/text retrieval and zero-shot image classification. I left in the pytest.skip for the captioning test as a result. |
| @mock_engine(rng_seed=0) | ||
| def test_visual_clip(engine, visual_input): | ||
| from deepsparse import Pipeline | ||
| from deepsparse.legacy import Pipeline |
| @mock_engine(rng_seed=0) | ||
| def test_text_clip(engine, text_input): | ||
| from deepsparse import Pipeline | ||
| from deepsparse.legacy import Pipeline |
| def model_folder(): | ||
| from huggingface_hub import snapshot_download | ||
|
|
||
| model_id = "neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds" |
There was a problem hiding this comment.
Could we add a quick comment/note indicating that this model is not from OpenCLIP and only used for zero-shot classification?
Since we've made a quantized CLIP (https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds), let's use it to test our pipeline!