[Feature] Support InstructBLIP #1685
Conversation
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## dev #1685 +/- ##
==========================================
- Coverage 68.16% 65.13% -3.03%
==========================================
Files 295 332 +37
Lines 23372 25847 +2475
Branches 3713 4130 +417
==========================================
+ Hits 15932 16836 +904
- Misses 6880 8393 +1513
- Partials 560 618 +58
Flags with carried forward coverage won't be shown. Click here to find out more.
☔ View full report in Codecov by Sentry. |
| @@ -0,0 +1,53 @@ | |||
| # MiniGPT4 | |||
There was a problem hiding this comment.
| # MiniGPT4 | |
| # InstructBLIP |
|
|
||
| ## Models and results | ||
|
|
||
| For Vicuna model, please refer to [MiniGPT-4 page](https://github.com/Vision-CAIR/MiniGPT-4) for preparation guidelines. |
There was a problem hiding this comment.
please change to instructblip page https://github.com/salesforce/LAVIS/tree/main/projects/instructblip
| @@ -52,8 +52,13 @@ | |||
| 'https://download.openmmlab.com/mmpretrain/v1.0/minigpt4/minigpt-4_qformer_20230615-1dfa889c.pth' # noqa | |||
| ), | |||
| lang_encoder=dict( | |||
| type='AutoModelForCausalLM', name_or_path='YOUR_PATH_TO_VICUNA'), | |||
| tokenizer=dict(type='LlamaTokenizer', name_or_path='YOUR_PATH_TO_VICUNA'), | |||
| type='AutoModelForCausalLM', | |||
There was a problem hiding this comment.
plz restore the modification
| llm_tokenizer=dict( | ||
| type='LlamaTokenizer', | ||
| name_or_path= | ||
| '/mnt/petrelfs/share_data/liuyuan/llm_weights/vicuna_weights_7b'), |
| text_backbone=dict( | ||
| type='AutoModelForCausalLM', | ||
| name_or_path= | ||
| '/mnt/petrelfs/share_data/liuyuan/llm_weights/vicuna_weights_7b'), |
There was a problem hiding this comment.
the same comment as above
demo/Confusing-Pictures.jpg
Outdated
There was a problem hiding this comment.
why do you upload this image
test.py
Outdated
There was a problem hiding this comment.
remove the this test file
|
|
||
|
|
||
| @MODELS.register_module() | ||
| class InstructBlipCaption(BaseModel): |
There was a problem hiding this comment.
plz remove caption suffix and add 'task' arg in init
|
when will merge this? we need to use it. Thanks |
Add InstructBLIP Implementation