Llm large reference #1915
Llm large reference #1915
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
pgmpablo157321
force-pushed
the
llm_large_reference
branch
2 times, most recently
from
3b33ce1
to
7be9b13
Compare
| no_eos_ids = [] | ||
| for qid, output in tqdm(run_outputs.items()): | ||
| L = list(output) | ||
| # Prune trailing 2s (EOS token) |
There was a problem hiding this comment.
EOS ID is not 2 for llama-405B. Need to use tokenizer.eos_token_id.
There was a problem hiding this comment.
What is the purpose of consolidate_results.py? I copied it from Llama2 but don't know why it is needed
There was a problem hiding this comment.
Might not be needed since we don't split pickle anymore. Please remove if not needed.
| predictions=preds, references=targets, use_stemmer=True, use_aggregator=False | ||
| ) | ||
|
|
||
| assert len(rouge_scores["rouge1"]) == 24576 |
| self.model = LLM( | ||
| self.model_path, | ||
| dtype=self.dtype, | ||
| tensor_parallel_size=self.tensor_parallel_size, |
There was a problem hiding this comment.
Is there a reason we are not using AsyncLLMEngine here? It may be more efficient since it will support continous batching
|
|
||
| def main(args): | ||
| # Set up decode and evaluation objects | ||
| tokenizer = LlamaTokenizerFast.from_pretrained(args.model_dir) |
There was a problem hiding this comment.
i may be wrong, but would it better to use AutoTokenizer here?
There was a problem hiding this comment.
I also suggest to use AutoTokenizer for robustness
pgmpablo157321
force-pushed
the
llm_large_reference
branch
2 times, most recently
from
5da4409
to
e44c62a
Compare
pgmpablo157321
force-pushed
the
llm_large_reference
branch
from
3fbdbb0
to
fe9c189
Compare
No description provided.