Docs Updates #61
Open
Docs Updates #61
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
parfeniukink
approved these changes
Sep 13, 2024
Comment on lines
+38
to
+40
| - The `--target` flag specifies the server hosting the model. In this case, it is a local vLLM server. | ||
| - The `--model` flag specifies the model to evaluate. The model name should match the name of the model deployed on the server | ||
| - By default, GuideLLM will run a `sweep` of performance evaluations across different request rates, each lasting 120 seconds. The results will be saved to a local directory. |
There was a problem hiding this comment.
-
I would rename
flagtoparametersince our CLI supports both: parameters and flags. If you specify a flag - there is no value next to it. If you specify parameter - the value is requied then. -
In some cases we may get an error if the tokenizer is not specified. I would add another item here. Text is below:
- The
--tokenizerparameter specifies the tokenizer to encount the number of tokens in the dataset. If you faced any issues try using--tokenizer neuralmagic/Meta-Llama-3.1-8B-quantized.w8a8.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This pull request introduces the GuideLLM CLI guide, README enhancements, image uploads, and the supported backends documentation to highlight all the backends that can be used with GuideLLM.
Test Cases:
The GuideLLM CLI has been tested with various LLM models and backends.
Unit tests ensure core functionalities work as expected.
Documentation:
Created documentation detailing the GuideLLM CLI usage and output metrics.
Created documentation detailing the openai-compatible API/HTTP pathway for TGI, llama.cpp, and DeepSparse in supported_backends.md
Additional Information:
The pull request includes changes to the docs/guides directory for the CLI documentation.
Binary files containing performance summary visualizations are added to the docs/assets directory.
Please review and provide feedback.