Does tensorRT-LLM support serving 4bit quantised unsloth Llama model

[jayakommuru]

We want to deploy https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-bnb-4bit which is 4-bit quantized version of llama-3.2-1B model. It is quantized using bitsandbytes. Can we deploy this using tensor RT-LLM backend ? If so, is there any documentation to refer?

[danielhanchen]

You can use vLLM https://docs.vllm.ai/en/latest/quantization/bnb.html