@cboettig sure, happy to look into it. What's the easiest way for me to get access to a live vllm endpoint?

@hadley amazing! Can I just email you a token for this NAIRR test platform we're using?

For comparison here's a quick gist showing tool-use via langchain on the same platform: https://gist.github.com/cboettig/6359fc23a637d328e5fa25a40499abb2

Is there an easy way for you to see what JSON langchain is sending to the server? That would make it easier for me to figure out what elmer is doing wrong.

library(elmer)
chat <- chat_openai(
  base_url = "https://llm.nrp-nautilus.io/",
  api_key = Sys.getenv("VLLM_KEY"),
  model = "gorilla"
)
chat$register_tool(tool(function() "2024-01-01", "Gets the current date"))
chat$chat("What's the current date in YMD format?")
#> Error in `req_perform_connection()`:
#> ! HTTP 400 Bad Request.
#> • litellm.BadRequestError: OpenAIException - Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tools', 0, 'function', 'strict'), 'msg': 'Extra inputs are not permitted', 'input': True}]", 'type': 'BadRequestError', 'param': None, 'code': 400}
#> Received Model Group=gorilla
#> Available Model Group Fallbacks=None

elmer:::last_request_json()
#> {
#>   "messages": [
#>     {
#>       "role": "user",
#>       "content": [
#>         {
#>           "type": "text",
#>           "text": "What's the current date in YMD format?"
#>         }
#>       ]
#>     }
#>   ],
#>   "model": "gorilla",
#>   "stream": true,
#>   "stream_options": {
#>     "include_usage": true
#>   },
#>   "tools": [
#>     {
#>       "type": "function",
#>       "function": {
#>         "name": "tool_001",
#>         "description": "Gets the current date",
#>         "strict": true,
#>         "parameters": {
#>           "type": "object",
#>           "description": "",
#>           "properties": {},
#>           "required": [],
#>           "additionalProperties": false
#>         }
#>       }
#>     }
#>   ]
#> }

^{Created on 2024-10-31 with reprex v2.1.0}

Ooooh I get it; I'm sending strict because I think that's generally what you for openAI, but it's clearly not supported by vllm. I think the best approach is probably to make a specific vllm backend. (Especially since the default model will be different.)

@hadley nice! Yeah, it looks like langchain is just sending like this, no stict like you said.

I'm a total novice at this, but just some notes from what I've gleaned in setting up things with VLLM for tool parsing.
Clearly the 'standards' are still shaking out here, looks like OpenAI introduced strict in August

because VLLM supports a wide range of models that do function calling differently, it has some configuration options for the --tool-call-parser. This gorilla model is using the llama3_json parser.

anyway I think a vllm-specific backend would be awesome, and I think a prototype based on the openai one but without stict would work get us off the ground here! But down-the-road it might need pluggable options to support the different tool-calling-parsers of different vllm models?

{
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "multiply",
                "description": "Multiply 'x' times 'y'.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "x": {"type": "number"},
                        "y": {"type": "number"}
                    },
                    "required": ["x", "y"]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "exponentiate",
                "description": "Raise 'x' to the 'y'.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "x": {"type": "number"},
                        "y": {"type": "number"}
                    },
                    "required": ["x", "y"]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "add",
                "description": "Add 'x' and 'y'.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "x": {"type": "number"},
                        "y": {"type": "number"}
                    },
                    "required": ["x", "y"]
                }
            }
        }
    ]
}

Yeah, I think once you get a vllm provider, it's easy enough to customise. I'll probably take a stab at it in the next week or two, unless you get to it first 😄

Can use https://platform.openai.com/docs/api-reference/models/list to list all available models (to provide interface like ollama)

Yup!

Just for clarity: the web API here is actually LiteLLM, see endpoints: https://litellm-api.up.railway.app/. This includes the models list.

curl https://llm.nrp-nautilus.io/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"

{"data":[{"id":"phi3","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"mixtral","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"llava","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"embed-mistral","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"gemma2","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"gorilla","object":"model","created":1677610602,"owned_by":"openai"},
{"id":"llama3","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}

(obviously the metadata here isn't super, and these models aren't actually owned by 'openai')

on the NRP platfrom LiteLLM API sits in front of the VLLMs, each model is hosted as it's own separate vllm app which the user can then select from via litellm api. All those endpoints in the LiteLLM docs should work given a token. (sorry I know this complexifies whether we think of this as the 'vllm' backend or the 'litellm' backend).

It's possible to use LiteLLM as interface to non-VLLM-backed models, including commercial models. VLLM is nice both because of it's ability to support many concurrent requests and also to deploy on a wide variety of GPU hardware, and can be used to directly serve most models that are on huggingface (and elsewhere), instead of hoping someone packages it for Ollama. I believe this streamlines the process of adding new models to the library and facilitates development & competition in these models.

also another small clarification -- while we have this hosted version setup to serve a large classroom of students without local installations, just to note if you have a local (or remote) GPU you can pip install vllm and serve like you do with ollama (works nicely with nvidia & most other gpus used by cloud providers, though at this time I don't think it supports apple's chips).

e.g. for function calling this would just be:

vllm serve gorilla-llm/gorilla-openfunctions-v2 --dtype=half --trust-remote-code --enable-auto-tool-choice  --tool-call-parser llama3_json

(add --host and --port if necessary)

where by default the model name is the huggingface repo (e.g. https://huggingface.co/gorilla-llm/gorilla-openfunctions-v2). (and apologies if this was obvious or not relevant, I'm just learning as I go along here).