Could not install TabbyML hosted in K8s & integrated with Cloud Based APIs like OpenAI

[siddharthgaur2590]

Describe the bug
Trying to self host the TabbyML on K8s using Cloud Based API like openAI ones which have the GPUs capability but facing below error.

WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:98: llama-server exited with status code 127, args: Command { std: "/opt/tabby/bin/llama-server" "-m" "/data/models/TabbyML/Nomic-Embed-Text/ggml/model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }
2024-09-05T08:16:26.589Z | 2024-09-05T08:16:26.589532Z WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: : /opt/tabby/bin/llama-server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

Information about your version
tabbyml/tabby:20240826

Information about your GPU
Using the config mentioned herein: https://tabby.tabbyml.com/docs/references/models-http-api/openai/

Additional context
Following config are used to direct the tabbyML to use GPU based models present in OpenAI platform.

    "model.chat.http.api_endpoint": "https://api.openai.com/v1/chat/completions",
"model.chat.http.api_key": "$open-ai-key",
"model.chat.http.kind": "openai/chat",
"model.chat.http.model_name": "gpt-3.5-turbo",

"model.completion.http.api_endpoint": "https://api.openai.com/v1/completions",
"model.completion.http.api_key": "$open-ai-key",
"model.completion.http.kind": "openai/completion",
"model.completion.http.model_name": "gpt-3.5-turbo",

"model.embedding.http.api_endpoint": "https://api.openai.com/v1/embeddings",
"model.embedding.http.api_key": "$open-ai-key",
"model.embedding.http.kind": "openai/embedding",
"model.embedding.http.model_name": "text-embedding-ada-002"

[wsxiaoys]

Hello, it appears that your configuration file is not in the config.toml format that Tabby currently utilizes. Could you provide some additional details?

[siddharthgaur2590]

Hi @wsxiaoys, Thanks for reverting back.

Just to give a little background. I'm trying to self-host aforesaid open-source docker image version of tabbyml onto my K8s cluster. As a part of self-hosting I have used following K8s resources:

• deployment (having open-source docker image of tabby)

• configMap (having the openAI platform metadata to connect to the models present in OpenAI and having GPUs capability) (attached here in this response)
  Tabbyml self-hosting - Config.txt

• persistent volume

• persistent volume claim

• service

• ingress

K8s installation of any product never have file with name config.toml. Pls guide how to make tabbyml work while self-hosting.

Just FYI, my pods are up and running but still instances are not working as it expects the llama-server model to be present in my k8s infra instead of picking up from openAI model.

Many thanks

[zwpaper]

@siddharthgaur2590 how do you mount the configMap into your pod? looks like its not formatted in toml.

maybe your should create a configMap by a file containing toml configuration described in https://tabby.tabbyml.com/docs/administration/model/, and then mount the configMap into the tabby pod under ~/.tabby/config.toml

[siddharthgaur2590]

@wsxiaoys
I tried exactly. Created the configmap with the toml configurations mentioned here for openAI: https://tabby.tabbyml.com/docs/references/models-http-api/openai/ & placed it in the directory ~/.tabby/config.toml.
Still not working...
May I know I still get below error which means it's connecting to default configuration still. how can I locate the default directory of config.toml which has this llama-cpp-server configurations as I cannot find it here: ~/.tabby/config.toml.

WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:98: llama-server exited with status code 127, args: Command { std: "/opt/tabby/bin/llama-server" "-m" "/data/models/TabbyML/Nomic-Embed-Text/ggml/model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }
2024-09-09T07:46:42.953Z | 2024-09-09T07:46:42.953111Z WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: : /opt/tabby/bin/llama-server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory