Could somene help me out configuring ollama http api?

[estevaoem]

My current setup is a 3060 with a ryzen 5 3600, 16gb of ddr4 ram

My Tabby instance is configured as follows

services:
  tabby:
    container_name: tabby
    restart: always
    image: registry.tabbyml.com/tabbyml/tabby
    command: serve --device cuda
    volumes:
      - "$HOME/.tabby:/data"
    environment:
      - RUST_BACKTRACE=full
    ports:
      - 8080:8080
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

and my tabby config file is

# Completion model
[model.completion.http]
kind = "ollama/completion"
model_name = "codellama:13b"
api_endpoint = "http://localhost:11434"
prompt_template = "<PRE> {prefix} <SUF>{suffix} <MID>"  # Example prompt template for the CodeLlama model series.

I got ollama running with codellama-13b and that consumes the following amount of memory

Sun Oct  6 10:34:02 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:07:00.0  On |                  N/A |
|  0%   48C    P8             15W /  170W |    9441MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1251059      G   /usr/lib/xorg/Xorg                            124MiB |
|    0   N/A  N/A   1251207      G   /usr/bin/gnome-shell                           43MiB |
|    0   N/A  N/A   1304966      C   ...unners/cuda_v11/ollama_llama_server       9198MiB |
|    0   N/A  N/A   1306818      G   ...erProcess --variations-seed-version         46MiB |
+-----------------------------------------------------------------------------------------+
Sun Oct  6 10:34:03 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:07:00.0  On |                  N/A |
|  0%   48C    P8             15W /  170W |    9441MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1251059      G   /usr/lib/xorg/Xorg                            124MiB |
|    0   N/A  N/A   1251207      G   /usr/bin/gnome-shell                           43MiB |
|    0   N/A  N/A   1304966      C   ...unners/cuda_v11/ollama_llama_server       9198MiB |
|    0   N/A  N/A   1306818      G   ...erProcess --variations-seed-version         46MiB |
+-----------------------------------------------------------------------------------------+

However, when I start the docker container (with the ollama server already running ofc), I get the following error:

⠴   234.313 s	StaThe application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: An error occurred with ollama-rs: error sending request for url (http://localhost:11434/api/show)

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <ollama_rs::Ollama as ollama_api_bindings::model::OllamaModelExt>::model_available::{{closure}}
   2: <ollama_rs::Ollama as ollama_api_bindings::model::OllamaModelExt>::select_model_or_default::{{closure}}
   3: http_api_bindings::completion::create::{{closure}}
   4: tabby::serve::main::{{closure}}
   5: tokio::runtime::park::CachedParkThread::block_on
   6: tabby::main
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: std::rt::lang_start::{{closure}}
   9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
  10: std::panicking::try::do_call
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  11: std::panicking::try
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  12: std::panic::catch_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  13: std::rt::lang_start_internal::{{closure}}
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
  14: std::panicking::try::do_call
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  15: std::panicking::try
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  16: std::panic::catch_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  17: std::rt::lang_start_internal
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
  18: main
  19: <unknown>
  20: __libc_start_main
  21: _start
Location: /root/workspace/crates/ollama-api-bindings/src/completion.rs:65

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 9 frames hidden ⋮                               
  10: http_api_bindings::completion::create::{{closure}}::h65b608a930156aea
      at <unknown source file>:<unknown line>
  11: tabby::serve::main::{{closure}}::h55900fa254ec9d48
      at <unknown source file>:<unknown line>
  12: tokio::runtime::park::CachedParkThread::block_on::h2fe37a5bb361fa5a
      at <unknown source file>:<unknown line>
  13: tabby::main::hdc6683cc13df07ff
      at <unknown source file>:<unknown line>
  14: std::sys_common::backtrace::__rust_begin_short_backtrace::h487cee32ddec08d1
      at <unknown source file>:<unknown line>
  15: std::rt::lang_start::{{closure}}::h62dde838d7c658ba
      at <unknown source file>:<unknown line>
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h37600b1e5eea4ecd
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284
  17: std::panicking::try::do_call::hb4bda49fa13a0c2b
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552
  18: std::panicking::try::h8bbf75149211aaaa
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516
  19: std::panic::catch_unwind::h8c78ec68ebea34cb
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142
  20: std::rt::lang_start_internal::{{closure}}::hffdf44a19fd9e220
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148
  21: std::panicking::try::do_call::hcb3194972c74716d
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552
  22: std::panicking::try::hcdc6892c5f0dba4c
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516
  23: std::panic::catch_unwind::h4910beb4573f4776
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142
  24: std::rt::lang_start_internal::h6939038e2873596b
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148
  25: main<unknown>
      at <unknown source file>:<unknown line>
  26: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
  27: _start<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

Since i can curl to /api/generate

curl http://localhost:11434/api/generate -d '{
  "model": "codellama:13b",
  "prompt": "Why is the sky blue?"
}'

I have no no idea why ollama-rs crate is not able to reach the localhost as mentioned in the panic stacktrace.

Any ideas? Thanks