Image and Video generation

[choonspin]

Hello, can I ask if I can add models which allow for image or video such as OpenAI's Dall-e-3 model?

[nsarrazin]

hi! The way we do this in HuggingChat is by having tools and a tool calling compatible model. The tools are hosted on Hugging Face Spaces, for example the image generation one is black-forest-labs/FLUX.1-schnell

You can check out our prod config here: prod.yaml

Of interest to you are the MODELS and TOOLS env variable

  MODELS: >
    [
      //...
      {
        "name": "meta-llama/Meta-Llama-3.1-70B-Instruct",
        "id": "meta-llama/Meta-Llama-3.1-70B-Instruct",
        "tokenizer": "nsarrazin/llama3.1-tokenizer",
        // ...
        "tools": true,
     // ...
     }
    ]

  TOOLS: >
    [
      {
        "_id": "000000000000000000000001",
        "displayName": "Image Generation",
        "description": "Use this tool to generate images based on a prompt.",
        "color": "yellow",
        "icon": "camera",
        "baseUrl": "black-forest-labs/FLUX.1-schnell",
        "name": "image_generation",
        "endpoint": "/infer",
        "inputs": [
          {
            "name": "prompt",
            "description": "A prompt to generate an image from",
            "paramType": "required",
            "type": "str"
          },
          { "name": "seed", "paramType": "fixed", "value": "0", "type": "float" },
          {
            "name": "randomize_seed",
            "paramType": "fixed",
            "value": "true",
            "type": "bool"
          },
          {
            "name": "width",
            "description": "numeric value between 256 and 2048",
            "paramType": "optional",
            "default": 1024,
            "type": "float"
          },
          {
            "name": "height",
            "description": "numeric value between 256 and 2048",
            "paramType": "optional",
            "default": 1024,
            "type": "float"
          },
          {
            "name": "num_inference_steps",
            "paramType": "fixed",
            "value": "4",
            "type": "float"
          }
        ],
        "outputComponent": "image",
        "outputComponentIdx": 0,
        "showOutput": true
      },
    // ...
]

It's not the easiest feature to implement in the config but let me know if that helps.

[choonspin]

hello! I see thank you for your detailed reply. Right now I am not using Kubernetes to deploy and I am using the .env.local file as well a models.json file that I am adding models to.
I suppose the setup is similar and I can try adding the tools into my environment file. Thank you!