feat(ifr): support Molmo-72b (#3989)

ai-data/managed-inference/reference-content/molmo-72b-0924.mdx

@@ -0,0 +1,164 @@
+---
+meta:
+  title: Understanding the Molmo-72b-0924 model
+  description: Deploy your own secure Molmo-72b-0924 model with Scaleway Managed Inference. Privacy-focused, fully managed.
+content:
+  h1: Understanding the Molmo-72b-0924 model
+  paragraph: This page provides information on the Molmo-72b-0924 model
+tags: ai molmo inference
+dates:
+  validation: 2024-11-27
+  posted: 2024-11-27
+categories:
+  - ai-data
+---
+
+## Model overview
+
+| Attribute       | Details                            |
+|-----------------|------------------------------------|
+| Provider        | [Allen Institute for AI](https://molmo.allenai.org/blog)                         |
+| License        | Apache 2.0  |                |
+| Compatible Instances | H100-2 (FP8)                 |
+| Context size | 50k tokens    |
+
+## Model name
+
+```bash
+allenai/molmo-72b-0924:fp8
+```
+
+## Compatible Instances
+
+| Instance type  | Max context length |
+| ------------- |-------------|
+| H100-2      | 50k (FP8)
+
+## Model introduction
+
+Molmo 72B is the powerhouse of the Molmo family, multimodal models developed by the renowned research lab Allen Institute for AI.
+Vision-language models like Molmo can analyze an image and offer insights from visual content alongside text. This multimodal functionality creates new opportunities for applications that need both visual and textual comprehension. 
+
+Molmo is open-weight and distributed under the Apache 2.0 license. All artifacts (code, data set, evaluations) are also expected to be fully open-source.
+Its base model is Qwen2-72B ([Twonyi Qianwen license](https://huggingface.co/Qwen/Qwen2-72B/blob/main/LICENSE)).
+
+## Why is it useful?
+
+- Molmo-72b allows you to process real world and high resolution images, unlocking capacities such as transcribing handwritten files or payment receipts, extracting information from graphs, captioning images, etc.
+- This model achieves the [highest academic benchmark scores and ranks second on human evaluation](https://huggingface.co/allenai/Molmo-72B-0924#evaluations) at the time of writing (September 2024)
+
+<Message type="note">
+  Molmo-72b can understand and analyze images, not generate them. You will use it through the /v1/chat/completions endpoint.
+</Message>
+
+## How to use it
+
+### Sending inference requests
+
+<Message type="tip">
+  Unlike regular chat models, Molmo-72b can take an `image_url` in the content array.
+</Message>
+
+To perform inference tasks with your Molmo-72b model deployed at Scaleway, use the following command:
+
+```bash
+curl -s \
+-H "Authorization: Bearer <IAM API key>" \
+-H "Content-Type: application/json" \
+--request POST \
+--url "https://<Deployment UUID>.ifr.fr-par.scw.cloud/v1/chat/completions" \
+--data '{
+       "model": "allenai/molmo-72b-0924:fp8",
+       "messages": [
+        {
+          "role": "user",
+          "content": [
+              {"type" : "text", "text": "Describe this image in detail please."},
+              {"type": "image_url", "image_url": {"url": "https://picsum.photos/id/32/512/512"}}
+          ]
+        }
+       ],
+       "top_p": 1, 
+       "temperature": 0.7, 
+       "stream": false
+}'
+```
+
+Make sure to replace `<IAM API key>` and `<Deployment UUID>` with your actual [IAM API key](/identity-and-access-management/iam/how-to/create-api-keys/) and the Deployment UUID you are targeting.
+
+<Message type="tip">
+  The model name allows Scaleway to put your prompts in the expected format.
+</Message>
+
+### Known limitations
+
+- Molmo-72b was reported to struggle with transparent images. The official recommandation is to add white or dark background to images for the time being.
+- Molmo-72b chat template does not support the system role. Ensure that the `messages` array is properly formatted with user role and assistant role only.
+- Molmo-72b is not able to generate structured outputs (`response_format` parameter not supported).
+- Molmo-72b cannot do function calling (`tools` parameter not supported).
+
+### Passing images to Molmo-72b
+
+#### Image URLs
+If the image is available online, you can just include the image URL in your request as demonstrated above. This approach is simple and does not require any encoding.
+
+#### Base64 encoded image
+Base64 encoding is a standard way to transform binary data, like images, into a text format, making it easier to transmit over the internet.
+
+The following Python code sample shows you how to encode an image in base64 format and pass it to your request payload.
+
+```python
+import base64
+from io import BytesIO
+from PIL import Image
+
+def encode_image(img):
+    buffered = BytesIO()
+    img.save(buffered, format="JPEG")
+    encoded_string = base64.b64encode(buffered.getvalue()).decode("utf-8")
+    return encoded_string
+
+img = Image.open("path_to_your_image.jpg")
+base64_img = encode_image(img)
+
+payload = {
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "What is this image?"},
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
+                },
+            ],
+        }
+    ],
+    ... # other parameters
+}
+
+```
+
+### Receiving Managed Inference responses
+
+Upon sending the HTTP request to the public or private endpoints exposed by the server, you will receive inference responses from the managed Managed Inference server. 
+Process the output data according to your application's needs. The response will contain the output generated by the visual language model based on the input provided in the request.
+
+<Message type="note">
+  Despite efforts for accuracy, the possibility of generated text containing inaccuracies or [hallucinations](/ai-data/managed-inference/concepts/#hallucinations) exists. Always verify the content generated independently.
+</Message>
+
+## Frequently Asked Questions
+
+#### What types of images are supported by Molmo-72b?
+- Bitmap (or raster) image formats, meaning storing images as grids of individual pixels, are supported: PNG, JPEG, WEBP, and non-animated GIFs in particular.
+- Vector image formats (SVG, PSD) are not supported.
+
+#### Are other file types supported?
+Only bitmaps can be analyzed by Molmo. PDFs and videos are not supported.
+
+#### Is there a limit to the size of each image?
+The only limitation is the context window (1 token for each 16x16 pixel).
+
+#### What is the maximum amount of images per conversation?
+One conversation can handle a maximum of 1 image (per request). Sending more than one image will return a 400 error.

menu/navigation.json

@@ -693,6 +693,10 @@
                     "label": "Mixtral-8x7b-instruct-v0.1 model",
                     "slug": "mixtral-8x7b-instruct-v0.1"
                   },
+                  {
+                    "label": "Molmo-72b-0924 model",
+                    "slug": "molmo-72b-0924"
+                  },
                   {
                     "label": "Sentence-t5-xxl model",
                     "slug": "sentence-t5-xxl"