Follow-up on the run_metadata changes

.gitbook.yaml

@@ -18,6 +18,7 @@ redirects:
   how-to/setting-up-a-project-repository/best-practices: how-to/project-setup-and-management/setting-up-a-project-repository/set-up-repository.md
   getting-started/zenml-pro/system-architectures: getting-started/system-architectures.md
   how-to/build-pipelines/name-your-pipeline-and-runs: how-to/pipeline-development/build-pipelines/name-your-pipeline-runs.md
+  how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-steps: how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-a-step.md
 
   # ZenML Pro
   getting-started/zenml-pro/user-management: getting-started/zenml-pro/core-concepts.md

.github/workflows/update-templates-to-examples.yml

@@ -46,7 +46,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.11.20  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -118,7 +118,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.10.30  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -189,7 +189,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.10.30  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -261,7 +261,7 @@ jobs:
         with:
           python-version: ${{ inputs.python-version }}
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.11.08  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout

docs/book/how-to/model-management-metrics/track-metrics-metadata/README.md

@@ -5,10 +5,44 @@ description: Tracking metrics and metadata
 
 # Track metrics and metadata
 
-Logging metrics and metadata is standardized in ZenML. The most common pattern is to use the `log_xxx` methods, e.g.:
+ZenML provides a unified way to log and manage metrics and metadata through 
+the `log_metadata` function. This versatile function allows you to log 
+metadata across various entities like models, artifacts, steps, and runs 
+through a single interface. Additionally, you can adjust if you want to 
+automatically the same metadata for the related entities.
 
-* Log metadata to a [model](attach-metadata-to-a-model.md): `log_model_metadata`
-* Log metadata to an [artifact](attach-metadata-to-an-artifact.md): `log_artifact_metadata`
-* Log metadata to a [step](attach-metadata-to-steps.md): `log_step_metadata`
+### The most basic use-case
+
+You can use the `log_metadata` function within a step:
+
+```python
+from zenml import step, log_metadata
+
+@step
+def my_step() -> ...:
+    log_metadata(metadata={"accuracy": 0.91})
+    ...
+```
+
+This will log the `accuracy` for the step, its pipeline run, and if provided 
+its model version.
+
+### Additional use-cases
+
+The `log_metadata` function also supports various use-cases by allowing you to 
+specify the target entity (e.g., model, artifact, step, or run) with flexible 
+parameters. You can learn more about these use-cases in the following pages:
+
+- [Log metadata to a step](attach-metadata-to-a-step.md)
+- [Log metadata to a run](attach-metadata-to-a-run.md)
+- [Log metadata to an artifact](attach-metadata-to-an-artifact.md)
+- [Log metadata to a model](attach-metadata-to-a-model.md)
+
+{% hint style="warning" %}
+The older methods for logging metadata to specific entities, such as 
+`log_model_metadata`, `log_artifact_metadata`, and `log_step_metadata`, are 
+now deprecated. It is recommended to use `log_metadata` for all future 
+implementations.
+{% endhint %}
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-a-model.md

@@ -1,62 +1,101 @@
 ---
-description: >-
-  Attach any metadata as key-value pairs to your models for future reference and
-  auditability.
+description: Learn how to attach metadata to a model.
 ---
 
 # Attach metadata to a model
 
+ZenML allows you to log metadata for models, which provides additional context
+that goes beyond individual artifact details. Model metadata can represent
+high-level insights, such as evaluation results, deployment information,
+or customer-specific details, making it easier to manage and interpret
+the model's usage and performance across different versions.
+
 ## Logging Metadata for Models
 
-While artifact metadata is specific to individual outputs of steps, model metadata encapsulates broader and more general information that spans across multiple artifacts. For example, evaluation results or the name of a customer for whom the model is intended could be logged with the model.
+To log metadata for a model, use the `log_metadata` function. This function
+lets you attach key-value metadata to a model, which can include metrics and
+other JSON-serializable values, such as custom ZenML types like `Uri`,
+`Path`, and `StorageSize`.
 
 Here's an example of logging metadata for a model:
 
 ```python
-from zenml import step, log_model_metadata, ArtifactConfig, get_step_context
 from typing import Annotated
+
 import pandas as pd
-from sklearn.ensemble import RandomForestClassifier
 from sklearn.base import ClassifierMixin
+from sklearn.ensemble import RandomForestClassifier
+
+from zenml import step, log_metadata, ArtifactConfig, get_step_context
+
 
 @step
-def train_model(dataset: pd.DataFrame) -> Annotated[ClassifierMixin, ArtifactConfig(name="sklearn_classifier")]:
-    """Train a model"""
-    # Fit the model and compute metrics
+def train_model(dataset: pd.DataFrame) -> Annotated[
+    ClassifierMixin, ArtifactConfig(name="sklearn_classifier")
+]:
+    """Train a model and log model metadata."""
     classifier = RandomForestClassifier().fit(dataset)
     accuracy, precision, recall = ...
+
+    step_context = get_step_context()
+
+    if step_context.model:
+       # Log metadata for the model
+       log_metadata(
+           metadata={
+               "evaluation_metrics": {
+                   "accuracy": accuracy,
+                   "precision": precision,
+                   "recall": recall
+               }
+           },
+           model_version_id=step_context.model.id,
+       )
 
-    # Log metadata for the model
-    # This associates the metadata with the ZenML model, not the artifact
-    log_model_metadata(
-        metadata={
-            "evaluation_metrics": {
-                "accuracy": accuracy,
-                "precision": precision,
-                "recall": recall
-            }
-        },
-        # Omitted model_name will use the model in the current context
-        model_name="zenml_model_name",
-        # Omitted model_version will default to 'latest'
-        model_version="zenml_model_version",
-    )
     return classifier
 ```
 
-In this example, the metadata is associated with the model rather than the specific classifier artifact. This is particularly useful when the metadata reflects an aggregation or summary of various steps and artifacts in the pipeline.
+In this example, the metadata is associated with the model rather than the
+specific classifier artifact. This is particularly useful when the metadata
+reflects an aggregation or summary of various steps and artifacts in the
+pipeline.
+
+{% hint style="info" %}
+You can use the `get_step_context()` function to get fetch the model and model 
+version that the step is using.
+{% endhint %}
+
+### Selecting Models with `log_metadata`
+
+When using `log_metadata` with a model, ZenML provides flexible options to
+attach metadata accurately:
+
+1. **Model Name and Version Provided**: If both a model name and version are
+   provided, ZenML will use these to identify and attach metadata to the
+   specific model version.
+2. **Model Name Only**: If only a model name is provided, ZenML will attach
+   metadata to the latest version of the model.
+3. **Model Version ID Provided**: If a model version ID is directly provided,
+   ZenML will use it to fetch and attach the metadata to that specific model
+   version.
 
 ## Fetching logged metadata
 
-Once metadata has been logged in an [artifact](attach-metadata-to-an-artifact.md), model, or [step](attach-metadata-to-steps.md), we can easily fetch the metadata with the ZenML Client:
+Once metadata has been attached to a model, it can be retrieved for inspection
+or analysis using the ZenML Client.
 
 ```python
 from zenml.client import Client
 
 client = Client()
 model = client.get_model_version("my_model", "my_version")
 
-print(model.run_metadata["metadata_key"].value)
+print(model.run_metadata["metadata_key"])
 ```
 
+{% hint style="info" %}
+When you are fetching metadata using a specific key, the returned value will 
+always reflect the latest entry.
+{% endhint %}
+
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-a-run.md

@@ -0,0 +1,87 @@
+---
+description: Learn how to attach metadata to a run.
+---
+
+# Attach Metadata to a Run
+
+In ZenML, you can log metadata directly to a pipeline run, either during or 
+after execution, using the `log_metadata` function. This function allows you 
+to attach a dictionary of key-value pairs as metadata to a pipeline run, 
+with values that can be any JSON-serializable data type, including ZenML 
+custom types like `Uri`, `Path`, `DType`, and `StorageSize`.
+
+## Logging Metadata Within a Run
+
+If you are logging metadata from within a step that’s part of a pipeline run, 
+calling `log_metadata` will attach the specified metadata to the current 
+pipeline run where the metadata key will have the `step_name:metadata_key` 
+pattern. This allows you to use the same metadata key from different steps 
+while the run's still executing.
+
+```python
+from typing import Annotated
+
+import pandas as pd
+from sklearn.base import ClassifierMixin
+from sklearn.ensemble import RandomForestClassifier
+
+from zenml import step, log_metadata, ArtifactConfig
+
+
+@step
+def train_model(dataset: pd.DataFrame) -> Annotated[
+    ClassifierMixin,
+    ArtifactConfig(name="sklearn_classifier", is_model_artifact=True)
+]:
+    """Train a model and log run-level metadata."""
+    classifier = RandomForestClassifier().fit(dataset)
+    accuracy, precision, recall = ...
+
+    # Log metadata at the run level
+    log_metadata(
+        metadata={
+            "run_metrics": {
+                "accuracy": accuracy,
+                "precision": precision,
+                "recall": recall
+            }
+        }
+    )
+    return classifier
+```
+
+## Manually Logging Metadata to a Pipeline Run
+
+You can also attach metadata to a specific pipeline run without needing a step, 
+using identifiers like the run ID. This is useful when logging information or 
+metrics that were calculated post-execution.
+
+```python
+from zenml import log_metadata
+
+log_metadata(
+    metadata={"post_run_info": {"some_metric": 5.0}},
+    run_id_name_or_prefix="run_id_name_or_prefix"
+)
+```
+
+## Fetching Logged Metadata
+
+Once metadata has been logged in a pipeline run, you can retrieve it using 
+the ZenML Client:
+
+```python
+from zenml.client import Client
+
+client = Client()
+run = client.get_pipeline_run("run_id_name_or_prefix")
+
+print(run.run_metadata["metadata_key"])
+```
+
+{% hint style="info" %}
+When you are fetching metadata using a specific key, the returned value will 
+always reflect the latest entry.
+{% endhint %}
+
+<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>