2.12.13

What's Changed

• enable heart beat daemon and error tracking exit hook by default by @savingoyal in #1953
• Add local configuration option by @romain-intel in #1850
• support compute pools by @savingoyal in #1952
• fix: Metaflow local config init by @saikonen in #1967
• Fix an issue with _graph_info by @romain-intel in #1966
• [Ready for Review] Fix issue where resuming on successful run will fail. by @darinyu in #1956
• Align cloning log msgs by @savingoyal in #1963
• fix: passing kubernetes compute pool option by @saikonen in #1968
• feature: better bootstrap dependencies by @saikonen in #1972
• fix(MetaflowData): Raise AttributeError on failed __getattr__ access by @mutongx in #1971
• Update version to 2.12.13 by @romain-intel in #1973

New Contributors

• @mutongx made their first contribution in #1971

Full Changelog: 2.12.12...2.12.13

2.12.12

What's Changed

• fix env var by @iamsgarg-ob in #1959
• release 2.12.12 by @savingoyal in #1960

Full Changelog: 2.12.11...2.12.12

2.12.11

What's Changed

• Sanitize path names when creating cache directories in FileCache by @romain-intel in #1931
• comment clean in argo workflows by @savingoyal in #1942
• Capture stack traces from Argo errors by @savingoyal in #1941
• add secrets to exit hook by @savingoyal in #1944
• Fix error capture commands by @iamsgarg-ob in #1945
• adding additional env vars to capture error by @iamsgarg-ob in #1946
• upgrade black by @savingoyal in #1949
• Inject ns by @madhur-ob in #1950
• 2.12.11 release by @savingoyal in #1951

Full Changelog: 2.12.10...2.12.11

2.12.10

Release Notes

This patch release introduces support for emitting run heartbeats for the entire duration of a run's lifecycle on Argo Workflows. Metaflow's UI relies on a complex state machine to ascertain the correct state of any run, step, task, and attempt. For runs executed on Workflow Orchestrators like AWS Step Functions, Argo Workflows, and Airflow, by design - there is no execution-wide supervisor process that can monitor and keep track of the state of the execution for Metaflow. We instead rely on a task-local process to exfiltrate this state to Metaflow - which aids in effortlessly scaling Metaflow to millions of tasks. However, if no task is scheduled for a while, Metaflow's UI might temporarily show the run as failed (red) before correcting it (green) when the task is scheduled, which can be confusing.

With the latest release, executions on Argo Workflows can kick off a daemon process that is alive for the entire duration of the execution. This daemon process will ensure that a liveness signal is emitted reliably throughout the entire duration of the execution. You can enable this by simply adding the --enable-heartbeat-daemon flag -

python flow.py argo-workflows create --enable-heartbeat-daemon

The next release will enable this functionality as default. If this turns out to be useful for you, or if you have any feedback, ping us at chat.metaflow.org!

2.12.9

What's Changed

• [card] bug fix in error card rendering by @valayDave in #1923
• [parallel-fixes] tag-catch test and @Secrets simplification for @parallel by @valayDave in #1917
• [parallel-fixes] core + test changes by @valayDave in #1925
• [refactor-jobsets] refactor to new implementation by @valayDave in #1914
• [argo] support for @parallel by @valayDave in #1927
• [ubf] bug fix when using merge_artifacts in UBF joins by @valayDave in #1928
• [Ready for Review] Improve native resume by @darinyu in #1884
• deployer with new injection mechanism by @madhur-ob in #1910
• [flowspec] add artifacts to exclude in merge_artifacts by @valayDave in #1929
• new release 2.12.9 by @savingoyal in #1933

Full Changelog: 2.12.8...2.12.9

2.12.8

What's Changed

• pin micromamba version by @savingoyal in #1920
• bump to 2.12.8 by @savingoyal in #1921

Full Changelog: 2.12.7...2.12.8

2.12.7

Improvements

Fix Argo Events escaping HTML characters

This release fixes an issue where values passed to flow parameters from an Argo event would unintentionally escape HTML characters. The value in the payload should now be passed as-is to the flow parameter.

What's Changed

• fix: do not escape html in Argo Events payload by @saikonen in #1911
• Fix issue with current_namespace when the namespace is None by @romain-intel in #1901
• Expose metaflow logger and monitor via singleton by @talsperre in #1794
• remove old py versions for s3 tests by @savingoyal in #1906
• bump version to 2.12.7 by @saikonen in #1912

New Contributors

• @talsperre made their first contribution in #1794

Full Changelog: 2.12.6...2.12.7

2.12.6

Improvements

Fix Argo Workflows issue with long static splits

This release fixes an issue where a join step of a static split would fail on Argo Workflows in rare cases where the length of step names exceeded a threshold.

Argo Events support for parameter names with dashes

Fixes an issue where values from an Argo Event payload did not correctly map to flow Parameters if the parameter name contained dashes.

More specific PyPI errors for package resolving

The errors for @pypi should now be more clear in cases where it is unable to successfully resolve an environment due to not finding a suitable package.

What's Changed

• Bump braces from 3.0.2 to 3.0.3 in /metaflow/plugins/cards/ui by @dependabot in #1898
• feature: polish pypi package errors by @saikonen in #1905
• fix: long names with static split fails join step on Argo Workflows by @saikonen in #1907
• feature: Conda env extension hooks by @saikonen in #1902
• fix: support for dashed parameters through argo events by @saikonen in #1908
• bump version to 2.12.6 by @saikonen in #1909

Full Changelog: 2.12.5...2.12.6

2.12.5

What's Changed

• add timestamps to conda debug logs by @savingoyal in #1889
• fix: decorator attributes being modified in get_environment by @saikonen in #1895
• bump version to 2.12.5 by @saikonen in #1896
• [cards] bug fix with error card renders by @valayDave in #1893

Full Changelog: 2.12.4...2.12.5

2.12.4

What's Changed

• Fix escape hatch to make getattr behavior more standard by @romain-intel in #1883
• Fix/remember namespace by @romain-intel in #1873
• Remove global nature of parameters and flow decorators by @romain-intel in #1886
• Install PyPI packages in interpreter's site packages by @savingoyal in #1890
• bump version to 2.12.4 by @saikonen in #1891

Full Changelog: 2.12.3...2.12.4