v1.2.2

Major Fixes

• Increase key length in message headers [#2223]

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.2/validating-webhook-install.yaml

Full Changelog: v1.2.1...v1.2.2

v1.4.0

Major Features 🔥

• Introducing MonoVertex backed by Rust 🦀
• Introducing gRPC bi-directional streaming between numa container and UDF
• Metrics visualiser for MonoVertex
• Flexible scaling with replicasPerScaleUp and replicasPerScaleDown
• Enabling resourceClaims and readyReplicas for Vertex and MonoVertex
• Rolling Update for both Pipeline and MonoVertex

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version.

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.0/validating-webhook-install.yaml

Other Changes

• docs: updated CHANGELOG.md by @github-actions in #1916
• chore: mono-vertex code review by @vigith in #1917
• feat: add server-info support and versioning to MonoVertex by @kohlisid in #1918
• chore: improve shutdown and health checks for MonoVertex by @yhl25 in #1919
• chore: rename proto name for sourcetransform by @vigith in #1921
• chore: reorganize rust code by @yhl25 in #1922
• Add Lockheed to Users.md by @mdwarne1 in #1934
• chore: add transformer to MonoVertex example by @vigith in #1935
• fix: retry failed messages for MonoVertex sink by @yhl25 in #1933
• chore: implement APIs for mono vertex UI server by @KeranYang in #1931
• chore: update last updated time for mvtx by @yhl25 in #1939
• chore: avoid partial acks by @yhl25 in #1938
• chore: handle drop messages from transformer by @yhl25 in #1942
• feat: autoscaling for MonoVertex by @whynowy in #1927
• fix: default resources mutated when applying templates by @whynowy in #1948
• test: add mono vertex e2e tests by @yhl25 in #1945
• feat: Mono vertex UI by @veds-g in #1941
• chore: add missed tests for java by @KeranYang in #1953
• chore: switch to scratch base image by @BulkBeing in #1955
• feat: enable fallback sink for mvtx by @yhl25 in #1957
• feat: add health for monovertex by @kohlisid in #1954
• fix: replicas derived in UI from mvtx status instead of spec by @veds-g in #1965
• chore: send negative value rateNotAvailable by @kohlisid in #1966
• refactor: re-arrange e2e tests by @KeranYang in #1961
• chore: version update for releasing by @KeranYang in #1944
• docs: updated CHANGELOG.md by @github-actions in #1968
• doc: update roadmap by @vigith in #1970
• chore: fix broken links by @ayildirim21 in #1972
• chore: kustomize transformation config for MonoVertex by @whynowy in #1977
• fix: remove coloring in logs by @vigith in #1975
• fix: minor perf improvements of mvtx fallback sink by @vigith in #1967
• chore: make nightly build pipeline faster by @BulkBeing in #1958
• fix: adding not available for negative processing rates by @veds-g in #1983
• chore: enable current/desired replicas by @whynowy in #1987
• fix: do not pass scale info to MonoVertex by @whynowy in #1990
• chore: metrics and https support for serving by @yhl25 in #1985
• fix: test coverage generation for Rust code by @BulkBeing in #1993
• chore: use numaflow models to create minimum pipeline spec for serving by @yhl25 in #1995
• fix: e2e testing isbsvc deletion timeout issue by @KeranYang in #1997
• chore: fix servesink by @yhl25 in #1999
• chore: disallow MonoVertex replicas less than min or greater than max by @whynowy in #1994
• test: avoid building image twice in e2e testing ci by @whynowy in #2001
• chore(deps): bump micromatch from 4.0.7 to 4.0.8 in /ui by @dependabot in #2002
• feat: more flexible scaling with replicasPerScaleUp and replicasPerScaleDown by @whynowy in #2003
• fix: log format with config load error by @xdevxy in #2000
• chore: mark imageErr as unhealthy by @kohlisid in #2004
• feat: enable resourceClaims for vertex and monovtx by @whynowy in #2009
• chore: refactor metric variable names by @vigith in #2012
• feat: introduce readyReplicas for Vertex and MonoVertex by @whynowy in #2014
• fix: add latency metrics for mvtx by @vigith in #2013
• chore: use RUST_LOG env to control logging in rust binary by @yhl25 in #2017
• chore(deps): bump webpack from 5.93.0 to 5.94.0 in /ui by @dependabot in #2018
• feat: allow configurable retryStrategy by @kohlisid in #2010
• chore: placeholders for rolling update by @whynowy in #2019
• docs: updated CHANGELOG.md by @github-actions in #2021
• chore: added Seekr to USERS.md by @lanedd in #2022
• chore: reduce 5 second wait time every time redis sink check runs in e2e by @KeranYang in #2023
• chore: convert onFailureStrategy to enum in rust by @kohlisid in #2020
• chore: add docs for retry by @kohlisid in #2024
• chore: use kube_codegen.sh instead in codegen. by @xdevxy in #2011
• fix: pause lifecyle changes and add drained status by @kohlisid in #2028
• chore: improve pending logs to be on single line by @vigith in #2037
• feat: rolling update for MonoVertex by @whynowy in #2029
• chore: minor pipeline controller refactor by @whynowy in #2039
• feat: rolling update for Pipeline Vertex by @whynowy in #2040
• chore: pin dependency versions in rust model by @whynowy in #2048
• fix: builtin transformer should keep the keys by @KeranYang in #2047
• fix: Fix numaflow-rs binary location in image by @BulkBeing in #2050
• chore: use readyReplicas to calculate desired replicas by @whynowy in #2052
• chore: remove server info dependencies on go sdk by @KeranYang in #2060
• chore(deps): bump express from 4.19.2 to 4.21.0 in /ui by @dependabot in #2061
• chore: patch instead of update and bugfix by @whynowy in #2059
• fix: skip updating phase for resource check by @kohlisid in #2065
• doc: example for PVC by @vigith in #2067
• feat: allow customization on readyz and livez config by @whynowy in #2068
• chore: suppress some misleading logs from raters by @whynowy in #2075
• fix: support version compatibility check for pre-release versions by @KeranYang in #2069
• Fix: Use Merge patch rather than json patch for pause-timestamp annotation apply by @juliev0 in #2078
• feat: Bidirectional Streaming for User Defined Source by @yhl25 in #2056
• fix: rollback codegen script by @whynowy in #2079
• chore: fix numaflow-rs by @vigith in #2081
• feat: add pause for monovertex by @kohlisid in #2077
• feat: container-type level version compatibility check by @KeranYang in #2087
• feat: Bidirectional Str...

v1.3.3

Major Changes

• Expose containerPort to user defined container (#2135)
• Fixed the race condition of terminating source vertex pods and draining the buffers during pipeline pausing (#2131)
• Create buffers and buckets before updating Vertices (#2112)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.3/validating-webhook-install.yaml

Full Changelog: v1.3.2...v1.3.3

v1.3.2

Major Changes

• Rolling update support for pipeline changes (#2040)
• Allow customization on readyz and livez config (#2068)
• Fixed the issue of keys not being kept in built-in transformers (#2047)
• Bugfix for resources not being orchestrated well for new pipelines with desiredPhase: Paused (#2028)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.2/validating-webhook-install.yaml

Full Changelog: v1.3.1...v1.3.2

v1.3.1

Major Changes

• Add retry mechanism for sinks [#2010]
• Deprecate replicasPerScale, and introduce replicasPerScaleUp and replicasPerScaleDown for more flexible autoscaling strategy [#2003]
• Enable resourceClaims for Pipeline Vertex [#2009]
• Introduce readyReplicas for Vertex status [#2014]

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.1/validating-webhook-install.yaml

Full Changelog: v1.3.0...v1.3.1

v1.3.0

Major Features 🔥

• Introducing Batch Map Capability
• Built-in Jetstream source

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version (internal message format optimizations)

New Features 🔥

• feat: read only view for UI by @veds-g in #1628
• feat: expose controller leader election duration and renew opts by @whynowy in #1657
• feat: controller change for fallback sink by @whynowy in #1664
• feat: Built-in Jetstream source implementation (Closes #1695) by @BulkBeing in #1723
• feat: adding numaflow version to the UI by @veds-g in #1744
• feat(config): standardize boolean value with YAML tags. Fixes #1742 by @charans29 in #1749
• feat: use protobuf to store header and messages in ISB by @yhl25 in #1771
• feat: use protobuf to store wmb in KV by @yhl25 in #1782
• feat: crate for retry with backoff strategy by @vigith in #1785
• feat: implement map batch by @kohlisid in #1778
• feat: enable restful daemon client option for UX server by @whynowy in #1826
• feat: add ObservedGeneration to Pipeline and ISBService Status by @xdevxy in #1799
• feat: use same server-info file for all map modes by @kohlisid in #1828
• feat: add controller and pipeline info metrics by @whynowy in #1855
• feat: pipeline and isbsvc resource health status metrics and detailed vertex type stats by @whynowy in #1856
• feat: new edge path by @veds-g in #1864
• feat: expose replica metrics for ISB Service and Vertex by @whynowy in #1859
• feat: Sync the health status of ISBService, pipeline and vertex by @chandankumar4 in #1860
• feat: Add ObservedGeneration field in vertex status and use it for calculating status by @chandankumar4 in #1892
• feat: Rust k8s model for Numaflow by @vigith in #1898

Major Fixes 🐞

• fix: dedup in user defined source by @yhl25 in #1613
• fix: allow pipeline to start with redis isbsvc (Fixes: #1513) by @QuentinFAIDIDE in #1567
• fix: watermark progression during pods creation/deletion by @yhl25 in #1619
• fix: numaflow package style fixes by @veds-g in #1622
• fix: bug in late message handling for sliding window by @yhl25 in #1471
• fix: race condition while publishing wm inside reduce by @yhl25 in #1599
• fix: readonly view by @veds-g in #1640
• fix: Dedup not working for multi-partitioned edge by @yhl25 in #1639
• fix: flaky TestDropOnFull by @yhl25 in #1647
• fix: avoid publishing watermarks for duplicate messages. by @yhl25 in #1649
• fix: pass headers to transfomer by @yhl25 in #1651
• fix: add headers to custom sinkrequest by @nagarajatantry in #1653
• fix: routing fixes by @veds-g in #1671
• fix: version downgrade for monaco-editor by @veds-g in #1673
• fix: summary bar overlay fix for plugin by @veds-g in #1710
• fix: height fixes to render pipeline view by @veds-g in #1720
• fix: Read from Oldest Offset for Idle Source Kafka e2e by @yhl25 in #1731
• fix: update SDKs to stable image by @samhith-kakarla in #1746
• fix: add retries when writing to redis and resp headers const by @vigith in #1766
• fix: save trait should accept Self as mutable by @vigith in #1795
• fix: value can be null by @vigith in #1831
• fix: should never scale down to < min by @whynowy in #1832
• fix(#1832): scale down to >=min, but not 0 when there's direct back pressure by @whynowy in #1834
• fix: api docs for jetstream service by @KeranYang in #1851
• fix: configure discard policy for WorkQueue/Interest by @kohlisid in #1884

New Contributors

• @QuentinFAIDIDE made their first contribution in #1567
• @nagarajatantry made their first contribution in #1653
• @magelisk made their first contribution in #1733
• @tolmanam made their first contribution in #1729
• @samhith-kakarla made their first contribution in #1746
• @charans29 made their first contribution in #1750

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0/validating-webhook-install.yaml

Full Changelog: v1.2.0...v1.3.0

v1.3.0-rc1

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.3.0-rc1/validating-webhook-install.yaml

Full Changelog: v1.2.1...v1.3.0-rc1

v1.2.1

Major Fixes

• fix: pipeline page rendering issues by @veds-g in #1720

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.1/validating-webhook-install.yaml

Full Changelog: v1.2.0...v1.2.1

v1.2.0

Major Features 🔥

• Introducing Session Window and Streaming Reduce Capability
• Ability to pass headers from Source to Sink
• Fallback Sink

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version
• Python SDK has interface changes.

New Features 🔥

• feat: health status implementation by @kohlisid in #1406
• feat: Session Window and Reduce Streaming by @yhl25 in #1384
• feat: update tcp client connections by @kohlisid in #1429
• feat: enhance autoscaling peeking logic by @akashjkhamkar in #1432
• feat: pipeline health status for UI by @veds-g in #1460
• feat: configure standardResources via controller configmap by @inishchith in #1490
• feat: add support for SASL SCRAM 256 and 512 for Kafka by @Dnomyar in #1518
• feat: noop persistence store for reduce by @yhl25 in #1532
• feat: unaligned wal by @yhl25 in #1511
• feat: numaflow package by @veds-g in #1579
• feat: support headers for message by @yhl25 in #1578
• feat: Fallback Sink by @yhl25 in #1669

Major Fixes 🐞

• fix: terminate reduce vertex pods when pausing pipeline by @dpadhiar in #1481
• fix: configmap const name by @whynowy in #1423
• fix: server-secrets-init container restart by @veds-g in #1433
• fix: Kafka source reads duplicated messages by @jy4096 in #1438
• fix: UI Filter by status for pipelines doesn't work as expected by @inishchith in #1444
• fix: GetDownstreamEdges is not cycle safe by @inishchith in #1447
• fix: memory leak inside session windower by @yhl25 in #1445
• fix: bug where dashed line is treated as permalink + spelling/formatting by @ayildirim21 in #1467
• fix(controller): incorrect cpu/mem resources calculation by @whynowy in #1477
• fix: add idle handler offset nil check by @jy4096 in #1489
• fix: unknown for ISB details in pipeline card by @veds-g in #1497
• fix: avoid panic when ctx is canceled by @yhl25 in #1515
• fix: idle manager refactor for multi partitions by @jy4096 in #1512
• fix: podSpec incorrectly configured in case template exist by @inishchith in #1516
• fix: incorrect json schema for tls config by @whynowy in #1520
• fix(controller): vertex template metadata nil check by @inishchith in #1527
• fix: break from retry loop when key is not found by @yhl25 in #1535
• fix: initialize inflightAcks channel to not nil channel by @afugazzotto in #1548
• fix(codegen): protobuf upgrade by @whynowy in #1558
• fix: error message that cause Buttons overflow in the UI by @a49ibrahim in #1591
• fix: flaky e2e tests by @vigith in #1590
• fix: dedup in user-defined source #1613
• fix: allow pipeline to start with redis isbsvc (Fixes: #1513) #1567
• fix: watermark progression during pods creation/deletion #1619
• fix: numaflow package style fixes in #1622
• fix: bug in late message handling for sliding window #1471
• fix: race condition while publishing wm inside reduce #1599
• fix: read-only view for UI #1628

New Contributors ❤️

• @akashjkhamkar made their first contribution in #1432
• @inishchith made their first contribution in #1444
• @ayildirim21 made their first contribution in #1467
• @syayi made their first contribution in #1510
• @Dnomyar made their first contribution in #1518
• @afugazzotto made their first contribution in #1548
• @a49ibrahim made their first contribution in #1591
• @QuentinFAIDIDE made their first contribution in #1567

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0/validating-webhook-install.yaml

Full Changelog: v1.1.0...v1.2.0

v1.2.0-rc5

Major Changes

• Skip sdk compatibility check if server info does not contain sdk version (#1703)

Pull Requests

v1.2.0-rc4...v1.2.0-rc5

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc5/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc5/validating-webhook-install.yaml