Scaling capability of Numaflow

[samirsss]

Hi,

We're evaluating Numaflow for some of our usecases and we see a lot of potential of Numaflow for a lot of additional flows that we have.

We want to use the capabilities like following:

1. Dynamically adding sources for debugging or more insight into the pipeline
2. Visually seeing the entire pipeline and its performance and see where the bottleneck is
3. Dynamically changing the Vertex counts.
4. Having extensive metrics for monitoring and observability.

What's lacking at the moment

1. Wider adoption at the moment and its very understandable - since its a relatively new framework
2. Lot of empirical data on scaling/pipeline complexity. This is the main idea i wanted to ask if we can share some comparative study to show how Numaflow is better than Apache Flink to highlight its strength.
3. More medium articles to highlight use-cases that Numaflow was used for. This can be done by using the community of developers and encouraging them to post their blogs. I can commit to writing one up when we go to production and have atleast some use of the framework.

This would really help in adoption and hence build more enhancements and features.

Thanks!

[vigith]

thank you @samirsss for trying out Numaflow and giving us the feedback.

This is the main idea i wanted to ask if we can share some comparative study to show how Numaflow is better than Apache Flink to highlight its strength.

Absolutely, please share the results.

[samirsss]

Hi @vigith - can we have some comparative data being published by Numaflow core team too? something like https://www.timescale.com/blog/timescale-cloud-vs-amazon-rds-postgresql-up-to-350-times-faster-queries-44-faster-ingest-95-storage-savings-for-time-series-data/ or https://www.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/

It really helps decision making for any early adopters.

[vigith]

We will write some doc around Numaflow vs. Flink. Please keep in mind that Numaflow was not written to replace Flink, though it supports a lot of streaming capabilities. Main aim of Numaflow is to make stream processing easily accessible to non-data engineers and also for non-jvm-based languages.

[syayi]

Hi @samirsss! Thanks for trying out Numaflow, sharing the feedback and being open to share your use case to the broader community. Would you open to having a chat with the Numaflow team sometime?

[samirsss]

@syayi - We can definitely connect in a few weeks once we have a solid understanding of how we will use Numaflow. Please bear with me till then.

[syayi]

Absolutely, @samirsss! Looking forward to our chat and would love to hear your thoughts on Numaflow