"Real" JOIN support

[th0ger]

Hey Numaflow!
I'm new but have read the docs to evaluate if this is something for us.
I'm wondering how to do "real" stream JOINs (in SQL terms). You have a JOIN reduce operator, but it assumes identical schemas and the use cases provided are more like a UNION (in SQL terms).

How would you go about lookups between two streams? Like https://developer.confluent.io/tutorials/join-a-stream-to-a-table/ksql.html

Or timeinterval range joins? Like https://docs.databricks.com/en/optimizations/range-join.html

Thanks.

[vigith]

Numaflow is a bare platform that supports building Streaming operators (UDFs). Numaflow doesn't have an out-of-the-box SQL kind of JOIN operator yet.

You have a JOIN reduce operator, but it assumes identical schemas and the use cases provided are more like a UNION (in SQL terms).

It does not assume identical schema but will look for the same keys to join elements in two streams. It is up to the operator/UDF to do what seems fit for their use case.

How would you go about lookups between two streams? Like https://developer.confluent.io/tutorials/join-a-stream-to-a-table/ksql.html

The same keys will always end up in the same operator/UDF. It is not like a lookup, rather you can say something like a streaming merge.

Or timeinterval range joins? Like https://docs.databricks.com/en/optimizations/range-join.html

As of today we assume that both streams have the same time order, so we do not support comparing a stream with older time with a more current stream.
Just curious, does Databricks support out-of-time time interval range joins for streaming sources?

[th0ger]

By coincidence I stumbled about the Contributor Guide > Specifications > Side Inputs. This is sort of the lookup behavior I was referring to in the first use case with kafka ksql, since they take one stream and lookup keys in the (latest version) of a slower stream.
Is this something being worked on?

[vigith]

Yes, you can use side-input to broadcast changes and then do lookups in your user-defined function. The side-input is a working feature; the key point is that we do not have an out-of-the-box operator for lookups, and you will have to write a UDF yourself. Writing UDFs in Numaflow is quite straightforward and easy.

Also, the side-input data is generated by a custom user-defined function.