Skip to content

Latest commit

 

History

History
121 lines (84 loc) · 6.36 KB

README.md

File metadata and controls

121 lines (84 loc) · 6.36 KB
Raw

Build status Doc reference

Materialize is the streaming data warehouse.

Materialize lets you ask questions of your streaming data, and get the current answers in real time – offering the power and flexibility of a SQL data warehouse for the world of real-time data. Materialize is powered by timely dataflow.

Materialize computes correct answers. We support a large fraction of PostgreSQL, and are actively working on supporting more builtin PostgreSQL functions. Please file an issue if there's something that you expected to work that didn't!

Get data in

Materialize reads Avro, Protobuf, JSON, and newline-delimited text. Need something else? Just ask.

Materialize can read data from Kafka topics or tail local files.

Coming soon:

  • Support for AWS Kinesis streams and Azure Event Hub
  • Reading ORC and Parquet files on object storage (i.e. join your streams against your data lake in real-time)
  • Getting data in from arbitrary HTTP endpoints

Transform, manipulate, and read your data

Once you've got the data in, define views and perform reads via the PostgreSQL protocol. Use your favorite PostgreSQL CLI, including the psql you probably already have on your system.

Materialize supports a comprehensive variery of SQL features, all using the PostgreSQL dialect and protocol:

  • Joins, Joins, Joins! Materialize supports multi-column join conditions, multi-way joins, self-joins, cross-joins, inner joins, outer joins, etc.
  • Delta-joins avoid intermediate state blowup compared to systems that can only plan nested binary joins - tested on joins of up to 64 relations.
  • Support for subqueries. Materialize's SQL optimizer performs subquery decorrelation out-of-the-box, avoiding the need to manually rewrite subqueries into joins.
  • Materialize supports streams that contain CDC data (currently supporting the Debezium format). Materialize can incrementally maintain views in the presence of arbitrary inserts, updates, and deletes. No asterisks.
  • All the aggregations. GROUP BY , MIN, MAX, COUNT, SUM, STDDEV, HAVING, etc.
  • ORDER BY
  • LIMIT
  • DISTINCT
  • JSON support in the PostgreSQL dialect including operators and functions like ->, ->>, @>, ?, jsonb_array_element, jsonb_each. Materialize automatically plans lateral joins for efficient jsonb_each support.
  • Nest views on views on views!
  • Multiple views that have overlapping subplans can share underlying indices for space and compute efficiency, so just declaratively define what you want, and we'll worry about how to efficiently maintain them.

Just show us what it can do!

Here's an example join query that works fine in Materialize, TPC-H query 15:

-- Views define commonly reused subqueries.
CREATE VIEW revenue (supplier_no, total_revenue) AS
    SELECT
        l_suppkey,
        SUM(l_extendedprice * (1 - l_discount))
    FROM
        lineitem
    WHERE
        l_shipdate >= DATE '1996-01-01'
        AND l_shipdate < DATE '1996-01-01' + INTERVAL '3' month
    GROUP BY
        l_suppkey;

-- Materialized views are maintained automatically, and can depend on non-materialized views.
CREATE MATERIALIZED VIEW tpch_q15 AS
  SELECT
    s_suppkey,
    s_name,
    s_address,
    s_phone,
    total_revenue
FROM
    supplier,
    revenue
WHERE
    s_suppkey = supplier_no
    AND total_revenue = (
        SELECT
            max(total_revenue)
        FROM
            revenue
    )
ORDER BY
    s_suppkey

Stream inserts, updates, and deletes on the underlying tables (lineitem and supplier), and Materialize keeps the materialized view incrementally updated. You can type SELECT * FROM tpch_q15 and expect to see the current results immediately!

Get data out

Pull based: Use any PostgreSQL-compatible driver in any language/environment to make SELECT queries against your views. Tell them they're talking to a PostgreSQL database, they don't ever need to know otherwise.

Push based: Or configure Materialize to stream results to a Kafka topic as soon as the views change.

If you want to use an ORM, chat with us. They're surprisingly tricky.

Get started

Check out our getting started guide.

Documentation

Check out our documentation.

License

Materialize is source-available and licensed under the BSL 1.1, converting to the open-source Apache 2.0 license after 4 years. As stated in the BSL, Materialize is free forever on a single node.

Materialize is also available as a paid cloud service with additional features such as high availability via multi-active replication.

How does it work?

Materialize is built on top of differential dataflow and timely dataflow, and builds on a decade of cutting-edge stream processing research.

For developers

Materialize is written entirely in Rust.

Developers can find docs at doc/developer, and Rust API documentation is hosted at https://mtrlz.dev/api/. The Materialize development roadmap is divided up into roughly month-long milestones, and managed in GitHub.

Contributions are welcome. Prospective code contributors might find the good first issue tag useful. We value all contributions equally, but bug reports are more equal.

Credits

Materialize is lovingly crafted by a team of developers and one bot. Join us.