Ballista is a distributed SQL query engine powered by the Rust implementation of Apache Arrow and DataFusion.
If you are looking for documentation for a released version of Ballista, please refer to the Ballista User Guide.
Ballista implements a similar design to Apache Spark (particularly Spark SQL), but there are some key differences:
- The choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic processing times.
- Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized processing (SIMD) and efficient compression. Although Spark does have some columnar support, it is still largely row-based today.
- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of distributed compute.
- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged efficiently between executors using the Flight Protocol, and between clients and schedulers/executors using the Flight SQL Protocol
- Supports HDFS as well as cloud object stores. S3 is supported today and GCS and Azure support is planned.
- DataFrame and SQL APIs available from Python and Rust.
- Clients can connect to a Ballista cluster using Flight SQL.
- JDBC support via Arrow Flight SQL JDBC Driver
- Scheduler web interface and REST UI for monitoring query progress and viewing query plans and metrics.
- Support for Docker, Docker Compose, and Kubernetes deployment, as well as manual deployment on bare metal.
We run some simple benchmarks comparing Ballista with Apache Spark to track progress with performance optimizations. These are benchmarks derived from TPC-H and not official TPC-H benchmarks. These results are from running individual queries at scale factor 10 (10 GB) on a single node with a single executor and 24 concurrent tasks.
The tracking issue for improving these results is #339.
The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.
Ballista supports a wide range of SQL, including CTEs, Joins, and Subqueries and can execute complex queries at scale.
Refer to the DataFusion SQL Reference for more information on supported SQL.
Ballista is maturing quickly and is now working towards being production ready. See the following roadmap for more details.
There is an excellent discussion in apache#30 about the future of the project, and we encourage you to participate and add your feedback there if you are interested in using or contributing to Ballista.
The current focus is on the following items:
- Make production ready
- Shuffle file cleanup
- Periodically (#185)
- Add gRPC & REST interfaces for clients/UI to actively call the cleanup for a job or the whole system
- Fill functional gaps between DataFusion and Ballista
- Improve task scheduling and data exchange efficiency
- Better error handling
- Scheduler restart
- Improve monitoring, logging, and metrics
- Auto scaling support
- Better configuration management
- Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support sharding for scalability and more efficient caching.
- Shuffle file cleanup
- Shuffle improvement
- Scheduler Improvements
- All-at-once job task scheduling
- Executor deployment grouping based on resource allocation
- Cloud Support
- Performance and scalability
- Python Support
- Support Python UDFs (#173)
There are currently no up-to-date architecture documents available. You can get a general overview of the architecture by watching the Ballista: Distributed Compute with Rust and Apache Arrow talk from the New York Open Statistical Programming Meetup (Feb 2021).
Please see the Contribution Guide for information about contributing to Ballista.