Skip to content

Latest commit

 

History

History
264 lines (202 loc) · 13.4 KB

README.md

File metadata and controls

264 lines (202 loc) · 13.4 KB
Raw

Delta Sharing Server

delta-sharing-rs is a Rust-based Delta Sharing server that includes administration functionality. Unlike the reference implementation of a Delta Sharing server, which primarily focuses on the API specification and uses static file-based sharing information, delta-sharing-rs manages its sharing information through an API.

Supported Platforms

Amazon AWS Google GCP Microsoft Azure
🟩 🟩 🟥

Configure Credentials for Cloud Storage Backends

  1. Amazon AWS

To access the S3 Delta table backend, you need to create an IAM user with an Amazon S3 permissions policy. Once you've created the IAM user, you must configure the profile name and region to allow the Delta Sharing server to access the S3 bucket. The location of the credentials file is specified by the environment variable AWS_SHARED_CREDENTIALS_FILE. If this variable is not set, the credentials file should be located at ~/.aws/credentials. delta-sharing-rs utilizes the Object Store crate with the aws-config feature, which requires the AWS_PROFILE and AWS_REGION environment variables if you use S3 Delta table backend.

  1. Google GCP

To access the GCS Delta table backend, you need to create a GCS service account. The location of the GCP service account private key JSON is specified by the environment variable GOOGLE_APPLICATION_CREDENTIALS. If this variable is not set, the private key JSON file should be located at ~/.gcp/service-account-file.json.

  1. Microsoft Azure

Microsoft Azure backed Delta tables will be supported in the near future.

Starting Delta Sharing Server

Due to the donation on 07/07/2023, we haven't prepared a designated Docker repository offering the latest image of delta-sharing-rs. Therefore, you will need to build your own release manually. To build the release binary, please run the following command in the project directory:

$ just build

The following files might be helpful for you in creating your own docker-compose file.

You can start Delta Sharing using one of the following two options:

1. Docker Hub 2. Docker Compose

Please choose the option that best fits your needs and follow the instructions in the corresponding link to start Delta Sharing. You can also find deployment examples here. Please note that these two repositories are not part of the delta-incubator project due to the donation, but they still maintain the latest image of the official build. This arrangement may change in the future.

Starting the Development Server

Since the implementation is still in the early stages, only the development server is currently available. A Helm chart will be added to the project in the near future.

To run the development server, execute the following commands in this directory:

 $ just docker
 $ just server

To run the unit tests, execute the following commands in this directory:

 $ just docker
 $ just test
 $ just testdb

Create a New Sharing via the API

Once you've started the development server, you can create a new sharing via the API. Follow these steps:

  1. Log in to Delta Sharing server and get the admin access token by running the following command:
 $ curl -s -X POST http://localhost:8080/admin/login -H "Content-Type: application/json" -d '{"account": "deltars", "password": "password"}' | jq '.'
{
  "profile": {
    "shareCredentialsVersion": 1,
    "endpoint": "http://127.0.0.1:8080",
    "bearerToken": "YOUR_ADMIN_ACCESS_TOKEN",
    "expirationTime": "2023-04-09 19:34:04 UTC"
  }
}
  1. Register a new share by running the following command:
  $ curl -s -X POST "http://localhost:8080/admin/shares" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "name": "share1" }' | jq '.'
{
  "share": {
    "id": "6986c361-5e6a-4554-b698-11875d6598e0",
    "name": "share1"
  }
}
  1. Register a new table by running the following command:
 $ curl -s -X POST "http://localhost:8080/admin/tables" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "name": "table1", "location": "s3://delta-sharing-test/examination" }' | jq '.'
{
  "table": {
    "id": "579df9cd-a674-459d-9599-d38d54583cd0",
    "name": "table1",
    "location": "s3://delta-sharing-test/examination"
  }
}
  1. Register a new table as a part of schema1 in the share1 by running the following command:
 $ curl -s -X POST "http://localhost:8080/admin/shares/share1/schemas/schema1/tables" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" -d'{ "table": "table1" }' | jq '.'
{
  "schema": {
    "id": "689ed733-bec8-4796-a2dd-4f82dce6beab",
    "name": "schema1"
  }
}
  1. Issue a new recipient profile by running the following command:
 $ curl -s -X GET "http://localhost:8080/admin/profile" -H "Authorization: Bearer YOUR_ADMIN_ACCESS_TOKEN" -H "Content-Type: application/json" | jq '.'
{
  "profile": {
    "shareCredentialsVersion": 1,
    "endpoint": "http://127.0.0.1:8080",
    "bearerToken": "YOUR_RECIPIENT_ACCESS_TOKEN",
    "expirationTime": "2023-04-09 19:55:19 UTC"
  }
}

Delta Sharing Configuration

All TOML, JSON, YAML, INI, RON, and JSON5 files located in the configuration directory will be loaded as configuration files1. The path to the configuration directory can be set using the DELTA_SHARING_RS_CONF_DIR environment variable. You can also configure Delta Sharing using the corresponding environment variables, which is helpful when setting up a Kubernetes cluster2. Please be sure that the environment variables AWS_SHARED_CREDENTIALS_FILE and GOOGLE_APPLICATION_CREDENTIALS are set properly if necessary. Below is a list of the configuration variables:

Name Environment Variable Required Description
db_url DELTA_SHARING_RS_DB_URL yes URL of PostgreSQL server
server_addr DELTA_SHARING_RS_SERVER_ADDR yes URL of Delys Sharing server which will be used for sharing profile
server_bind DELTA_SHARING_RS_SERVER_BIND yes IP address of Korosiro Sharing server which will be used for Axum server binding
admin_name DELTA_SHARING_RS_ADMIN_NAME yes Default admin user name
admin_email DELTA_SHARING_RS_ADMIN_EMAIL yes Default admin user email
admin_password DELTA_SHARING_RS_ADMIN_PASSWORD yes Default admin user password
admin_namespace DELTA_SHARING_RS_ADMIN_NAMESPACE yes Default admin user namespace
admin_ttl DELTA_SHARING_RS_ADMIN_TTL yes Default admin user access token TTL in seconds
signed_url_ttl DELTA_SHARING_RS_SIGNED_URL_TTL yes Valid duration of signed URL of cloud backends in seconds
jwt_secret DELTA_SHARING_RS_JWT_SECRET yes JWT secret key
use_json_log DELTA_SHARING_RS_USE_JSON_LOG yes If this value set to be true, log outputs in JSON format
log_filter DELTA_SHARING_RS_LOG_FILTER yes Tracing log filter

API

SEE ALSO

Status Official Method URL
✔️ 🟥 GET /swagger-ui
✔️ 🟥 POST /admin/login
✔️ 🟥 GET /admin/profile
✔️ 🟥 GET /admin/accounts
✔️ 🟥 POST /admin/accounts
✔️ 🟥 GET /admin/accounts/{account}
✔️ 🟥 POST /admin/shares
✔️ 🟥 GET /admin/tables
✔️ 🟥 POST /admin/tables
✔️ 🟥 GET /admin/tables/{table}
✔️ 🟥 POST /admin/shares/{share}/schemas/{schema}/tables
🟥 POST /admin/shares/{share}/all-tables
✔️ 🟩 GET /shares
✔️ 🟩 GET /shares/{share}
✔️ 🟩 GET /shares/{share}/schemas
✔️ 🟩 GET /shares/{share}/schemas/{schema}/tables
✔️ 🟩 GET /shares/{share}/all-tables
✔️ 🟩 GET /shares/{share}/schemas/{schema}/tables/{table}/version
✔️ 🟩 GET /shares/{share}/schemas/{schema}/tables/{table}/metadata
✔️ 🟩 POST /shares/{share}/schemas/{schema}/tables/{table}/query
🟩 GET /shares/{share}/schemas/{schema}/tables/{table}/changes

TODO

  • API
    • CDF Related API
    • Microsoft Azure Pre-Signed URL
  • Documentation
    • README
    • Wiki
  • DevOps
    • Dockerfile
  • Admin Console (React/Frontend)
  • Data Access Audit
    • Enrich Access Log
    • Share Namespaces
    • Token Blacklist

References

Official

  1. Delta Sharing: An Open Protocol for Secure Data Sharing

You can find the Delta Sharing open protocol specification here.

  1. Open source self-hosted Delta Sharing server

My blog post on the official Delta Lake community.

Related Projects

  1. Riverbank

This project was primarily started as a kotosiro-sharing project and was highly motivated by the riverbank project. If the preceding riverbank project had not existed, the road of development would have been much harder.

  1. delta-rs

Needless to say, this great Rust crate allowed me low level access to Delta tables in Rust.

Contributing

We encourage you to reach out, and are committed to provide a welcoming community.

Footnotes

  1. An example configuration can also be found at config directory. ↩

  2. When delta-sharing-rs detects duplicated configuration variables, the values from environment variables take precedence over those from configuration files. ↩