Possibility of supporting DuckDB as a loader!

[shaunakv1]

I have been experimenting with https://duckdb.org/ recently and have realized it's an incredibly powerful tool. It already supports a JavaScript client. Apart from the it's own file database format, it seamlessly allows super high performance query on other formats directly such as CSV, parquet files etc. It also supports having that data in S3 and Azure blobs.

Makes me wonder if having DuckDB as a loader which can support it's queries would be possible. Any thoughts?

Some use-cases I can think of are:

1. Cross format Joins:

Say I have a few parquet files, csv file and a postgres database. DuckDB allows me to make a cross database joins and run analysis. Would be slick if loader.gl can support querying this.

2. Bring SQL to land of loader.gl

3. DuckDB via it's extensions have full support for postgis-esq geospatial queries. Even supports h3 etc. This would fit right into rendering scenarios on Deck.gl without having to wrangle all data into specific loader format.

[ibgreen]

@shaunakv1 Thanks for engaging.

Yes DuckDB is on our radar, however current DuckDB-related focus is on improving support for Apache Arrow formatted output across the vis.gl stack (loaders.gl/deck.gl/ ... ) which should make everything work very efficiently with DuckDB's Arrow output (binary, zero-copy data handling).

I have (perhaps incorrectly) been assuming that applications that want to use DuckDB would like to use its API directly...

So, I am not quite sure how a DuckDB wrapper in loaders.gl would look. Maybe you can give some pseudo-code examples of how such a loader would be used?

import {DuckDBLoader} from `@loaders.gl/duckdb`;
import {load} from '@loaders.gl/core';

const table = load('SELECT * from ...`, DuckDBLoader);
// How would we get table data into DuckDB before the select?
```;