DuckDb Init Scripts

[kevgliss]

I'm trying to bootstrap a given harlequin environment using duckdb and init scripts. My init script looks like:

INSTALL httpfs;
LOAD httpfs;
INSTALL aws;
LOAD aws;

CREATE SECRET aws_secret (
    TYPE S3,
    PROVIDER CONFIG,
    KEY_ID 'XXX',
    SECRET 'XXX',
    SESSION_TOKEN 'XXXX'
)

CREATE VIEW s3_data AS SELECT * FROM read_json_auto('s3://path-to-file');

The above works fine, when entered in the query window after harlequin has already been loaded. But bootstraping via init doesn't seem to work. Any ideas? There are no errors afaict just no view. I'm passing the init via --init-path and my config via --config-path the config itself isn't special:

[profiles.particle-s3]
limit = 200_000
adapter = "duckdb"

[tconbeer]

I think you need a semicolon between the last 2 statements?

[kevgliss]

You're absolutely right, is there an error log somewhere that might have tipped me off? Unfortunately, my setup is still a little more complicated as I am attempting to wrap the harlequin command in my own typer app (via subprocess). For example:

particle harlequin run translates to harlequin -e httpfs -e aws --config-path /some/path/config.toml --init-path /some/path/script.sql Weirdly, the raw command, when run directly, works fine (e.g., the init script is run), but when called via subprocess, it doesn't seem to run even with the correct env and shell=True is set.

If there are any debug logs I could take a look I would be super 😍 .

[tconbeer]

You're probably bumping into this issue: #408

Harlequin's cli options are built dynamically, and the python API probably doesn't work as you expect. If you don't need harlequin to parse the CLI options, you can just use the textual API directly (app=Harlequin(**kwargs); app.run()) inside your typer command