Cannot save as Shapefile

[felix-schott]

Environment Information

• open_buildings version: 0.10.0
• Python version: 3.11
• Operating System: Debian Bullseye (docker image)

Description

Trying to save the output as a shapefile fails, see command and traceback below.

What I Did

$ echo '{ "type": "Feature", "properties": {}, "geometry": {"coordinates": [[[-0.13085471468215815, 51.50945096318702], [-0.13085471468215815, 51.50612362847875], [-0.12508113856225123, 51.50612362847875], [-0.12508113856225123, 51.50945096318702], [-0.13085471468215815, 51.50945096318702]]], "type": "Polygon"}}' | ob get_buildings - buildings.shp --country_iso GB 
[2023-10-14 10:12:02] Querying and downloading data for quadkey 0313131311 in country GB...
[2023-10-14 10:12:02] Expect query times of at least 5-10 seconds
[2023-10-14 10:12:02] Installing DuckDB spatial extension...
[2023-10-14 10:12:56] Downloaded 65 features into DuckDB.
[2023-10-14 10:12:56] Writing to buildings.shp...
terminate called after throwing an instance of 'duckdb::IOException'
  what():  IO Error: Could not write file "buildings.dbf": Bad file descriptor
Aborted (core dumped)

[mtravis]

@felix-schott I've just had a go at this and can't reproduce the error.

Environment Information

• open_buildings version: 0.10.0
• Python version: 3.10
• Operating System: Pop!_OS 22.04

Is it possible you don't have write access to the folder you're outputting to?

[felix-schott]

Hi @mtravis,

Thanks for trying to reproduce. I have write access and I can output to all other formats. It even successfully produces buildings.shp and buildings.shx but fails at buildings.dbf. The "bad file descriptor" error seems to indicate that the file gets prematurely closed which might be a DuckDB bug in this case. I do run this in a docker container (see #44) but I don't think that would affect it.

Good to know it's dependent on environment. #45 makes it more difficult to reproduce, do you mind sharing what version of the duckdb package you're using? I'm on 0.9.1.

[cholmes]

Thanks for reporting @felix-schott - I also wasn't able to reproduce, but I do feel like I may have seen a similar error once before.

I'm on duckdb 0.9.1 too (os/x and python 3.11).

It's likely a duckdb problem, should just try to write any shapefile out from duckdb.

[felix-schott]

Hi guys,
I filed a bug report with the duckdb-spatial maintainers (duckdb/duckdb_spatial#144). Let's see what they say and maybe leave this issue open for now as it may require action on our side to resolve it (e.g. update to a patched duckdb version if it is confirmed to be a bug).

[mtravis]

@felix-schott great, looks like it's not specific to just shapefile but other formats that use .db

duckdb/duckdb_spatial#141 (comment)

[mtravis]

@felix-schott also, I haven't upgraded to DuckDB 0.9.1 as of yet.

[cholmes]

@felix-schott (or anyone else) - If you want to chat in general we're starting to use the Cloud Native Geo Foundation slack can join at https://join.slack.com/t/cloudnativegeo/shared_invite/zt-235w8flfo-TW5Tpi1sPqQFWeMy~7ROHA (or let me know your email and I can invite you).

[felix-schott]

Cheers, I joined :)

[Maxxen]

Hi! I think I have a partial fix for this in duckdb/duckdb_spatial#145, at least ShapeFiles, GeoJSONSeq and formats not using GDAL's internal thread pool should work.

[Maxxen]

Hi! Just letting you guys know we just got distribution of development builds for spatial working, so you should be able to install the latest spatial version for duckdb v0.9.1 (where this is fixed) by executing:

FORCE INSTALL spatial FROM 'http://nightly-extensions.duckdb.org';

We will probably backport the fixes to the latest "stable" build you'd get by default when you call INSTALL spatial soon-ish too.