Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Getting Data section with Ibis #104

Open
ncclementi opened this issue Sep 12, 2024 · 2 comments
Open

Add Getting Data section with Ibis #104

ncclementi opened this issue Sep 12, 2024 · 2 comments

Comments

@ncclementi
Copy link

Hello, I work on Ibis and with the default backend (duckdb) we can query overture maps very easily.

Would you be open to a PR contribution to your docs, to add this to the "Getting Data Section"? I can work on multiple examples, but to start it would be something like this.

# install instructions
$ pip install 'ibis-framework[duckdb,geospatial]'
import ibis
from ibis import _

# by default it gets duckdb backend
con = ibis.get_backend()

# example of how to get infrastructure data of `subtype="power"` for a `bbox` around Washington DC 
# and selecting only a few columns

url = "s3://overturemaps-us-west-2/release/2024-07-22.0/theme=base/type=infrastructure/*"
t = con.read_parquet(url, table_name="infra-power-dc")

# so far no execution 
expr = t.filter(_.bbox.xmin > -77.119795,
                _.bbox.xmax < -76.909366,
                _.bbox.ymin > 38.791631,
                _.bbox.ymax < 38.995968,
                _.subtype=="power"
                ).select(["names",
                          "geometry", 
                          "sources", 
                          "class"])

con.to_parquet(expr, "infra-power-dc-ibis.parquet")

t_power_dc = con.read_parquet("infra-power-dc-ibis.parquet") 

# turn interactive mode to preview data
ibis.options.interactive = True   

# inspect data where `class=="power_line", and select only few columns

t_infra.filter(t_infra["class"]=="power_line")["names", "geometry","class"]

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ namesgeometryclass      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ struct<primary: string, common: map<string, string>, rules: array<struct<varian… │ geospatial:geometrystring     │
├──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼────────────┤
│ NULL<LINESTRING (-77.113 38.805, -77.113 38.805, -77.112 38.806, -77.111 38.806,...>power_line │
│ NULL<LINESTRING (-77.11 38.818, -77.11 38.818, -77.111 38.818, -77.112 38.819, -...>power_line │
│ NULL<LINESTRING (-77.098 38.848, -77.097 38.848, -77.097 38.848, -77.097 38.848,...>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.097 38.848, -77.096 38.848, -77.095 38.847, -77.094 38.847,...>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.059 38.796, -77.061 38.796)>power_line │
│ NULL<LINESTRING (-77.06 38.799, -77.06 38.799, -77.061 38.797, -77.061 38.796)>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.057 38.795, -77.059 38.796)>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.055 38.797, -77.055 38.796, -77.055 38.796, -77.055 38.795,...>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.055 38.797, -77.055 38.796, -77.055 38.795, -77.057 38.795)>power_line │
│ {'primary': '230kV transmission line', 'common': None, ... +1}                   │ <LINESTRING (-77.087 38.844, -77.086 38.844, -77.085 38.844, -77.084 38.844,...>power_line │
│ …                                                                                │ …                                                                                │ …          │
└──────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┴────────────┘                     
@danabauer
Copy link
Contributor

danabauer commented Sep 12, 2024

Hi @ncclementi! I'm familiar with Ibis and I think it'd be great to get an example in our docs. I suggest opening a PR to add it to the examples section. Thanks in advance for your contribution and let me know if you have any questions along the way.

@ncclementi
Copy link
Author

Awesome, it sounds great, I would do that then. I'm currently hitting a bug with duckdb but as soon as I get it sort it out I'll work on the examples page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants