Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_parquet_dask load_divisions with bounds #63

Open
brl0 opened this issue Feb 12, 2021 · 0 comments
Open

read_parquet_dask load_divisions with bounds #63

brl0 opened this issue Feb 12, 2021 · 0 comments

Comments

@brl0
Copy link
Collaborator

brl0 commented Feb 12, 2021

While using read_parquet_dask to read files written with pack_partitions_to_parquet method, passing bounds and load_divisions=True causes a KeyError. Reading the same file with one option or the other works.

Example:

from spatialpandas.io import read_parquet_dask
sdf = read_parquet_dask(path, bounds=bounds, load_divisions=True)

Error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3079             try:
-> 3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: -1

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-22-56d4a7063dd6> in <module>
----> 1 sdf_div = read_parquet_dask(pre_path, bounds=bounds, load_divisions=True)
      2 sdf_div

/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/spatialpandas/io/parquet.py in read_parquet_dask(path, columns, filesystem, load_divisions, geometry, bounds, categories)
    231         path, columns, filesystem,
    232         load_divisions=load_divisions, geometry=geometry, bounds=bounds,
--> 233         categories=categories
    234     )
    235 

/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/spatialpandas/io/parquet.py in _perform_read_parquet_dask(paths, columns, filesystem, load_divisions, geometry, bounds, categories)
    371 
    372     if load_divisions:
--> 373         divisions = div_mins + [div_maxes[-1]]
    374         if divisions != sorted(divisions):
    375             raise ValueError(

/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/pandas/core/series.py in __getitem__(self, key)
    822 
    823         elif key_is_scalar:
--> 824             return self._get_value(key)
    825 
    826         if is_hashable(key):

/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/pandas/core/series.py in _get_value(self, label, takeable)
    930 
    931         # Similar to Index.get_value, but we do not fall back to positional
--> 932         loc = self.index.get_loc(label)
    933         return self.index._get_values_for_loc(self, loc, label)
    934 

/home/conda/store/816032bda5816104d47e08949e4ec085fd6b9a98be07c2a55cf29c652743653e-datum/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:
-> 3082                 raise KeyError(key) from err
   3083 
   3084         if tolerance is not None:

KeyError: -1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant