Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/explorer v2 #343

Merged
merged 50 commits into from
Sep 13, 2024
Merged

Feature/explorer v2 #343

merged 50 commits into from
Sep 13, 2024

Conversation

rwiker
Copy link
Contributor

@rwiker rwiker commented Sep 12, 2024

Issue

Short description

  • Most of the functionality is now in the class SearchContext (something similar was done earlier, but the idea has been extended and generalized.)
  • Actual search is done as late as possible.
  • Searching only picks up document uuids. A PIT is used transparently while collecting the full set of uuids.
  • Accessing document contents first looks in a cache; if the document is not cached, it is fetched from Sumo.
  • When a document is fetched from Sumo, up to 100 documents are actually fetched (the documen that is actually asked for, plus up to 99 documents following it in the list of uuids.)
  • Filtering has been generalized, and is now described via a table.
  • Document property access has been generalized, and is described via a table.
  • The full set of property values is now extracted via composite aggregation. A PIT is used transparently while collecting the values.
  • iteration and realization are now available as virtual objects (this will be changed as soon as these have been added as actual, stored and indexed objects).
  • The filter method can now take a complex parameter, where custom elasticsearch queries can be specified.
  • Filters can be specified in arbitrary order. Example: exp.cubes.cases now returns a set of cases that contain cubes.
  • .cases, .iterations and .realizations now returns cases, realizations and iterations present in a selection.

Pre-review checklist

  • Read through all changes. No redundant print() statements, commented-out code, or other remnants from the development. 👀
  • Make sure tests pass after every commit. ✅
  • New/refactored code is following same conventions as the rest of the code base. 🧬
  • New/refactored code is tested. ⚙
  • Documentation has been updated 🧾

Pre-merge checklist

  • Commit history is consistent and clean. 👌

Raymond Wiker added 30 commits September 12, 2024 12:40
New class: SearchContext (intended as replacement for DocumentCollect…
2b7102e 
…ion and ChildCollection).
WIP.
b88fdea
WIP: tests/test_explorer.py now passing.
43dc31b
WIP: more systematic approach to accessors (properties).
c873de5
WIP: refactor accessor (@Property) setup for Document/Case/Child.
0d48ea3
WIP: more refactoring.
8c28a24
No need to copy the metadata in order to look up an attribute.
58d9593
Use raw strings for regular expressions.
61e80a3
Copied .timestamps and .intervals into class SearchContext, and corre…
95fe866 
…cted implementation for .timestamps. Also, added complex filter.
Set size to 0 when aggregating intervals (only interested in aggregat…
5aca4f6 
…ions).
(Re-)implemented "has" filter.
07dec52
Corrected return values for filter generation.
9cab87f
Add forwarding filter() method to Explorer.
a27ad40
Implement caching and prefetch for class SearchContext.
8d94643
Smarter implementation of __len__ and async version. Add maxhits para…
7b46983 
…meter to __search_all and async version to make _maybe_refresh work better.
Simpler and faster test for test_get_cases.
378f131
Speedup for get_*_by_uuid
5b4cc10
WIP: optimize searches.
99d7713
Cleanup, and add optimizations to async operations.
96e49b0
Remove (most of) the documentation for the constructor parameter keep…
5337f02 
…_live for Explorer class.
Moved 'contexts' into Filters.
b9378a9
Bug fix: need different tests for key inclusion in dicts and lists.
c660b83
Added virtual objects for iteration and realization.
b01ac52 
Deleted class-specific collection classes, as these are now handled by SearchContext.
Black formatting.
a557a71
Make Explorer, Case, Iteration and Realization inherit from SearchCon…
6dc95f2 
…text so they can be used for further filtering.
Typo.
8cb2a07
Make Case.iterations and Case.iterations_async great again.
a8d9cac
Generate documentation for SearchContext.filter from filter specifica…
7a7f8f8 
…tion.
Auto-generate bucket methods (.names, .columns, etc), rather than wri…
b6ce486 
…ting a lot of near-identical code.
Use simple aggregation rather than composite for fast path bucket agg…
533b3c5 
…regation.
equinor-ruaj
equinor-ruaj reviewed Sep 12, 2024
View reviewed changes
src/fmu/sumo/explorer/objects/_search_context.py Outdated Show resolved Hide resolved
examples/explorer2.ipynb
"metadata": {},
"outputs": [],
"source": [
"print(json.dumps(myrealization.metadata, indent=2))"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get SearchContext object has not attribute metadata

src/fmu/sumo/explorer/objects/realizations.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/_search_context.py Show resolved Hide resolved
src/fmu/sumo/explorer/explorer.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/iteration.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/iterations.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/iterations.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/realization.py Outdated Show resolved Hide resolved
src/fmu/sumo/explorer/objects/_search_context.py Outdated Show resolved Hide resolved
equinor-ruaj
equinor-ruaj approved these changes Sep 13, 2024
View reviewed changes
Copy link
Contributor

@equinor-ruaj equinor-ruaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@rwiker rwiker merged commit 5b924b5 into main Sep 13, 2024
25 checks passed
@rwiker rwiker deleted the feature/explorer-v2 branch September 13, 2024 11:34
This was linked to issues Sep 16, 2024
Change how the Explorer handles PIT #254
Closed
Implement more "advanced" filtering mechanism #250
Closed
Feature: Make xtgeo dependency optional #335
Closed
Advanced filters #334
Closed
The Case.iterations_async property does not return all iterations #326
Closed
Represent Realization and Iteration as virtual objects #312
Closed
Filter cases by object type, content etc. #324
Closed
Investigate the need for all packages on explorer #296
Closed
All column names not contained in columns attribute #255
Closed
@equinor-ruaj equinor-ruaj mentioned this pull request Sep 16, 2024
Closed
This was unlinked from issues Sep 16, 2024
Implement more "advanced" filtering mechanism #250
Closed
Change how the Explorer handles PIT #254
Closed
All column names not contained in columns attribute #255
Closed
Investigate the need for all packages on explorer #296
Closed
Represent Realization and Iteration as virtual objects #312
Closed
Filter cases by object type, content etc. #324
Closed
The Case.iterations_async property does not return all iterations #326
Closed
Advanced filters #334
Closed
Feature: Make xtgeo dependency optional #335
Closed
@equinor-ruaj equinor-ruaj linked an issue Sep 16, 2024 that may be closed by this pull request
Represent Realization and Iteration as virtual objects #312
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@equinor-ruaj equinor-ruaj equinor-ruaj approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Represent Realization and Iteration as virtual objects
2 participants
@rwiker @equinor-ruaj