Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add forge methods to acces/query datasets from external sources #367

Draft
wants to merge 50 commits into
base: master
Choose a base branch
from

Commits on Dec 12, 2023

  1. Add web service

    crisely09 committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    cc8b557 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. More changes

    crisely09 committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    f6f853e View commit details
    Browse the repository at this point in the history
  2. Clean code and add tests

    crisely09 committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    47e8464 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. Configuration menu
    Copy the full SHA
    2ff9d7d View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Load local configuration when configuration tests (#370)

    * load local configuration when configuration tests
    
    * keep commons imports together
    
    * fix duplicate keyword argument issue
    
    * use context path from default config in test_to_resource test
    
    * rm extra store_config.pop
    
    * refactor store config
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    380eb07 View commit details
    Browse the repository at this point in the history
  2. Pass view when forge.sparql, forge.elastic, forge.search call (#373)

    * pass view when sparql, elastic call, todo search
    
    * rm unless constants from store
    
    * turn view to endpoint
    
    * endpoint to view param
    
    * rename param
    
    * rename param2
    
    * keyword param from forge call
    
    * missing underscore
    
    * git status
    
    * make endpoint refac
    
    * edit querying notebook to showcase feature, todo set up better view
    
    * refac notebook edit
    
    * change view creation mapping
    
    * check filters not provided as keyword arg
    
    * fix querying notebook, retrieve using resource url
    
    * test make endpoint function
    
    * use *filters for the store interface and implementations
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    b173780 View commit details
    Browse the repository at this point in the history
  3. filter exception fix (#375)

    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    16e1d94 View commit details
    Browse the repository at this point in the history
  4. added timeouts to every requests. call (#368)

    * added timeouts to every requests. call
    
    * centralise default request timeout
    
    * rm import
    
    * use constant in file get
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    5fc964c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    caab280 View commit details
    Browse the repository at this point in the history
  6. deflatten is independent of order of items. If both prop and prop.som…

    …ething it raises an error
    NicoRicardi authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    d6388b8 View commit details
    Browse the repository at this point in the history
  7. removed prints and fixed linting

    NicoRicardi authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    e358acf View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    9ca0398 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    922bfd3 View commit details
    Browse the repository at this point in the history
  10. Update resource schema (#374)

    * change signatures to allow for boolean change_schema on update
    
    * change_schema implemented for single resource update
    
    * refactor batch actions
    
    * lint
    
    * began batch update schema
    
    * change schema many
    
    * progress
    
    * rm / join
    
    * rm useless change
    
    * fix
    
    * change schema to update schema
    
    * update instead of change
    
    * example notebook for schema method, todo update
    
    * notebook example with update
    
    * lint
    
    * improve notebook
    
    * fix one test
    
    * keep unconstrained schema only for update endpoint else _
    
    * same url building in one and many
    
    * add timeout
    
    * schema id optional in update
    
    * rename local parse
    
    * rename keep_unconstrained to use_unconstrained_id
    
    * rm extra docstring param
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    659bc1d View commit details
    Browse the repository at this point in the history
  11. Rm metadata fetch when forge.retrieve (#376)

    * change signatures to allow for boolean change_schema on update
    
    * change_schema implemented for single resource update
    
    * refactor batch actions
    
    * lint
    
    * began batch update schema
    
    * change schema many
    
    * progress
    
    * rm / join
    
    * rm useless change
    
    * fix
    
    * change schema to update schema
    
    * update instead of change
    
    * example notebook for schema method, todo update
    
    * notebook example with update
    
    * lint
    
    * improve notebook
    
    * fix one test
    
    * keep unconstrained schema only for update endpoint else _
    
    * same url building in one and many
    
    * add timeout
    
    * schema id optional in update
    
    * rename local parse
    
    * rm second request for metadata
    
    * add query param annotate only if retrieve source is true
    
    * retrieval error if issue creating resource from response
    
    * rm cross bucket check with source
    
    * add todo
    
    * separate metadata call if cross bucket and source
    
    * refac
    
    * fixes and notebook update
    
    * check deployment endpoint self, may need to be checking multiple values
    
    * revert to self.endpoint
    
    * updated notebook to show retrieval
    
    * fix response name
    
    * add query param annotate only if retrieve source is true
    
    * separate metadata call if cross bucket and source
    
    * rename keep_unconstrained to use_unconstrained_id
    
    * rm extra docstring
    
    * better comments
    
    * clarify comment
    
    * comment fix
    
    * improve markdown
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    76e2f7e View commit details
    Browse the repository at this point in the history
  12. re-do metadata fetch until endpoint is fixed (#381)

    * re-do metadata fetch until endpoint is fixed
    
    * better notebook
    
    * rename variables
    
    * code style
    
    * fix replace in comments
    
    * update comments
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    41503dc View commit details
    Browse the repository at this point in the history
  13. return resources as dictionaries when forge.elastic if as_resource = …

    …False (#382)
    
    * return resource as_json optionally when forge.elastic
    
    * as_resource instead of as_json, default True
    
    * skeleton to enable building resources from different values in the es response payload
    
    * example of forge.elastic as_resource = False in getting started Querying notebook
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    647912a View commit details
    Browse the repository at this point in the history
  14. Set jsonld context version to 1.1 (#387)

    * Set jsonld context version to 1.1
    * this enables non IRI-delimiting character not present in rdflib.plugins.shared.jsonld.context.URI_GEN_DELIMS (e.g '_' in "NCBITaxon": "http://purl.obolibrary.org/obo/NCBITaxon_") to be used when defining jsonld context prefix
    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    c38b3c4 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    a050dc0 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    d6f38de View commit details
    Browse the repository at this point in the history
  17. Fixes pyshacl deep nodeshape path eval error (#393)

    Currently the pySHACL throws a ReportableRuntimeError("Evaluation path too deep!\n{}".format(path_str)) exception when evaluating a shape if the length of its transitive closure of the sh:node property is bigger or equal to 30.
    
    Give a node shape, this PR addresses this by:
    
    * Fixes pyshacl deep nodeshape path eval error by first recursively collecting all the property shapes directly defined (through sh:property) by the node shape or inherited from the node shape parents then link those collected property shapes to the node shape through sh:property
    and finally remove the node shape <-> parent shape relatioinships
    * Aligned the expected data model for a shacl shape between the RDF StoreService and DirectoryService:
    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    ede675a View commit details
    Browse the repository at this point in the history
  18. Fixed coupled of issues (#394)

    * Fixed coupled of issues:
    * forge.prefixes() was raising pandas.Dataframe error "if using all scalar values, you must pass an index"
    * fixed forge.types() to properly collect types from rdfsercice.class_to_shape
    * fixed forge.template() when using an rdf  model based on a store service: Unable to generate template:'tuple' object has no attribute 'traverse'
    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    6b666f1 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    85d6c6f View commit details
    Browse the repository at this point in the history
  20. linting

    NicoRicardi authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    a36f42c View commit details
    Browse the repository at this point in the history
  21. Validation with ontology (#396)

    * Added support for inference when using the RdfModel:
         * support for importing ontologies from schemas using owl:imports
         * use forge.validate(resource, inference="inference_value", type_='AType') with inference_value as in https://github.com/RDFLib/pySHACL/blob/v0.25.0/pyshacl/validate.py#L81. inference_value="rdfs" seems to be enough to extend the resource with the transitive closures of type subClassOf and/or property subPropertyOf relations as per the
                              RDFS entailment rules (https://www.w3.org/TR/rdf-mt/).
    
    * Validation now fails when a type not in the resource is provided as value of the type_ argument unless inference is enabled (with inference='rdfs' for example) and the resource type is a subClassOf of type_
    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    baf26f0 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    f12b465 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    7342b58 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    f31e6cc View commit details
    Browse the repository at this point in the history
  25. Added AttributeError exception when a None resolver is provided to re…

    …cursive_resolve when resolving a str jsonld context (#402)
    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    8106ed8 View commit details
    Browse the repository at this point in the history
  26. Add alternateName to agent resolver (#404)

    * Add alternateName to agent resolver
    
    * Added property also to the agent-to-resource mapping
    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    29280b6 View commit details
    Browse the repository at this point in the history
  27. Add add_image method to Dataset entity (#389)

    * Make add_image method of a Dataset, and not of KnowledgeGraphForge
    
    * Update notebooks with example of dataset.add_image method
    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    3d8a37c View commit details
    Browse the repository at this point in the history
  28. Update prod-forge-nexus.yml

    resolve with target species returns only species and strain returns strains
    NicoRicardi authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    8db6ae8 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    eb6a56d View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    89634dd View commit details
    Browse the repository at this point in the history
  31. Add method when initializing forge to export pyshacl environment vari…

    …able (#406)
    
    * Add method when initializing forge to export environment variable
    
    * Remove addition in setup.py and try os.environ instead of os.system
    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    c5538fe View commit details
    Browse the repository at this point in the history
  32. Remove nexus_sdk from nexus store when uploading files and add `x-nxs…

    …-file-content-length` to header (#403)
    
    * Remove nexus_sdk from nexus store when uploading files
    
    * Change content length header to be
    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    c895b7c View commit details
    Browse the repository at this point in the history
  33. File fetch (#408)

    * split file get call and property access
    
    * split prepare download one
    
    * lint
    
    ---------
    
    Co-authored-by: Leonardo Cristella <[email protected]>
    2 people authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    9476d95 View commit details
    Browse the repository at this point in the history
  34. rm nexus sdk usage and dependency (#407)

    * rm sdk usage from bluebrainexus store file
    
    * rm sdk usage from utils
    
    * rm sdk usage from service
    
    * rm nexussdk from test
    
    * lint
    
    * fix patch
    
    * change usage of project_fetch function for successful patching in tests
    
    * rename module of sdk methods
    
    * remove leftover image from store
    
    * missing file
    
    * restore config
    
    * remove s from file name
    
    * missing the file, again
    
    ---------
    
    Co-authored-by: Cristina E. González-Espinoza <[email protected]>
    2 people authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    536995b View commit details
    Browse the repository at this point in the history
  35. Fix org and project metadata (#409)

    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    c198389 View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    559e3bd View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    aa0cb9a View commit details
    Browse the repository at this point in the history
  38. Fixed building a SPARQL query from a Filter with __ne__ operator and …

    …Literal value (#397)
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    50130ec View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    0b3b91f View commit details
    Browse the repository at this point in the history
  40. Refactor batch request mechanism (#383)

    * Added batched retrieval
    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    7e3844b View commit details
    Browse the repository at this point in the history
  41. rewrite uri as static method (#413)

    ssssarah authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    7848a54 View commit details
    Browse the repository at this point in the history
  42. Check for schema id to use schema endpoint (#415)

    * Check for schema id to use schema endpoint
    
    * Add example in unit test
    
    * Add missing  parameter inside _register_one. Add notebook example of schema handling
    
    * check if resource_id is given
    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    42b6f8c View commit details
    Browse the repository at this point in the history
  43. Use schema_id in prepare_update (#416)

    crisely09 authored and MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    6777f70 View commit details
    Browse the repository at this point in the history
  44. Merge master in dev-datasets

    MFSY committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    3e26c2e View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    e5fcf75 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Fixed tox errors

    MFSY committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    b1f8d09 View commit details
    Browse the repository at this point in the history