Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Databases object #339

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a3e35d8
Added DatabaseSources to forge.
crisely09 Oct 3, 2022
afd5ba5
Improved DatabaseSource class. Made possible to load mappings from lo…
crisely09 Oct 4, 2022
f03f941
Added option to retrieve mappings from bd_sources also from the forge.
crisely09 Oct 4, 2022
9ace785
Added option to display types of mappings, a model and a store to eac…
crisely09 Oct 5, 2022
1da8883
Updated notebook
crisely09 Oct 5, 2022
9b3c88a
Did a little clean-up of the notebook
crisely09 Oct 5, 2022
a3e4d74
Change SPARQLStore to Specilized UniProtStore. Added Service and spaq…
crisely09 Oct 10, 2022
3e002cd
Added SPARQLWrapper to dependency list.
crisely09 Oct 10, 2022
47f8b0c
Recreated the SPARQLStore, and added the specialized UniProtStore. St…
crisely09 Oct 11, 2022
5200b3b
Modified the search method of the sparql store. Resources are retriev…
crisely09 Oct 12, 2022
a01af1e
Updated notebook with search example.
crisely09 Oct 12, 2022
8d7bea7
Added a draft of UniProt mappings.
crisely09 Oct 12, 2022
686aea4
Make Databases be new archetype. Updated most methods, missing search…
crisely09 Oct 17, 2022
573ae50
Adding the missing file.
crisely09 Oct 18, 2022
2520b7f
Sorted mapping functions. Missing tests.
crisely09 Oct 19, 2022
79b44bc
Improved Protein mappings.
crisely09 Oct 19, 2022
2cfabf3
Implemented mappings methods in RdfModel and update Database classes …
crisely09 Oct 24, 2022
a7da898
Rebased and added unit tests for SPARQLStore and StoreDatabase.
crisely09 Oct 27, 2022
b837bc8
Added WebServiceDatabase. Missing tests and improved download methods.
crisely09 Nov 2, 2022
9433b1f
Adding missing file.
crisely09 Nov 2, 2022
761e114
Some changes.
crisely09 Nov 7, 2022
96c227c
Added DemoDB folder.
crisely09 Nov 7, 2022
ad6b5fd
Make types in db_sources optional
crisely09 Nov 7, 2022
539c75c
Some changes.
crisely09 Nov 8, 2022
264ba9a
Added initial unit tests for WebService Database.
crisely09 Nov 10, 2022
dd7d22d
Update tests for webservice databases.
crisely09 Nov 15, 2022
5fd39bc
Resolve conflicts with old methods
crisely09 Sep 18, 2023
0f64c3d
Merge branch 'master' into dev-resources
MFSY Oct 15, 2023
d94e5a7
Merge branch 'master' into dev-resources
crisely09 Oct 19, 2023
b26fefc
Add mapper and mapping methods to SPARQL store
crisely09 Oct 19, 2023
811ed11
Merge branch 'master' into dev-resources
crisely09 Oct 26, 2023
b443b1c
Small mixes
crisely09 Oct 26, 2023
3c6f952
Add self to tests in classes
crisely09 Oct 26, 2023
29b481e
Merge branch 'master' into dev-resources
crisely09 Oct 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions examples/configurations/database-sources/prod-nexus-sources.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
Model:
name: RdfModel
origin: store
source: BlueBrainNexus
context:
iri: "https://bbp.neuroshapes.org"
bucket: "neurosciencegraph/datamodels"
Store:
name: BlueBrainNexus
endpoint: https://bbp.epfl.ch/nexus/v1
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
elastic:
endpoint: "https://bbp.epfl.ch/neurosciencegraph/data/views/aggreg-es/dataset"
mapping: "https://bbp.epfl.ch/neurosciencegraph/data/views/es/dataset"
default_str_keyword_field: "keyword"
vocabulary:
metadata:
iri: "https://bluebrain.github.io/nexus/contexts/metadata.json"
local_iri: "https://bluebrainnexus.io/contexts/metadata.json"
namespace: "https://bluebrain.github.io/nexus/vocabulary/"
deprecated_property: "https://bluebrain.github.io/nexus/vocabulary/deprecated"
project_property: "https://bluebrain.github.io/nexus/vocabulary/project"
max_connection: 50
versioned_id_template: "{x.id}?rev={x._store_metadata._rev}"
file_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-store/file-to-resource-mapping.hjson

Resolvers:
ontology:
- resolver: OntologyResolver
origin: store
source: BlueBrainNexus
targets:
- identifier: terms
bucket: neurosciencegraph/datamodels
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
result_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-resolver/term-to-resource-mapping.hjson
agent:
- resolver: AgentResolver
origin: store
source: BlueBrainNexus
targets:
- identifier: agents
bucket: bbp/agents
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
result_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-resolver/agent-to-resource-mapping.hjson

Formatters:
identifier: https://bbp.epfl.ch/neurosciencegraph/data/{}/{}
identifier_neuroelectro: http://neuroelectro.org/api/1/{}/{}
identifier_neurolex: http://neurolex.org/wiki/{}

Databases:
UniProt:
origin: store
source: SPARQLStore
searchendpoints:
sparql:
endpoint: "https://sparql.uniprot.org/sparql"
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/UniProt
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/UniProt/jsonld_context.json
NeuroElectro:
origin: store
source: BlueBrainNexus
bucket: bbp/neuroelectro
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/NeuroElectro
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/models/neuroshapes_context.json
NeuroMorpho:
origin: web_service
source: http://neuromorpho.org/api/neuron
health: http://neuromorpho.org/api/health
files_download:
endpoint: https://neuromorpho.org/dableFiles
Accept: "*/*"
max_connection: 50
content_type: application/json;charset=UTF-8
accept: "*/*"
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/Neuromorpho
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/Neuromorpho/jsonld_context.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
id: forge.format('identifier', "examples", x.id.split('/')[-1])
type: [
Entity
]
name: x.name
label: x.label
subject: forge.resolve(x.subject, scope='ontology') if forge.resolve(x.subject, scope='ontology') is not None else {"label": x.subject}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
{
type: [
Dataset
NeuronMorphology
]
id: forge.format("identifier", "neuronmorphologies/mouselight", x.neurons[0]["idString"])
brainLocation:
{
type: BrainLocation
brainRegion:
{
id: f"http://api.brain-map.org/api/v2/data/Structure/{x.neurons[0]['soma']['allenId']}"
label: x.neurons[0]["allenLabel"]
}
coordinatesInBrainAtlas:
{
valueX: x.neurons[0]["soma"]["x"]
valueY: x.neurons[0]["soma"]["y"]
valueZ: x.neurons[0]["soma"]["z"]
}
}
contribution:
{
type: Contribution
agent:
{
type: Organization
id: https://www.grid.ac/institutes/grid.443970.d
label: "Janelia Research Campus"
}
}
distribution: forge.attach(f"./mouselight/{x.neurons[0]['idString']}.swc", content_type="application/swc")
identifier: x.neurons[0]["idString"]
name: x.neurons[0]["idString"]
generation:
{
type: Generation
activity:
{
type: "nsg:NeuronMorphologyReconstruction"
hadProtocol:{

}
}
}
subject:
{
type: Subject
species: {
id: http://purl.obolibrary.org/obo/NCBITaxon_10090
label: Mus musculus
}
strain: {
label: x.neurons[0]["sample"]["strain"]
}
}
license:
{
type: License
id: https://mouselight.janelia.org
}
objectOfStudy:
{
type: ObjectOfStudy
id: http://bbp.epfl.ch/neurosciencegraph/taxonomies/objectsofstudy/singlecells
label: Single Cell
}
dateCreated: x.neurons[0]["sample"]["date"]
version: x.neurons[0]["annotationSpace"]["version"]
description: x.neurons[0]["annotationSpace"]["description"]
virus: x.neurons[0]["label"]["virus"]
fluorophore: x.neurons[0]["label"]["fluorophore"]
}
8 changes: 8 additions & 0 deletions examples/database-sources/MouseLight/metadata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"description": "The MouseLight project generates datasets of whole mouse brains imaged at submicron resolution that allows reconstructions of complete axonal arbors of individual neurons across the entire mouse brain. ",
"url": "https://www.janelia.org/project-team/mouselight/neuronbrowser",
"license" : {
"id": "https://creativecommons.org/licenses/by-nc/4.0",
"label": "CC BY-NC 4.0"
}
}
158 changes: 158 additions & 0 deletions examples/database-sources/NeuroElectro/jsonld_context.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
{
"@context": {
"@vocab": "http://example.org/vocab/",
"Activity": {
"@id": "prov:Activity"
},
"Agent": {
"@id": "prov:Agent"
},
"Class": {
"@id": "owl:Class"
},
"Entity": {
"@id": "prov:Entity"
},
"Association": {
"@id": "schema:Association"
},
"BrainRegion": {
"@id": "nsg:BrainRegion"
},
"Building": {
"@id": "http://schema.org/Building"
},
"Employee": {
"@id": "http://www.example.com/Employee"
},
"Organization": {
"@id": "schema:Organization"
},
"Person": {
"@id": "schema:Person"
},
"PostalAddress": {
"@id": "schema:PostalAddress"
},
"address": {
"@id": "schema:address"
},
"agent": {
"@id": "schema:agent"
},
"birthDate": {
"@id": "schema:birthDate",
"@type": "xsd:date"
},
"citation": {
"@id": "schema:citation"
},
"contractor": {
"@id": "schema:contractor"
},
"deathDate": {
"@id": "schema:deathDate",
"@type": "xsd:date"
},
"department": {
"@id": "schema:department"
},
"description": {
"@id": "schema:description"
},
"endedAtTime": {
"@id": "prov:endedAtTime"
},
"familyName": {
"@id": "schema:familyName"
},
"founder": {
"@id": "schema:founder"
},
"gender": {
"@id": "schema:gender"
},
"generated": {
"@id": "prov:generated"
},
"geo": {
"@id": "schema:geo"
},
"givenName": {
"@id": "schema:givenName"
},
"image": {
"@id": "http://schema.org/image",
"@type": "@id"
},
"isDefinedBy": {
"@id": "rdfs:isDefinedBy",
"@type": "@id"
},
"label": "rdfs:label",
"latitude": {
"@id": "schema:latitude",
"@type": "xsd:float"
},
"longitude": {
"@id": "schema:longitude"
},
"name": {
"@id": "schema:name"
},
"notation": "skos:notation",
"nsg": "https://neuroshapes.org/",
"owl": "http://www.w3.org/2002/07/owl#",
"postalCode": {
"@id": "schema:postalCode"
},
"prefLabel": "skos:prefLabel",
"prov": "http://www.w3.org/ns/prov#",
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"schema": "http://schema.org/",
"skos": "http://www.w3.org/2004/02/skos/core#",
"startDate": {
"@id": "schema:startDate",
"@type": "xsd:date"
},
"startedAtTime": {
"@id": "prov:startedAtTime",
"@type": "xsd:dateTime"
},
"status": {
"@id": "schema:status"
},
"streetAddress": {
"@id": "schema:streetAddress"
},
"subClassOf": {
"@id": "rdfs:subClassOf",
"@type": "@id"
},
"supervisor": {
"@id": "schema:supervisor",
"@type": "@id"
},
"type": {
"@id": "rdf:type",
"@type": "@id"
},
"used": {
"@id": "prov:used"
},
"validated": {
"@id": "schema:validated"
},
"value": {
"@id": "schema:value"
},
"wasStartedBy": {
"@id": "prov:wasStartedBy",
"@type": "@id"
},
"xsd": "http://www.w3.org/2001/XMLSchema#",
"foaf": "http://xmlns.com/foaf/0.1/"
},
"@id": "http://context.example.org"
}
Loading
Loading