PGO-LinkML-commands.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7df8b3b-c05f-411e-898e-6a0823a5050e",
   "metadata": {},
   "source": [
    "# Using [linkML](https://linkml.io) to define schema in the context of PGO:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b669d85-7ce3-43b3-a162-ab24d1b62023",
   "metadata": {},
   "source": [
    "## Background:\n",
    "\n",
    "* [linkML](https://linkml.io)  is a \"flexible modeling language that allows you to author schemas in YAML that describe the structure of your data. Additionally, it is a framework for working with and validating data in a variety of formats (JSON, RDF, TSV), with generators for compiling linkML schemas to other frameworks.\"\n",
    "* PGO, or Pistoia Alliance Pharma General Ontology is a project aiming at documenting 20 key concepts in the context of Pharmaceutical Industry R&D. The process of documenting is meant to first identify these key entities and second, identify open or well characteristerised authoritative resources which can be used as reference, community approved resources, which can be communicated or recommended when defining a data exchange process.\n",
    "* **Why is PGO looking at linkML?** * There are 3 answers for that question:\n",
    "    * Define a data structure for describing each type/class defined by PGO\n",
    "    * Define a data structure for describing one or more established and authoritative resources to reference instances of the PGO types.\n",
    "    * Define a data structure for defining a data transfer between a CRO and a Pharma customer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e53c058-4273-4fc9-adab-b88655c08b89",
   "metadata": {},
   "source": [
    "## Notebook scope:\n",
    "\n",
    "The main goals of this notebook are:\n",
    "* to highlight the key functions provided by the linkml framework to validate schema and instance level data.\n",
    "* to showcase one possible representation mode of data dictionaries using the linkML framework.\n",
    "* to discuss strengths and weaknesses of the framework."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2b1f57-da33-41b2-93c3-4a0025d9e11a",
   "metadata": {},
   "source": [
    "### 1. Viewing a linkML-based PGO Resource Description Minimal Information profile schema:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "68fb948e-a0c5-4ded-bcb3-f7dbd8a00f02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./git/Pistoia-Alliance-Pharma-General-Ontology/doc'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "# os.chdir(\"/change_me_path_to/git/linkml/data_dictionary/outputs/\")\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "64d6cf0d-bc1f-422d-bbb3-85ab34881672",
   "metadata": {},
   "outputs": [],
   "source": [
    "# os.chdir(\"/change_me_path_to/git/Pistoia-Alliance-Pharma-General-Ontology/doc\")\n",
    "# os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "0a7e470c-6e54-4ff4-a1e1-8f77ed4ce0f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id: https://w3id.org/linkml/PGO/core-entities\n",
      "name: core-entities\n",
      "prefixes:\n",
      "  linkml: https://w3id.org/linkml/\n",
      "  namespace: http://example.com\n",
      "\n",
      "version: 0.0.0\n",
      "license: https://creativecommons.org/publicdomain/zero/1.0/\n",
      "\n",
      "imports:\n",
      "  - linkml:types\n",
      "  - ./pgo-enums\n",
      "  - ./pgo-entity-named-thing\n",
      "  - ./pgo-entity-substance\n",
      "  - ./pgo-entity-cell\n",
      "  - ./pgo-entity-drug\n",
      "\n",
      "default_range: string\n"
     ]
    }
   ],
   "source": [
    "!cat  ./linkml/pgo-entities-by-import.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9e2a229-f128-4a8f-bd88-12d7e07d1d9f",
   "metadata": {},
   "source": [
    "### 2. Linting and Validating a linkML schema:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ac0f49-22d0-429b-b46d-4c5956e4aceb",
   "metadata": {},
   "source": [
    "#### 2.1 Let's check if this LinkML schema validates. \n",
    "\n",
    "To do so, we can invoke the `linkml-lint` command. First, let's check the options available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "44247818-f548-444c-a2b2-868cb625b762",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: linkml-lint [OPTIONS] SCHEMA\n",
      "\n",
      "  Run linter on SCHEMA.\n",
      "\n",
      "  SCHEMA can be a single LinkML YAML file or a directory. If it is a directory\n",
      "  every YAML file found in the directory (recursively) will be linted.\n",
      "\n",
      "Options:\n",
      "  -a, --all                       Process files that start with '.'.\n",
      "  -c, --config FILE               Custom linter configuration file.\n",
      "  -f, --format [terminal|markdown|json|tsv]\n",
      "                                  Report format.  [default: terminal]\n",
      "  --validate                      Validate the schema against the LinkML\n",
      "                                  Metamodel before linting.\n",
      "  --validate-only                 Validate the schema against the LinkML\n",
      "                                  Metamodel and then exit without checking\n",
      "                                  linter rules.\n",
      "  -v, --verbose\n",
      "  -o, --output FILENAME           Report file name.\n",
      "  --ignore-warnings               Do not exit with an error status if only\n",
      "                                  warnings are found.\n",
      "  --max-warnings INTEGER          Do not exit with an error status if up to\n",
      "                                  this number of warnings (and no errors) are\n",
      "                                  found.  [default: 0]\n",
      "  --fix / --no-fix\n",
      "  -V, --version                   Show the version and exit.\n",
      "  --help                          Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61679ffa-2765-4576-9860-b44cc28de8ec",
   "metadata": {},
   "source": [
    "#### 2.2 We'll now invoke the command over the `PGO Resource Description Minimal Information profile` linkml schema without any option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "7a9d3d5f-4e69-439f-9584-4381d4f44934",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m✓\u001b[0m No problems found\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --validate ./linkml/pgo-entity-named-thing.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "34eb65d3-d843-4b4b-ab39-237b4f9269c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m✓\u001b[0m No problems found\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --validate ./linkml/pgo-entity-drug.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "32d3afc9-5ae2-444f-9f0c-bcb95df7b061",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[4m/git/Pistoia-Alliance-Pharma-General-Ontology/doc/linkml/pgo-entity-cell.yaml\u001b[0m\n",
      "\u001b[31m  error    \u001b[0mIn classes > Cell: Additional properties are not allowed ('definitions' was unexpected)  \u001b[2m(valid-schema)\u001b[0m\n",
      "\n",
      "\u001b[31m✖\u001b[0m Found 1 problem in 1 schema\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --validate ./linkml/pgo-entity-cell.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "ff94de98-3044-4b9e-8379-146e8e8a7f7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m✓\u001b[0m No problems found\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --validate ./linkml/pgo-entity-substance.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f414a4-9267-4916-8bf3-cf1d2b8374ee",
   "metadata": {},
   "source": [
    "#### 2.3 Let's invoke to same command but let's skip the warning from the report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "d8096b8a-d31d-4315-9076-179300dc918a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[4m//git/Pistoia-Alliance-Pharma-General-Ontology/doc/linkml/pgo-resource.yaml\u001b[0m\n",
      "\u001b[31m  error    \u001b[0mFile is not a valid LinkML schema. Use --validate for more details.\n",
      "\n",
      "\u001b[31m✖\u001b[0m Found 1 problem in 1 schema\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --ignore-warnings ./linkml/pgo-resource.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b4c063-3954-4605-ac6c-18128cd6de76",
   "metadata": {},
   "source": [
    "```note\n",
    "The root cause of the error is the attempt to provide an array of values for the class_iri attribute, which is not supported by the linkml parser.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df1fb8a1-7e2b-412b-9b37-f3f657d03f25",
   "metadata": {},
   "source": [
    "#### 2.4 Invoking the linter with the `--validate` option:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "d5beae49-8e32-4eee-8fc1-46d97bb7e08f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !linkml-lint --validate ./linkml/pgo-entities-by-import.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf3e118a-1f41-4d09-95ad-f1726c14de6f",
   "metadata": {},
   "source": [
    "#### 2.5 Understanding the validation report:\n",
    "\n",
    "The difference between `linting` and `validating` is that the latter will check the validity of syntax while the former checks stylistic compliance. \n",
    "\n",
    "#### Warnings:\n",
    "All the warnings reporting by the linter are mundane and relate to variable names which do not follow the convention adopted by linkML group. These can be ignored but it should be noted that our use departs from the recommendations and may result in a reduction is reusability.\n",
    "\n",
    "#### Errors:\n",
    "The linkml linter/validator reports an `error` for the slot `abbreviation`. The error simply says that the `slot_uri` can not be left empty and, if present, the attribute must be filled with a valid `uniform resource identifier`. A compact uri (curie) may be used if a namespace has been declared in the schema.\n",
    "\n",
    "### Fixes\n",
    "To clear the error in the linkML schema, one would have to set the value for the slot_uri for the `abbreviation`. For instance, something like the following curie 'dcterms:description' .\n",
    "\n",
    "```\n",
    " abbreviation:\n",
    "    slot_uri: schema:alternateName\n",
    "    required: true\n",
    "    description: the abbreviation of the resource\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "a2358c70-4f1e-4513-abf8-ac16ab5ad10f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[4m//git/Pistoia-Alliance-Pharma-General-Ontology/doc/linkml/pgo-entity.yaml\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'ClinicalStudy' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Disease' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Indication' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Compound' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Drug' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Assay' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Person' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Specimen' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'CellType' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'CellLine' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Biomarker' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Target' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Protein' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Device' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Product' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Substance' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Site' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Program' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Project' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mClass 'Vocabulary' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'study_local_id' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'study_regulatory_id' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'study_type' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'manufacturer' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'license' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'funded_by' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'domain' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'biomarker_type' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'ptm' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'scope' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'origin' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'cell_descriptor' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'morphology' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'affiliation' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'technology' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'coordinates' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'inchi_code' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'regulatory_code' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'diseases' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'disease_agent' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\u001b[33m  warning  \u001b[0mSlot 'units' does not have recommended slot 'description'  \u001b[2m(recommended)\u001b[0m\n",
      "\n",
      "\u001b[31m✖\u001b[0m Found 41 problems in 1 schema\n"
     ]
    }
   ],
   "source": [
    "!linkml-lint --validate ./linkml/pgo-entity.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "186f3e35-27b7-4738-b673-bcca69cf89cd",
   "metadata": {},
   "source": [
    "### 3. Generating an Excel Spreadsheet template from the LinkML schema for use by the Business:\n",
    "\n",
    "The linkml framework provides a number of generators which take a linkML schema as an input and generator specific products.\n",
    "One such product is tabular template suitable for data entry by non-technical personel.\n",
    "The command is detailed below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "160b9f19-5397-4d0f-8b5a-d766ed54c1a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ValueError: ['schema:name', 'dcterms:title'] is not a valid URI or CURIE\n"
     ]
    }
   ],
   "source": [
    "!gen-excel ./linkml/pgo-resource.yaml -o  ./linkml/pgo-resource_in_linkml_as_excel_template.xlsx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de2a0169-795a-4acd-bc11-edae081e81b6",
   "metadata": {},
   "source": [
    "*observations*\n",
    "The resulting Microsoft Excel template creates a worksheet for each of the linkML Class declared in the schema. \n",
    "The header of each worksheet correspond to the Class attributes, `slot` in linkML speak.\n",
    "\n",
    "`+` for slots where the range is an enumeration (a set of controlled values), the corresponding excel field will be provisioned with a data validation drop-down showing the value set defined in the linkML schema\n",
    "\n",
    "`-` in spite of textual description for each field declared in the linkML schema, the resulting excel rendering is void of contextual information or tooltips.\n",
    "`-` in spite of specify some field ranges to be of a certain LinkML class, the resulting excel rendering does not show the cross-referencing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "141acf0c-3547-46c6-9886-d25a9cd59a8c",
   "metadata": {},
   "source": [
    "### 4. Let's open the excel file resulting from the operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2eca3ddb-5839-4d31-b925-104034f5ee9a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file ./git/Pistoia-Alliance-Pharma-General-Ontology/doc/linkml/pgo-resource_in_linkml_as_excel_template.xlsx does not exist.\n"
     ]
    }
   ],
   "source": [
    "!open ./linkml/pgo-resource_in_linkml_as_excel_template.xlsx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c59c95d6-ac76-4d16-b788-150808000cf1",
   "metadata": {},
   "source": [
    "### 5. Validate instance data against a linkML schema:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee272c77-375e-45d8-ace7-e76972146e8c",
   "metadata": {},
   "source": [
    "In this section of the tutorial, we will show how to validate instance data against a linkML schema.\n",
    "While linkML is meant to be capable of validating instances expressed in various forms (e.g., yaml, tsv), we will highlight some inconsistencies and maturity issues associated with the technology stack.\n",
    "In this section, we will be using a simpler example relying on a much simpler schema, the `Personinfo.yaml` linkML schema and the associated instances file, `person_instance_data.yaml`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "470bee45-e521-4385-b91b-06462ff7a4ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat: ./linkml/pgo_resource_instance_data.yaml: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!cat ./linkml/pgo_resource_instance_data.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b681ee1f-fc7a-4282-9803-c223b4cec111",
   "metadata": {},
   "source": [
    "### Let's validate this instance data yaml file against the schema:\n",
    "\n",
    "Note: This is different from checking the linkML schema is valid against its schema definition. Here, we want to check that all instance data is compliant with the requirements described by the `Personinfo.yaml` document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6e00d13a-1839-49bb-b78a-d0e51678b012",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/git/venv/bin/linkml-validate\", line 8, in <module>\n",
      "    sys.exit(cli())\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 1157, in __call__\n",
      "    return self.main(*args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 1078, in main\n",
      "    rv = self.invoke(ctx)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 1434, in invoke\n",
      "    return ctx.invoke(self.callback, **ctx.params)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 783, in invoke\n",
      "    return __callback(*args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/decorators.py\", line 33, in new_func\n",
      "    return f(get_current_context(), *args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml/validator/cli.py\", line 191, in cli\n",
      "    validator = Validator(config.schema_path, validation_plugins=plugins, strict=exit_on_first_failure)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml/validator/validator.py\", line 39, in __init__\n",
      "    self._schema: SchemaDefinition = yaml_loader.load(schema, SchemaDefinition)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py\", line 76, in load\n",
      "    results = self.load_any(*args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py\", line 41, in load_any\n",
      "    data_as_dict = self.load_as_dict(source, base_dir=base_dir, metadata=metadata)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py\", line 32, in load_as_dict\n",
      "    return yaml.load(data, DupCheckYamlLoader)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/yaml/__init__.py\", line 81, in load\n",
      "    return loader.get_single_data()\n",
      "  File \"/git/venv/lib/python3.9/site-packages/yaml/constructor.py\", line 49, in get_single_data\n",
      "    node = self.get_single_node()\n",
      "  File \"yaml/_yaml.pyx\", line 673, in yaml._yaml.CParser.get_single_node\n",
      "  File \"yaml/_yaml.pyx\", line 687, in yaml._yaml.CParser._compose_document\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 729, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 808, in yaml._yaml.CParser._compose_sequence_node\n",
      "  File \"yaml/_yaml.pyx\", line 860, in yaml._yaml.CParser._parse_next_event\n",
      "yaml.scanner.ScannerError: while scanning a plain scalar\n",
      "  in \"pgo-resource.yaml\", line 66, column 31\n",
      "found unexpected ':'\n",
      "  in \"pgo-resource.yaml\", line 66, column 38\n"
     ]
    }
   ],
   "source": [
    "! linkml-validate --schema ./linkml/pgo-resource.yaml --target-class Resource ./linkml/pgo-resource-instances.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62043ac-372d-4de4-be7d-f863a4925c59",
   "metadata": {},
   "source": [
    "### Let's now the same using instance data formatted in tsv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b18f8fd8-62bd-4535-b530-b820fd367674",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: linkml-validate [OPTIONS] [DATA_SOURCES]...\n",
      "Try 'linkml-validate --help' for help.\n",
      "\n",
      "Error: Invalid value for '[DATA_SOURCES]...': Path './linkml/pgo_resource_instance_data_as_excel_template-populated.csv' does not exist.\n"
     ]
    }
   ],
   "source": [
    "!linkml-validate --schema ./linkml/pgo-resource.yaml ./linkml/pgo_resource_instance_data_as_excel_template-populated.csv --target-class Resource    \n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "98970d59-5cf4-4de1-a2b7-ee3a8f29f140",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yaml.scanner.ScannerError: while scanning a plain scalar\n",
      "  in \"pgo-resource.yaml\", line 66, column 31\n",
      "found unexpected ':'\n",
      "  in \"pgo-resource.yaml\", line 66, column 38\n"
     ]
    }
   ],
   "source": [
    "!gen-jsonld-context --base https://schema.org ./linkml/pgo-resource.yaml > ./linkml/pgo_resource_new_context.jsonld"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fab3b784-66ff-41c3-8bd3-e2a5a77105d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: gen-jsonld-context [OPTIONS] YAMLFILE\n",
      "\n",
      "  Generate jsonld @context definition from LinkML model\n",
      "\n",
      "Options:\n",
      "  --base TEXT                     Base URI for model\n",
      "  --prefixes / --no-prefixes      Emit context for prefixes\n",
      "                                  (default=--prefixes)  [default: prefixes]\n",
      "  --model / --no-model            Emit context for model elements\n",
      "                                  (default=--model)  [default: model]\n",
      "  --flatprefixes / --no-flatprefixes\n",
      "                                  Emit non-JSON-LD compliant prefixes as an\n",
      "                                  object (deprecated: use gen-prefix-map\n",
      "                                  instead).  [default: no-flatprefixes]\n",
      "  -V, --version                   Show the version and exit.\n",
      "  -f, --format [context|json]     Output format  [default: context]\n",
      "  --metadata / --no-metadata      Include metadata in output  [default:\n",
      "                                  metadata]\n",
      "  --useuris / --metauris          Use class and slot URIs over model uris\n",
      "                                  [default: useuris]\n",
      "  -im, --importmap FILENAME       Import mapping file\n",
      "  --log_level [CRITICAL|ERROR|WARNING|INFO|DEBUG]\n",
      "                                  Logging level  [default: WARNING]\n",
      "  -v, --verbose                   Verbosity. Takes precedence over\n",
      "                                  --log_level.\n",
      "  --mergeimports / --no-mergeimports\n",
      "                                  Merge imports into source file\n",
      "                                  (default=mergeimports)\n",
      "  --stacktrace / --no-stacktrace  Print a stack trace when an error occurs\n",
      "                                  [default: no-stacktrace]\n",
      "  --help                          Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "!gen-jsonld-context --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "66562c3a-64e1-4c3a-97ed-430ace8e9281",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: linkml-convert [OPTIONS] INPUT\n",
      "\n",
      "  Converts instance data to and from different LinkML Runtime serialization\n",
      "  formats.\n",
      "\n",
      "  The instance data must conform to a LinkML model, and either a path to a\n",
      "  python module must be passed, or a path to a schema.\n",
      "\n",
      "  The converter works by first using a linkml-runtime *loader* to instantiate\n",
      "  in-memory model objects, then a *dumper* is used to serialize. A validation\n",
      "  step is optionally performed in between\n",
      "\n",
      "  When converting to or from RDF, a path to a schema must be provided.\n",
      "\n",
      "  For more information, see https://linkml.io/linkml/data/index.html\n",
      "\n",
      "Options:\n",
      "  -m, --module TEXT               Path to python datamodel module\n",
      "  -o, --output TEXT               Path to output file\n",
      "  -f, --input-format [yml|yaml|json|rdf|ttl|json-ld|csv|tsv]\n",
      "                                  Input format. Inferred from input suffix if\n",
      "                                  not specified\n",
      "  -t, --output-format [yml|yaml|json|rdf|ttl|json-ld|csv|tsv]\n",
      "                                  Output format. Inferred from output suffix\n",
      "                                  if not specified\n",
      "  -C, --target-class TEXT         name of class in datamodel that the root\n",
      "                                  node instantiates\n",
      "  --target-class-from-path / --no-target-class-from-path\n",
      "                                  Infer the target class from the filename,\n",
      "                                  should be ClassName-<other-\n",
      "                                  chars>.{yaml,json,...}  [default: no-target-\n",
      "                                  class-from-path]\n",
      "  -S, --index-slot TEXT           top level slot. Required for CSV\n",
      "                                  dumping/loading\n",
      "  -s, --schema TEXT               Path to schema specified as LinkML yaml\n",
      "  -P, --prefix TEXT               Prefixmap base=URI pairs\n",
      "  --validate / --no-validate      Validate against the schema  [default:\n",
      "                                  validate]\n",
      "  --infer / --no-infer            Infer missing slot values  [default: no-\n",
      "                                  infer]\n",
      "  -c, --context TEXT              path to JSON-LD context file\n",
      "  -V, --version                   Show the version and exit.\n",
      "  --help                          Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "!linkml-convert --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "be8dfef3-3554-4ce2-910f-666a30de353d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/git/venv/bin/linkml-convert\", line 8, in <module>\n",
      "    sys.exit(cli())\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 1157, in __call__\n",
      "    return self.main(*args, **kwargs)\n",
      "  File \"git/venv/lib/python3.9/site-packages/click/core.py\", line 1078, in main\n",
      "    rv = self.invoke(ctx)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 1434, in invoke\n",
      "    return ctx.invoke(self.callback, **ctx.params)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/click/core.py\", line 783, in invoke\n",
      "    return __callback(*args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml/utils/converter.py\", line 108, in cli\n",
      "    python_module = PythonGenerator(schema).compile_module()\n",
      "  File \"<string>\", line 29, in __init__\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml/generators/pythongen.py\", line 75, in __post_init__\n",
      "    self.schemaview = SchemaView(self.schema, base_dir=self.base_dir)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/utils/schemaview.py\", line 158, in __init__\n",
      "    schema = load_schema_wrap(schema)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/utils/schemaview.py\", line 85, in load_schema_wrap\n",
      "    schema = yaml_loader.load(path, target_class=SchemaDefinition, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py\", line 76, in load\n",
      "    results = self.load_any(*args, **kwargs)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py\", line 41, in load_any\n",
      "    data_as_dict = self.load_as_dict(source, base_dir=base_dir, metadata=metadata)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py\", line 32, in load_as_dict\n",
      "    return yaml.load(data, DupCheckYamlLoader)\n",
      "  File \"/git/venv/lib/python3.9/site-packages/yaml/__init__.py\", line 81, in load\n",
      "    return loader.get_single_data()\n",
      "  File \"/git/venv/lib/python3.9/site-packages/yaml/constructor.py\", line 49, in get_single_data\n",
      "    node = self.get_single_node()\n",
      "  File \"yaml/_yaml.pyx\", line 673, in yaml._yaml.CParser.get_single_node\n",
      "  File \"yaml/_yaml.pyx\", line 687, in yaml._yaml.CParser._compose_document\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 731, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 845, in yaml._yaml.CParser._compose_mapping_node\n",
      "  File \"yaml/_yaml.pyx\", line 729, in yaml._yaml.CParser._compose_node\n",
      "  File \"yaml/_yaml.pyx\", line 808, in yaml._yaml.CParser._compose_sequence_node\n",
      "  File \"yaml/_yaml.pyx\", line 860, in yaml._yaml.CParser._parse_next_event\n",
      "yaml.scanner.ScannerError: while scanning a plain scalar\n",
      "  in \"pgo-resource.yaml\", line 66, column 31\n",
      "found unexpected ':'\n",
      "  in \"pgo-resource.yaml\", line 66, column 38\n"
     ]
    }
   ],
   "source": [
    "!linkml-convert -s ./linkml/pgo-resource.yaml   -c ./linkml/pgo_resource_new_context.jsonld -f yaml  ./linkml/pgo-resource-instances.yaml  -t ttl -o ./linkml/pgo_resources_instance_data_in_turtle.ttl\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea3f715-e88f-4523-8bb6-bbacfe0593e1",
   "metadata": {},
   "source": [
    "### 6. Generating documentation fror the schema:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a15c3372-c97f-4e3c-b6d2-67fb15fa3fb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yaml.scanner.ScannerError: while scanning a plain scalar\n",
      "  in \"pgo-resource.yaml\", line 66, column 31\n",
      "found unexpected ':'\n",
      "  in \"pgo-resource.yaml\", line 66, column 38\n"
     ]
    }
   ],
   "source": [
    "!gen-doc  ./linkml/pgo-resource.yaml  -d ./linkml/pgo_resource_schema_documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ff5024a-f966-4419-91f7-da0393df8323",
   "metadata": {},
   "source": [
    "### 7. Generating SQL tables from the linkML schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "41d762f6-cc0e-48e4-baf0-bfb43e5d35b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yaml.scanner.ScannerError: while scanning a plain scalar\n",
      "  in \"pgo-resource.yaml\", line 66, column 31\n",
      "found unexpected ':'\n",
      "  in \"pgo-resource.yaml\", line 66, column 38\n"
     ]
    }
   ],
   "source": [
    "!gen-sqlddl ./linkml/pgo-resource.yaml > ./linkml/resources_linkml.ddl.sql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8794fdf2-410b-452a-8a18-4b7e519d3bc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!grep -A35 'CREATE TABLE \"Resource\"' ./linkml/resources_linkml.ddl.sql"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c28ca67b-a451-45f1-949d-7ccdc0533496",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "\n",
    "With this last step, we reach the end of this tutorial. \n",
    "In the next guidance document, we will demonstrate how to generate a basic prototype for a user interface using streamlit and the streamlit-sqlalchemy packages."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}