diff --git a/notebooks/external_model.ipynb b/notebooks/external_model.ipynb index 64c39faeb..2b51dce25 100644 --- a/notebooks/external_model.ipynb +++ b/notebooks/external_model.ipynb @@ -40,11 +40,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3xwTmHM)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -170,7 +166,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer_checkpoint = \"gpt2\"\n", "tokenizer = AutoTokenizer.from_pretrained(tokenizer_checkpoint)" ] @@ -503,7 +499,7 @@ " \"optimizer_state_offchip\": True,\n", " \"replicated_tensor_sharding\": True,\n", " \"enable_half_partials\": True,\n", - " \"device_iterations\": 1, \n", + " \"device_iterations\": 1,\n", " \"inference_device_iterations\": 5,\n", " \"gradient_accumulation_steps\": 512,\n", " \"executable_cache_dir\": executable_cache_dir,\n", diff --git a/notebooks/flan_t5_inference.ipynb b/notebooks/flan_t5_inference.ipynb index ad745e6dd..b462c6769 100644 --- a/notebooks/flan_t5_inference.ipynb +++ b/notebooks/flan_t5_inference.ipynb @@ -32,11 +32,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/AWzcPN)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/image_classification.ipynb b/notebooks/image_classification.ipynb index 55d24ede5..4a8cf7ad8 100644 --- a/notebooks/image_classification.ipynb +++ b/notebooks/image_classification.ipynb @@ -49,11 +49,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3QxTCyU)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -339,7 +335,7 @@ }, "outputs": [], "source": [ - "from datasets import load_dataset \n", + "from datasets import load_dataset\n", "from pathlib import Path\n", "\n", "# load a custom dataset from local/remote files or folders using the ImageFolder feature\n", @@ -866,7 +862,7 @@ "from optimum.graphcore import IPUTrainingArguments, IPUTrainer, IPUConfig\n", "\n", "model = AutoModelForImageClassification.from_pretrained(\n", - " model_checkpoint, \n", + " model_checkpoint,\n", " label2id=label2id,\n", " id2label=id2label,\n", " ignore_mismatched_sizes = True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint\n", diff --git a/notebooks/introduction_to_optimum_graphcore.ipynb b/notebooks/introduction_to_optimum_graphcore.ipynb index 39d53b79c..61448998b 100644 --- a/notebooks/introduction_to_optimum_graphcore.ipynb +++ b/notebooks/introduction_to_optimum_graphcore.ipynb @@ -95,11 +95,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3CExwVy)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -609,7 +605,7 @@ "source": [ "trainer = IPUTrainer(model=model,\n", " ipu_config=ipu_config,\n", - " args=training_args, \n", + " args=training_args,\n", " train_dataset=train_dataset,\n", " eval_dataset=validation_features,\n", " )" @@ -820,7 +816,7 @@ "input_tensor = torch.tensor(input_encoding[\"input_ids\"]).unsqueeze(0)\n", "attention_tensor= torch.tensor(input_encoding[\"attention_mask\"]).unsqueeze(0)\n", "token_types=torch.tensor(input_encoding[\"token_type_ids\"]).unsqueeze(0)\n", - " \n", + "\n", "# Get model and load the fine-tuned weights\n", "model = transformers.BertForQuestionAnswering.from_pretrained(\"/tmp/outputs\")" ] @@ -900,8 +896,8 @@ "outputs": [], "source": [ "# Make sure you have git-lfs and huggingface-hub\n", - "!apt-get update && apt-get upgrade -y && apt-get install -y git git-lfs \n", - "# !pip install -y huggingface-hub " + "!apt-get update && apt-get upgrade -y && apt-get install -y git git-lfs\n", + "# !pip install -y huggingface-hub" ] }, { diff --git a/notebooks/language_modelling_from_scratch.ipynb b/notebooks/language_modelling_from_scratch.ipynb index d9c2be357..5564cc1a5 100644 --- a/notebooks/language_modelling_from_scratch.ipynb +++ b/notebooks/language_modelling_from_scratch.ipynb @@ -57,11 +57,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/414XiNp)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -287,7 +283,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, ClassLabel):\n", @@ -391,7 +387,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(tokenizer_checkpoint)" ] }, diff --git a/notebooks/mt5_translation.ipynb b/notebooks/mt5_translation.ipynb index f7f94cc75..6e4236069 100644 --- a/notebooks/mt5_translation.ipynb +++ b/notebooks/mt5_translation.ipynb @@ -47,11 +47,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)]()\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -243,7 +239,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, datasets.ClassLabel):\n", @@ -299,7 +295,7 @@ "source": [ "from transformers import AutoTokenizer\n", "from dataclasses import dataclass\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\n", "\n", "@dataclass\n", @@ -361,7 +357,7 @@ " # Setup the tokenizer for targets\n", " with tokenizer.as_target_tokenizer():\n", " labels = tokenizer(targets, max_length=tokenizer_config.max_length, padding=tokenizer_config.padding, truncation=tokenizer_config.truncation)\n", - " \n", + "\n", " # Since we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n", " # padding in the loss.\n", " labels[\"input_ids\"] = [\n", @@ -510,7 +506,7 @@ "\n", "# Gradient accumulation steps = 128 for 1 replica.\n", "# Increase replicas if we have additional ipus available and\n", - "# adjust gradient accumulation steps. The trainer will automatically \n", + "# adjust gradient accumulation steps. The trainer will automatically\n", "# use data parallel training by using the `n_ipu` argument\n", "replication_factor = n_ipu // ipu_config.ipus_per_replica\n", "gradient_accumulation_steps = 128 // replication_factor\n", diff --git a/notebooks/mt5_xnli.ipynb b/notebooks/mt5_xnli.ipynb index b9572b4a0..736fead9a 100644 --- a/notebooks/mt5_xnli.ipynb +++ b/notebooks/mt5_xnli.ipynb @@ -38,11 +38,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)]()\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/multiple_choice.ipynb b/notebooks/multiple_choice.ipynb index 3c14d597c..f5ee16ce2 100644 --- a/notebooks/multiple_choice.ipynb +++ b/notebooks/multiple_choice.ipynb @@ -45,11 +45,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3lI3fzQ)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -377,7 +373,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, ClassLabel):\n", @@ -474,7 +470,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)" ] }, @@ -551,15 +547,15 @@ " # Grab all second sentences possible for each context.\n", " question_headers = examples[\"sent2\"]\n", " second_sentences = [[f\"{header} {examples[end][i]}\" for end in ending_names] for i, header in enumerate(question_headers)]\n", - " \n", + "\n", " # Flatten everything\n", " first_sentences = sum(first_sentences, [])\n", " second_sentences = sum(second_sentences, [])\n", - " \n", + "\n", " # Tokenize\n", " tokenized_examples = tokenizer(\n", - " first_sentences, \n", - " second_sentences, \n", + " first_sentences,\n", + " second_sentences,\n", " truncation=True,\n", " max_length=max_seq_length,\n", " padding=\"max_length\",\n", diff --git a/notebooks/name-entity-extraction.ipynb b/notebooks/name-entity-extraction.ipynb index 496f122c8..11a4b3df3 100644 --- a/notebooks/name-entity-extraction.ipynb +++ b/notebooks/name-entity-extraction.ipynb @@ -36,11 +36,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3XgZ7V2)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -144,9 +140,9 @@ "metadata": {}, "outputs": [], "source": [ - "ner_pipeline = pipelines.pipeline(\"ner\", \n", - " ipu_config=inference_config, \n", - " padding='max_length', \n", + "ner_pipeline = pipelines.pipeline(\"ner\",\n", + " ipu_config=inference_config,\n", + " padding='max_length',\n", " max_length=256)" ] }, @@ -212,7 +208,7 @@ "out = ner_pipeline(prompt)\n", "\n", "def app_for_pipeline(pipeline, examples=[], description=\"\", label_description=\"\"):\n", - " demo = gr.Blocks( \n", + " demo = gr.Blocks(\n", " title=description,\n", " )\n", " with demo:\n", @@ -398,9 +394,9 @@ "source": [ "medical_model = \"d4data/biomedical-ner-all\"\n", "ner_pipeline_medical = pipelines.pipeline(\n", - " \"ner\", model= medical_model, \n", + " \"ner\", model= medical_model,\n", " ipu_config=inference_config,\n", - " padding='max_length', \n", + " padding='max_length',\n", " max_length=256\n", ")\n", "medical_output = ner_pipeline_medical(prompt)" @@ -441,7 +437,7 @@ "outputs": [], "source": [ "app_for_pipeline(\n", - " ner_pipeline_medical, \n", + " ner_pipeline_medical,\n", " examples=medical_examples,\n", " description=\"Try prompting me with some medical anecdotes!\"\n", ").launch(share=share_gradio)" diff --git a/notebooks/question_answering.ipynb b/notebooks/question_answering.ipynb index 22cdb5e81..4f63ac424 100644 --- a/notebooks/question_answering.ipynb +++ b/notebooks/question_answering.ipynb @@ -41,11 +41,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3IutIto)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -389,7 +385,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, ClassLabel):\n", @@ -448,7 +444,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)" ] }, @@ -1497,7 +1493,7 @@ "\n", " min_null_score = None # Only used if squad_v2 is True.\n", " valid_answers = []\n", - " \n", + "\n", " context = example[\"context\"]\n", " # Looping through all the features associated to the current example.\n", " for feature_index in feature_indices:\n", @@ -1542,14 +1538,14 @@ " \"text\": context[start_char: end_char]\n", " }\n", " )\n", - " \n", + "\n", " if len(valid_answers) > 0:\n", " best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n", " else:\n", " # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n", " # failure.\n", " best_answer = {\"text\": \"\", \"score\": 0.0}\n", - " \n", + "\n", " # Let's pick our final answer: the best one or the null answer (only for squad_v2)\n", " if not squad_v2:\n", " predictions[example[\"id\"]] = best_answer[\"text\"]\n", diff --git a/notebooks/sentiment_analysis.ipynb b/notebooks/sentiment_analysis.ipynb index e4f010252..0b57e3e22 100644 --- a/notebooks/sentiment_analysis.ipynb +++ b/notebooks/sentiment_analysis.ipynb @@ -39,11 +39,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3X5wL0a)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a Python virtual environment with the PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a Python virtual environment with the PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -259,7 +255,7 @@ "outputs": [], "source": [ "ambiguous_prompts = [\n", - " \"How are you today?\", \n", + " \"How are you today?\",\n", " \"I'm a little tired, I didn't sleep well, but I hope it gets better\"\n", "]" ] @@ -355,7 +351,7 @@ "outputs": [], "source": [ "split_prompts_answer = tweet18_pipeline([\n", - " \"How are you today?\", \n", + " \"How are you today?\",\n", " *\"I'm a little tired, I didn't sleep well, but I hope it gets better\".split(\",\"),\n", "])\n", "split_prompts_answer" @@ -649,7 +645,7 @@ "outputs": [], "source": [ "emotion_pipeline([\n", - " \"How are you today?\", \n", + " \"How are you today?\",\n", " \"Don't make me go out, it's too cold!\",\n", " \"What is happening, I don't understand\",\n", " \"Where did you come from?\",\n", diff --git a/notebooks/stable_diffusion/image_to_image.ipynb b/notebooks/stable_diffusion/image_to_image.ipynb index aea16aec0..541a1c292 100644 --- a/notebooks/stable_diffusion/image_to_image.ipynb +++ b/notebooks/stable_diffusion/image_to_image.ipynb @@ -33,11 +33,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you. \n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3k8znM2)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -165,8 +161,8 @@ "outputs": [], "source": [ "pipe = IPUStableDiffusionImg2ImgPipeline.from_pretrained(\n", - " \"runwayml/stable-diffusion-v1-5\", \n", - " revision=\"fp16\", \n", + " \"runwayml/stable-diffusion-v1-5\",\n", + " revision=\"fp16\",\n", " torch_dtype=torch.float16,\n", " n_ipu=n_ipu,\n", " num_prompts=1,\n", diff --git a/notebooks/stable_diffusion/inpainting.ipynb b/notebooks/stable_diffusion/inpainting.ipynb index e03ad0951..e7c725c61 100644 --- a/notebooks/stable_diffusion/inpainting.ipynb +++ b/notebooks/stable_diffusion/inpainting.ipynb @@ -32,11 +32,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3W56b5R)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -165,8 +161,8 @@ "outputs": [], "source": [ "pipe = IPUStableDiffusionInpaintPipeline.from_pretrained(\n", - " \"runwayml/stable-diffusion-inpainting\", \n", - " revision=\"fp16\", \n", + " \"runwayml/stable-diffusion-inpainting\",\n", + " revision=\"fp16\",\n", " torch_dtype=torch.float16,\n", " n_ipu=n_ipu,\n", " num_prompts=1,\n", @@ -204,9 +200,9 @@ "source": [ "from PIL import Image\n", "\n", - "pipe(\"apple\", \n", - " image=Image.new(\"RGB\", image_dimensions), \n", - " mask_image=Image.new(\"RGB\", image_dimensions, (255, 255, 255)), \n", + "pipe(\"apple\",\n", + " image=Image.new(\"RGB\", image_dimensions),\n", + " mask_image=Image.new(\"RGB\", image_dimensions, (255, 255, 255)),\n", " guidance_scale=7.5\n", ");" ] diff --git a/notebooks/stable_diffusion/text_to_image.ipynb b/notebooks/stable_diffusion/text_to_image.ipynb index 1dc6e1d55..6f9e7f93f 100644 --- a/notebooks/stable_diffusion/text_to_image.ipynb +++ b/notebooks/stable_diffusion/text_to_image.ipynb @@ -34,11 +34,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you. \n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3iyhJkk)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -167,7 +163,7 @@ "source": [ "pipe = IPUStableDiffusionPipeline.from_pretrained(\n", " \"runwayml/stable-diffusion-v1-5\",\n", - " revision=\"fp16\", \n", + " revision=\"fp16\",\n", " torch_dtype=torch.float16,\n", " n_ipu=n_ipu,\n", " num_prompts=1,\n", diff --git a/notebooks/stable_diffusion/text_to_image_sd2.ipynb b/notebooks/stable_diffusion/text_to_image_sd2.ipynb index c2d7d7640..7b65736e2 100644 --- a/notebooks/stable_diffusion/text_to_image_sd2.ipynb +++ b/notebooks/stable_diffusion/text_to_image_sd2.ipynb @@ -31,11 +31,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3X3IMDh)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -167,7 +163,7 @@ "source": [ "pipe = IPUStableDiffusionPipeline.from_pretrained(\n", " \"stabilityai/stable-diffusion-2\",\n", - " revision=\"fp16\", \n", + " revision=\"fp16\",\n", " torch_dtype=torch.float16,\n", " requires_safety_checker=False,\n", " n_ipu=n_ipu,\n", diff --git a/notebooks/summarization.ipynb b/notebooks/summarization.ipynb index 0aef6bec5..965fec266 100644 --- a/notebooks/summarization.ipynb +++ b/notebooks/summarization.ipynb @@ -44,11 +44,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3Ky3VCj)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -302,7 +298,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, datasets.ClassLabel):\n", @@ -408,7 +404,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)" ] }, @@ -543,7 +539,7 @@ " # Setup the tokenizer for targets\n", " with tokenizer.as_target_tokenizer():\n", " labels = tokenizer(examples[\"summary\"], max_length=max_target_length, padding=\"max_length\", truncation=True)\n", - " \n", + "\n", " # Since we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n", " # padding in the loss.\n", " labels[\"input_ids\"] = [\n", @@ -686,9 +682,9 @@ "ipu_config = IPUConfig.from_pretrained(\n", " ipu_config_name,\n", " executable_cache_dir=executable_cache_dir,\n", - " # -1 wildcard, \n", - " # split encoder and decoder layers evenly across IPUs \n", - " # for inference \n", + " # -1 wildcard,\n", + " # split encoder and decoder layers evenly across IPUs\n", + " # for inference\n", " inference_layers_per_ipu=[-1]\n", ")" ] @@ -787,19 +783,19 @@ " decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)\n", " labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n", " decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n", - " \n", + "\n", " # Rouge expects a newline after each sentence\n", " decoded_preds = [\"\\n\".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]\n", " decoded_labels = [\"\\n\".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]\n", - " \n", + "\n", " result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)\n", " # Extract a few results\n", " result = {key: value.mid.fmeasure * 100 for key, value in result.items()}\n", - " \n", + "\n", " # Add mean generated length\n", " prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]\n", " result[\"gen_len\"] = np.mean(prediction_lens)\n", - " \n", + "\n", " return {k: round(v, 4) for k, v in result.items()}" ] }, diff --git a/notebooks/text_classification.ipynb b/notebooks/text_classification.ipynb index fb8c86f4c..605bf6397 100644 --- a/notebooks/text_classification.ipynb +++ b/notebooks/text_classification.ipynb @@ -72,11 +72,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3XDBUvQ)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -425,7 +421,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, datasets.ClassLabel):\n", @@ -551,7 +547,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)" ] }, diff --git a/notebooks/text_embeddings_models/text-embeddings-on-ipu.ipynb b/notebooks/text_embeddings_models/text-embeddings-on-ipu.ipynb index 82ae73bbe..5bfd5eb94 100644 --- a/notebooks/text_embeddings_models/text-embeddings-on-ipu.ipynb +++ b/notebooks/text_embeddings_models/text-embeddings-on-ipu.ipynb @@ -2,57 +2,54 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ "# General-purpose Text Embeddings on the IPU" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "This notebook describes how to use supported embeddings models to generate SOTA text embeddings on the IPU. You can use the following:\n", "* [E5 model](https://arxiv.org/pdf/2212.03533.pdf) (Emb**E**ddings from bidir**E**ctional **E**ncoder r**E**presentations) to generate text embeddings on the IPU.\n", "* [Sentence Transformers MPNet Base V2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), which is an embeddings model based on the MPNet base model.\n", "* [Sentence-T5](https://arxiv.org/abs/2108.08877), which runs on a pre-trained T5 model encoder." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/cPfXNO)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." - ], - "metadata": {} + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "First, install the requirements for running this notebook:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], "source": [ "# Install Optimum Graphcore if it is not in your environment\n", "! pip install optimum-graphcore==0.7.1 sentence-transformers==2.2.2 graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n", "\n", "%load_ext graphcore_cloud_tools.notebook_logging.gc_logger" - ], - "outputs": [], - "metadata": { - "scrolled": true - } + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In order to improve usability and support for future users, Graphcore would like to collect information about the applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n", "\n", @@ -61,19 +58,20 @@ "- Environment details\n", "\n", "You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Import the required modules for the notebook:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 2, + "metadata": {}, + "outputs": [], "source": [ "import os\n", "import torch\n", @@ -81,12 +79,11 @@ "import numpy as np\n", "from tqdm.notebook import tqdm\n", "import logging" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We need to instantiate some global parameters that will be used to run the models.\n", "\n", @@ -102,12 +99,13 @@ "```\n", "\n", "The model itself, through model pipelining, can also be run over 4 IPUs (by setting `ipus_per_replica` to 4), in which case the replication factor will be adjusted accordingly. The reason we might want to spread the model over more IPUs is to reduce the memory consumption of the model over a single machine allowing for higher batch sizes to be used. For example, with 4 IPUs, we compute far fewer layers per IPU, while with 1 IPU, all model layers are on a single IPU. This is particularly beneficial on an IPU-POD16 machine, as the 4-IPU pipelined version of the model can be run at a higher effective batch size (with a higher micro batch size) and achieve an even higher overall batched throughput." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 3, + "metadata": {}, + "outputs": [], "source": [ "logger = logging.getLogger(\"\")\n", "\n", @@ -118,20 +116,20 @@ "replication_factor = None\n", "\n", "random_seed = 42" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "To run embeddings models, we will set up a generic IPU embeddings class which loads the pre-trained model onto the IPU and runs the embedding pooling and normalisation stages in the forward pass along with the model. You may want to change the internal pooling (`pool(...)`) function in the class to support other pooling methods. The class currently supports averaging and classification using the encoder output state by passing `pool_type` when calling the model. " - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 4, + "metadata": {}, + "outputs": [], "source": [ "import logging\n", "from typing import Optional, List\n", @@ -148,11 +146,11 @@ " self.encoder = to_pipelined(model, ipu_config)\n", " self.encoder = self.encoder.parallelize()\n", " if fp16: self.encoder = self.encoder.half()\n", - " \n", + "\n", " def pool(self, last_hidden_states: torch.Tensor, attention_mask: torch.Tensor, pool_type: str) -> torch.Tensor:\n", - " \n", + "\n", " last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)\n", - " \n", + "\n", " if pool_type == \"avg\":\n", " emb = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]\n", " elif pool_type == \"cls\":\n", @@ -161,28 +159,28 @@ " raise ValueError(f\"pool_type {pool_type} not supported\")\n", "\n", " return emb\n", - " \n", + "\n", " def forward(self, pool_type: str ='avg', **kwargs) -> torch.Tensor:\n", " outputs = self.encoder(**kwargs)\n", - " \n", + "\n", " embeds = self.pool(outputs.last_hidden_state, kwargs[\"attention_mask\"], pool_type=pool_type)\n", " embeds = torch.nn.functional.normalize(embeds, p=2, dim=-1)\n", "\n", " return embeds" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Before setting up and running each of the models, let's create a simple `infer` function which handles loading the batches from a dataloader and generating the embeddings from any model:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 5, + "metadata": {}, + "outputs": [], "source": [ "import time\n", "\n", @@ -193,46 +191,46 @@ " lat = time.time()\n", " outputs = model(**batch_dict)\n", " lat = time.time() - lat\n", - " \n", + "\n", " encoded_embeds.append(outputs)\n", " print(f\"batch len: {len(batch_dict['input_ids'])} | batch latency: {lat}s | per_sample: {lat/len(batch_dict['input_ids'])}s | throughput: {len(batch_dict['input_ids'])/lat} samples/s\")\n", - " \n", + "\n", " return torch.cat(encoded_embeds, axis=0)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Embeddings with E5-Large" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "First, use `AutoConfig` from Transformers to load the model config for the E5 large model. E5 uses a bidirectional encoder, essentially the encoder stage of a BERT model, to generate the trained embeddings. The config will define the architecture of the model, such as the number of encoder layers and size of the hidden dimension within the model. The sequence length for the model is set by default to the maximum defined sequence length in the model config (`max_position_embeddings`) and can be adjusted by changing the `e5_seq_len` parameter, with a maximum value of 512.\n", "\n", "We also need to tokenize the dataset. For this we define a custom transform function which applies the pre-trained tokenization for each model to the dataset. We will call this function when loading the function, to avoid loading multiple tokenized datasets at the same time.\n", "\n", "We define some IPU-specific configurations to get the most out of the model. The `get_ipu_config` function will set up the IPU config according to the model config, taking into consideration the defined number of IPUs for model parallelism, the number of IPUs available and batching configurations." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 6, + "metadata": {}, + "outputs": [], "source": [ "from config import get_ipu_config\n", "from transformers import AutoConfig, AutoTokenizer, BatchEncoding" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 7, + "metadata": {}, + "outputs": [], "source": [ "e5_model_name = 'intfloat/e5-large'\n", "e5_tokenizer = AutoTokenizer.from_pretrained(e5_model_name)\n", @@ -251,27 +249,27 @@ "\n", "e5_ipu_config = get_ipu_config(\n", " e5_model_config, n_ipu, ipus_per_replica, device_iterations, replication_factor, random_seed)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Embeddings with All-MPNet" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "For MPNet, we do the same for the pre-trained model. The maximum value for the sequence length is 512." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 8, + "metadata": {}, + "outputs": [], "source": [ "mpnet_model_name = 'sentence-transformers/all-mpnet-base-v2'\n", "mpnet_tokenizer = AutoTokenizer.from_pretrained(mpnet_model_name)\n", @@ -287,32 +285,32 @@ " padding=\"max_length\",\n", " truncation=True\n", " )\n", - " \n", + "\n", "mpnet_ipu_config = get_ipu_config(\n", " mpnet_model_config, n_ipu, ipus_per_replica, device_iterations, replication_factor, random_seed)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Embeddings with Sentence-T5" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Note that for T5, we need to use `T5EncoderModel` instead of `AutoModel`. We must manually specify the encoder as T5 is an encoder-decoder model, and we don't want to load the decoder for embeddings generation. \n", "\n", "Transformers `AutoModel` supports a 1-to-1 mapping of architecture definitions to model types, and it will load the `T5Model` class by default. We can override this by directly importing and loading the pre-trained model using `T5EncoderModel`. For T5 the sequence length is determined by the `n_positions` parameter in the model config. The expected maximum sequence length for T5 is also 512." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from transformers.models.t5.modeling_t5 import T5EncoderModel\n", "\n", @@ -335,19 +333,18 @@ "\n", "t5_ipu_config = get_ipu_config(\n", " t5_model_config, n_ipu, ipus_per_replica, device_iterations * 4, replication_factor, random_seed)\n" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Creating the embeddings model" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We'll wrap this behaviour into a simple function so we can iteratively run all three models and initialise `poptorch.Dataloader` to create an IPU-ready batched dataloader. We pass an arbitrary call to the model using the first batch to ensure we have compiled the model executable (or loaded the already compiled executable).\n", "\n", @@ -356,12 +353,13 @@ "2. Converts the IPU config into an IPU options object and passes this to a `poptorch.inferenceModel` wrapper to prepare the model for the IPU.\n", "3. Initialises [`poptorch.Dataloader`](https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/batching.html) to batch the data according to the IPU options and the defined micro batch size.\n", "4. Runs the model once with a batch to compile or load the compiled executable. " - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 10, + "metadata": {}, + "outputs": [], "source": [ "from transformers import default_data_collator as data_collator\n", "\n", @@ -383,47 +381,47 @@ "\n", " model(**next(iter(dataloader)))\n", " return model, dataloader" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Let's load a dataset we'll use to try out the models. Using the Hugging Face `datasets` library we can load a pre-existing dataset from the Hugging Face Hub. In this case, let's use the `rotten_tomatoes` film review dataset. Later in the notebook, we will use this dataset to create a basic semantic search functionality." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from datasets import Dataset, load_dataset\n", "dataset = load_dataset(\"rotten_tomatoes\")\n", "print(dataset)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Run the E5 model" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The dataset first needs to be tokenized using the pre-trained tokenizer for each model, we can use the `map()` method to tokenize each of the inputs of the dataset using the model-specific transform function. Then we can convert the Hugging Face Arrow format dataset to a PyTorch-ready dataset with `set_format` which converts the tokenized inputs into tensors.\n", "\n", "To run the model, simply call the `infer` function we created earlier to generate embeddings for the full dataset. " - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "tokenized_dataset = dataset.map(e5_transform_func, batched=True)\n", "tokenized_dataset.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\"])\n", @@ -434,20 +432,20 @@ "e5_data_embeddings = infer(model, dataloader)\n", "\n", "model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Run the All-MPNet model" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "tokenized_dataset = dataset.map(mpnet_transform_func, batched=True)\n", "tokenized_dataset.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\"])\n", @@ -457,20 +455,20 @@ "mpnet_data_embeddings = infer(model, dataloader)\n", "\n", "model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Run the Sentence-T5 model" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "tokenized_dataset = dataset.map(t5_transform_func, batched=True)\n", "tokenized_dataset.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\"])\n", @@ -483,38 +481,40 @@ "t5_data_embeddings = infer(model, dataloader)\n", "\n", "model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The embeddings for a single sequence represent low-dimensional numerical representations of the word-level and sentence-level context for each token. These pre-trained embeddings can be used in applications like embedding retrieval for recommender systems, or semantic searches for query-matching using cosine-similarity. Both of these use cases take advantage of the generated embeddings space, by performing a relative comparison of the user input sequence embeddings using some proximity metric.\n", "\n", "We'll use the open source `sentence_transformers` library which provides utilities for embeddings tasks to perform a semantic search on a user query to retrieve the sequences from the dataset that are most similar to the query. This is a helpful utility for making, for example, more responsive FAQs." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Semantic search with generated embeddings" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Using the `rotten_tomatoes` dataset, lets create a simple similarity search engine using the `sentence_transformers` semantic search function, which uses cosine similarity to retrieve close-proximity sentences from a given set of embeddings to a given query. We have already generated embeddings for the dataset, so the next step is to do the same with a given query and perform the search.\n", "\n", "First, to process the query, we need to tokenize it and convert it to a single-batch input for the model. This has been wrapped into a simple function which tokenizes and prepares a dictionary of model inputs (`input_ids`, `attention_mask`, ...) to which we just need to pass a string." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], "source": [ "def prepare_query(query: str):\n", " t_query = mpnet_tokenizer(\n", @@ -525,22 +525,20 @@ " )\n", "\n", " return {k: torch.as_tensor([t_query[k]]) for k in t_query}" - ], - "outputs": [], - "metadata": { - "scrolled": true - } + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Next, to perform inference with a single input (so an effective batch size of 1) we re-instantiate the model by setting all device batching, replication and micro batch-size to 1 and re-compile the model. For this example, we use the All-MPNet model. The change in batch size necessitates a recompilation, since the input shape to the model has been changed. We will follow the steps to initiate the model outlined earlier in the notebook, with the only change being setting the `get_ipu_config` function to have all batching turned off." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "mpnet_infer_ipu_config = get_ipu_config(\n", " mpnet_model_config, n_ipu, ipus_per_replica=1, device_iterations=1, replication_factor=1, random_seed=random_seed)\n", @@ -549,20 +547,20 @@ "model = poptorch.inferenceModel(model, mpnet_infer_ipu_config.to_options(for_inference=True))\n", "\n", "o=model(**prepare_query(\"Running once to compile\"))" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Finally, we can use the model to embed a single query, and perform a semantic search across the full dataset embeddings to retrieve highly relevant reviews to the query." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from sentence_transformers.util import semantic_search\n", "\n", @@ -574,25 +572,23 @@ "print(f\"\\n SEARCH QUERY: {query}\")\n", "for n, res in enumerate(hits[0]):\n", " print(f\"\\n Result (rank {n+1}) | Score: {res['score']} | Text: {dataset['train']['text'][res['corpus_id']]} \")" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, - "source": [], + "metadata": {}, "outputs": [], - "metadata": {} + "source": [] } ], "metadata": { @@ -616,4 +612,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +} diff --git a/notebooks/text_summarization.ipynb b/notebooks/text_summarization.ipynb index 7a9903b99..967e80250 100644 --- a/notebooks/text_summarization.ipynb +++ b/notebooks/text_summarization.ipynb @@ -2,13 +2,14 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ "Copyright (c) 2023 Graphcore Ltd. All rights reserved." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "# Text Summarization on IPUs using BART-L - Inference\n", "\n", @@ -18,44 +19,40 @@ "| Domain | Tasks | Model | Datasets | Workflow | Number of IPUs | Execution time |\n", "|---------|-------|-------|----------|----------|--------------|--------------|\n", "| NLP | Text summarization | BART-L | - | Inference | Recommended: 2 | 5 min |\n" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/49bCUB)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." - ], - "metadata": {} + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Requirements\n", "\n", "Before running the model on IPUs you have to install the Python dependencies:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%pip install optimum-graphcore==0.7.1 wikipedia graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n", "\n", "%load_ext graphcore_cloud_tools.notebook_logging.gc_logger" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In order to improve usability and support for future users, Graphcore would like to collect information about the applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n", "\n", @@ -64,32 +61,33 @@ "- Environment details\n", "\n", "You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "import os\n", "\n", "exec_cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\")" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Model preparation\n", "\n", "We start by preparing the model. First, we define the configuration needed to run the model on the IPU. `IPUConfig` is a class that specifies attributes and configuration parameters to compile and put the model on the device:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from optimum.graphcore import IPUConfig\n", "\n", @@ -104,20 +102,20 @@ " \"on_device_generation_steps\": 16,\n", " }\n", ")" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Next, let's import `pipeline` from `optimum.graphcore` and create our summarization pipeline:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from optimum.graphcore import pipeline\n", "\n", @@ -130,90 +128,90 @@ " max_input_length=1024,\n", " truncation=True\n", ")" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We define an input to test the model." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "input_test = 'In computing, a compiler is a computer program that translates computer code written in one programming language (the source language) into another language (the target language). The name \"compiler\" is primarily used for programs that translate source code from a high-level programming language to a low-level programming language (e.g. assembly language, object code, or machine code) to create an executable program.'\n", "input_test" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Compilation time for the 1st run: ~ 2:30" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%%time\n", "summarizer(input_test, max_length=150, num_beams=3)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## A fairy tale long story short..." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The first call to the pipeline was a bit slow, it took several seconds to provide the answer. This behaviour is due to compilation of the model which happens on the first call.\n", "On subsequent prompts it is much faster:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "the_princess_and_the_pea = 'Once upon a time there was a prince who wanted to marry a princess; but she would have to be a real princess. He travelled all over the world to find one, but nowhere could he get what he wanted. There were princesses enough, but it was difficult to find out whether they were real ones. There was always something about them that was not as it should be. So he came home again and was sad, for he would have liked very much to have a real princess. One evening a terrible storm came on; there was thunder and lightning, and the rain poured down in torrents. Suddenly a knocking was heard at the city gate, and the old king went to open it. It was a princess standing out there in front of the gate. But, good gracious! what a sight the rain and the wind had made her look. The water ran down from her hair and clothes; it ran down into the toes of her shoes and out again at the heels. And yet she said that she was a real princess. Well, we\\'ll soon find that out, thought the old queen. But she said nothing, went into the bed-room, took all the bedding off the bedstead, and laid a pea on the bottom; then she took twenty mattresses and laid them on the pea, and then twenty eider-down beds on top of the mattresses. On this the princess had to lie all night. In the morning she was asked how she had slept. \"Oh, very badly!\" said she. \"I have scarcely closed my eyes all night. Heaven only knows what was in the bed, but I was lying on something hard, so that I am black and blue all over my body. It\\'s horrible!\" Now they knew that she was a real princess because she had felt the pea right through the twenty mattresses and the twenty eider-down beds. Nobody but a real princess could be as sensitive as that. So the prince took her for his wife, for now he knew that he had a real princess; and the pea was put in the museum, where it may still be seen, if no one has stolen it. There, that is a true story.'\n", "the_princess_and_the_pea" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%%time\n", "summarizer(the_princess_and_the_pea, max_length=150, num_beams=3)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Summarization of Wikipedia articles\n", "Now let's use the Wikipedia API to search for some long text that can be summarized:" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "import wikipedia\n", "\n", @@ -221,13 +219,13 @@ "page_title = \"Queen (band)\"\n", "text = wikipedia.page(page_title).content\n", "text" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%%time\n", "summarizer(\n", @@ -235,40 +233,40 @@ " max_length=150,\n", " num_beams=3,\n", ")" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Summarization of medical health records\n", "The summarization task may be also useful in summarising medical health records (MHR). Let's import an open source dataset with some medical samples." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "dataset = load_dataset(\"rungalileo/medical_transcription_40\")\n", "dataset" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We focus on the medical report labeled as \"text\" and from the training dataset select a random patient ID." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "import random\n", "\n", @@ -277,48 +275,46 @@ "\n", "exemplary_medical_report = dataset[\"train\"][random_patient_id][\"text\"]\n", "exemplary_medical_report" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%%time\n", "summarizer(exemplary_medical_report, max_length=150, num_beams=3)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Optional - Release IPUs in use\n", "\n", "The IPython kernel has a lock on the IPUs used to run the model, preventing other users from using them. For example, if you wish to use other notebooks after working your way through this one, it may be necessary to manually run the below cell to release IPUs from use. This will happen by default if you use the \"Run All\" option. More information on the topic can be found at [Managing IPU Resources](https://github.com/gradient-ai/Graphcore-HuggingFace/blob/main/useful-tips/managing_ipu_resources.ipynb)." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "summarizer.model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Conclusions and next steps\n", "\n", "This notebook demonstrated running a text summarization task on Graphcore IPUs, with BART-L using an inference pipeline from Optimum Graphcore.\n", "\n", "Try out the other [IPU-powered Jupyter Notebooks](https://www.graphcore.ai/ipu-jupyter-notebooks) to see how how IPUs perform on other tasks." - ], - "metadata": {} + ] } ], "metadata": { @@ -342,4 +338,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} \ No newline at end of file +} diff --git a/notebooks/token_classification.ipynb b/notebooks/token_classification.ipynb index 6a7a0a0fe..43419a9f1 100644 --- a/notebooks/token_classification.ipynb +++ b/notebooks/token_classification.ipynb @@ -53,10 +53,6 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3YCsqT1)\n", - "\n", "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, @@ -446,7 +442,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, ClassLabel):\n", @@ -505,7 +501,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)" ] }, diff --git a/notebooks/translation.ipynb b/notebooks/translation.ipynb index a910aea5f..1250efccc 100644 --- a/notebooks/translation.ipynb +++ b/notebooks/translation.ipynb @@ -43,11 +43,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](images/gradient-badge.svg)]()\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { @@ -281,7 +277,7 @@ " while pick in picks:\n", " pick = random.randint(0, len(dataset)-1)\n", " picks.append(pick)\n", - " \n", + "\n", " df = pd.DataFrame(dataset[picks])\n", " for column, typ in dataset.features.items():\n", " if isinstance(typ, datasets.ClassLabel):\n", @@ -383,7 +379,7 @@ "outputs": [], "source": [ "from transformers import AutoTokenizer\n", - " \n", + "\n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)" ] }, @@ -530,7 +526,7 @@ " # Setup the tokenizer for targets\n", " with tokenizer.as_target_tokenizer():\n", " labels = tokenizer(targets, max_length=max_target_length, padding=\"max_length\", truncation=True)\n", - " \n", + "\n", " # Since we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n", " # padding in the loss.\n", " labels[\"input_ids\"] = [\n", @@ -666,9 +662,9 @@ "ipu_config = IPUConfig.from_pretrained(\n", " ipu_config_name,\n", " executable_cache_dir=executable_cache_dir,\n", - " # -1 wildcard, \n", - " # split encoder and decoder layers evenly across IPUs \n", - " # for inference \n", + " # -1 wildcard,\n", + " # split encoder and decoder layers evenly across IPUs\n", + " # for inference\n", " inference_layers_per_ipu=[-1]\n", ")" ] diff --git a/notebooks/wav2vec2/wav2vec2-fine-tuning-checkpoint.ipynb b/notebooks/wav2vec2/wav2vec2-fine-tuning-checkpoint.ipynb index 145989352..4ac6bd556 100644 --- a/notebooks/wav2vec2/wav2vec2-fine-tuning-checkpoint.ipynb +++ b/notebooks/wav2vec2/wav2vec2-fine-tuning-checkpoint.ipynb @@ -58,11 +58,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3CGkbMq)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/wav2vec2/wav2vec2-inference-checkpoint.ipynb b/notebooks/wav2vec2/wav2vec2-inference-checkpoint.ipynb index bfc8409c4..7b56e56b8 100644 --- a/notebooks/wav2vec2/wav2vec2-inference-checkpoint.ipynb +++ b/notebooks/wav2vec2/wav2vec2-inference-checkpoint.ipynb @@ -38,11 +38,7 @@ "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/3CHite5)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine." ] }, { diff --git a/notebooks/whisper-quantized-example.ipynb b/notebooks/whisper-quantized-example.ipynb index 786bdec91..74734c47f 100644 --- a/notebooks/whisper-quantized-example.ipynb +++ b/notebooks/whisper-quantized-example.ipynb @@ -2,6 +2,7 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ "# Speech Transcription on IPUs using Whisper - Quantized Inference\n", "\n", @@ -19,49 +20,45 @@ "> **Hardware requirements:** All the Whisper models from `whisper-tiny` to `whisper-large-v2` can run in inference mode on smallest IPU-POD4 machine.\n", "\n", "[![Join our Slack Community](https://img.shields.io/badge/Slack-Join%20Graphcore's%20Community-blue?style=flat-square&logo=slack)](https://www.graphcore.ai/join-community)" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Environment setup\n", "\n", - "The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n", - "\n", - "[![Run on Gradient](https://assets.paperspace.io/img/gradient-badge.svg)](https://ipu.dev/Djq2SC)\n", - "\n", - "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." - ], - "metadata": {} + "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide for your system](https://docs.graphcore.ai/en/latest/getting-started.html) for details on how to do this. Also refer to the Jupyter Quick Start guide for how to set up Jupyter to be able to run this notebook on a remote IPU machine." + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Dependencies" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Install the dependencies the notebook needs." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ - "# Install optimum from source \n", + "# Install optimum from source\n", "!pip install optimum-graphcore==0.7.1 transformers librosa matplotlib graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n", "%load_ext graphcore_cloud_tools.notebook_logging.gc_logger" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In order to improve usability and support for future users, Graphcore would like to collect information about the applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n", "\n", @@ -70,19 +67,20 @@ "- Environment details\n", "\n", "You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "IPU Whisper with group quantization requires features from Poplar SDK version 3.3 or later. The following code checks whether these features can be enabled." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "import warnings\n", "from transformers.utils.versions import require_version\n", @@ -94,22 +92,22 @@ "except Exception:\n", " enable_sdk_features=False\n", " warnings.warn(\"SDK versions earlier than 3.3 do not support the functionality in this notebook. We recommend that you relaunch the Paperspace Notebook with the PyTorch SDK 3.3 image. You can use https://hub.docker.com/r/graphcore/pytorch-early-access\")\n" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Running Whisper on the IPU\n", "\n", "We start by importing the required modules, some of which are needed to configure the IPU.\n" - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "# Generic imports\n", "import os\n", @@ -127,23 +125,22 @@ "\n", "# HF-related imports\n", "from transformers import WhisperForConditionalGeneration" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "This notebook demonstrates how to run all sizes of Whisper. All sizes will fit on an IPU-POD4:\n", "\n", "- `whisper-tiny`, `base` and `small` only require 1 IPU\n", "- `whisper-medium` requires 2 IPUs\n", "- `whisper-large` requires 4 IPUs" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The Whisper model is available on Hugging Face in several sizes, from `whisper-tiny` with 39M parameters to `whisper-large` with 1550M parameters.\n", "\n", @@ -154,42 +151,43 @@ "\n", "The `IPUConfig` object helps to configure the model to be pipelined across the IPUs.\n", "The number of transformer layers per IPU can be adjusted by using `layers_per_ipu`." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "num_available_ipus=int(os.getenv(\"NUM_AVAILABLE_IPU\", 4))\n", "cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"./exe_cache\")\n", "\n", "configs = {\n", - " \"tiny\": (\"openai/whisper-tiny.en\", \n", + " \"tiny\": (\"openai/whisper-tiny.en\",\n", " IPUConfig(executable_cache_dir=cache_dir,\n", " ipus_per_replica=1,\n", " explicit_ir_inference=True,\n", " )),\n", - " \n", - " \"base\": (\"openai/whisper-base.en\", \n", + "\n", + " \"base\": (\"openai/whisper-base.en\",\n", " IPUConfig(executable_cache_dir=cache_dir,\n", " ipus_per_replica=1,\n", " explicit_ir_inference=True,\n", " )),\n", "\n", - " \"small\": (\"openai/whisper-small.en\", \n", + " \"small\": (\"openai/whisper-small.en\",\n", " IPUConfig(executable_cache_dir=cache_dir,\n", " ipus_per_replica=1,\n", " explicit_ir_inference=True,\n", " )),\n", - " \n", + "\n", " \"medium\": (\"openai/whisper-medium.en\",\n", " IPUConfig(executable_cache_dir=cache_dir,\n", " ipus_per_replica=2,\n", " explicit_ir_inference=True,\n", " )),\n", "\n", - " \"large\": (\"openai/whisper-large-v2\", \n", + " \"large\": (\"openai/whisper-large-v2\",\n", " IPUConfig(executable_cache_dir=cache_dir,\n", " ipus_per_replica=4,\n", " layers_per_ipu=[-1, -1, 14, 18],\n", @@ -206,41 +204,41 @@ " model_checkpoint = custom_checkpoint\n", "\n", " print(f\"Using whisper-{size} config with the checkpoint '{model_checkpoint}'.\")\n", - " return model_checkpoint, ipu_config " - ], - "outputs": [], - "metadata": {} + " return model_checkpoint, ipu_config" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Select the Whisper size bellow, try `\"tiny\"`,`\"base\"`, `\"small\"`, `\"medium\"`, `\"large\"`." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, - "source": [ - "model_checkpoint, ipu_config = select_whisper_config(\"tiny\", custom_checkpoint=None) " - ], + "metadata": {}, "outputs": [], - "metadata": {} + "source": [ + "model_checkpoint, ipu_config = select_whisper_config(\"tiny\", custom_checkpoint=None)" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "You can also use a custom checkpoint from Hugging Face Hub using the argument `custom_checkpoint` above. In this case, you have to make sure that `size` matches the checkpoint model size.\n", "\n", "Two features of Optimum Graphcore are demonstrated below:\n", "1. `use_cond_encoder` : This enables putting the Whisper encoder and decoder on a single IPU and switching between them using a compiled `cond` operation. This is only available if `ipus_per_replica == 1`.\n", "2. `use_group_quantized_linears` : This enables compressing all the weights of the transformer block's linear layers to INT4 using group quantization." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "# Instantiate processor and model\n", "processor = WhisperProcessorTorch.from_pretrained(model_checkpoint)\n", @@ -251,8 +249,8 @@ "\n", "pipelined_model = to_pipelined(model, ipu_config)\n", "pipelined_model = pipelined_model.parallelize(\n", - " for_generation=True, \n", - " use_cache=True, \n", + " for_generation=True,\n", + " use_cache=True,\n", " batch_size=1,\n", " num_beams=num_beams,\n", " max_length=448,\n", @@ -261,22 +259,22 @@ " use_cond_encoder=ipu_config.ipus_per_replica == 1,\n", " use_group_quantized_linears=True # Enables quantization!\n", ").half()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Now we can load the dataset and process an example audio file.\n", "If precompiled models are not available, then the first run of the model triggers two graph compilations.\n", "This means that our first test transcription could take a minute or two to run, but subsequent runs will be much faster." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "# load the dataset and read an example sound file\n", "ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n", @@ -291,29 +289,29 @@ " input_features,\n", " use_cache=True,\n", " num_beams=num_beams,\n", - " max_length=448, \n", + " max_length=448,\n", " min_length=3)\n", " transcription = processor.batch_decode(sample_output, skip_special_tokens=True)[0]\n", " return transcription\n", "\n", "test_transcription = transcribe(test_sample[\"audio\"][\"array\"], sample_rate)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In the next cell, we compare the expected text from the dataset with the transcribed result from the model.\n", "There will typically be some small differences, but even `whisper-tiny` does a great job! It even adds punctuation.\n", "\n", "You can listen to the audio and compare the model result yourself using the controls below." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "print(f\"Expected: {test_sample['text']}\\n\")\n", "print(f\"Transcribed: {test_transcription}\")\n", @@ -321,23 +319,23 @@ "plt.figure(figsize=(14, 5))\n", "librosa.display.waveshow(test_sample[\"audio\"][\"array\"], sr=sample_rate)\n", "IPython.display.Audio(test_sample[\"audio\"][\"array\"], rate=sample_rate)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The model only needs to be compiled once. Subsequent inferences will be much faster.\n", "In the cell below, we repeat the exercise but with a random example from the dataset.\n", "\n", "You might like to re-run this next cell multiple times to get different comparisons." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "idx = random.randint(0, ds.num_rows - 1)\n", "data = ds[idx][\"audio\"][\"array\"]\n", @@ -349,28 +347,27 @@ "plt.figure(figsize=(14, 5))\n", "librosa.display.waveshow(data, sr=sample_rate)\n", "IPython.display.Audio(data, rate=sample_rate, autoplay=True)" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Finally, we detach the process from the IPUs when we are done to make the IPUs available to other users." - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "pipelined_model.detachFromDevice()" - ], - "outputs": [], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Next Steps\n", "\n", @@ -387,18 +384,17 @@ "You can also try using beam search by setting `num_beams>1` in the calls to `parallelize` and `generate` above. `whisper-small` will fit on 1 IPU with `num_beams=5`.\n", "\n", "For `whisper-medium` with `num_beams>1` the model will need 4 IPUs to fit. For `whisper-large` with `num_beams>1` you will need more than the 4 IPUS in an IPU-POD4. On Paperspace, you can use either an IPU-POD16 or a Bow Pod16 machine, each with 16 IPUs. Please contact Graphcore if you need assistance running these larger models.\n" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Conclusion\n", "\n", "In this notebook, we demonstrated using Whisper and group quantization for speech recognition and transcription on the IPU.\n", "We used the Optimum Graphcore package to interface between the IPU and the 🤗 Transformers library. This meant that only a few lines of code were needed to get this state-of-the-art automated speech recognition model running on IPUs." - ], - "metadata": {} + ] } ], "metadata": { @@ -422,4 +418,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +}