Py214_PhylogeneticTrees_ETE3.BEG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computing and parsing phylogenetic trees\n",
    "(Víctor Sojo | vsojo@amnh.org)\n",
    "\n",
    "In the previous lesson we installed and used MAFFT to align multiple sequences of the 12S rRNA gene for several mammals. Here we will use that alignment to compute a phylogenetic tree using FastTree.\n",
    "\n",
    "We will then parse this tree using ETE3, a mighty tool for analysing phylogenetic trees.\n",
    "\n",
    "Both of these tools are available through `conda`, so we will install and load them from there.\n",
    "\n",
    "**References**<br/>\n",
    "+ The help for FastTree.\n",
    "+ The [ETE manual](http://etetoolkit.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "&emsp;[Installing required software via conda](#Installing-required-software-via-conda)<br/>\n",
    "&emsp;[Importing required modules](#Importing-required-modules)<br/>\n",
    "&emsp;[Exploring the alignment file](#Exploring-the-alignment-file)<br/>\n",
    "&emsp;[Computing a phylogenetic tree with FastTree](#Computing-a-phylogenetic-tree-with-FastTree)<br/>\n",
    "&emsp;[Using ETE to parse and manipulate phylogenetic trees](#Using-ETE-to-parse-and-manipulate-phylogenetic-trees)<br/>\n",
    "&emsp;&emsp;[Loading a tree into ete3](#Loading-a-tree-into-ete3)<br/>\n",
    "&emsp;&emsp;[Exploring the tree's leaves](#Exploring-the-tree's-leaves)<br/>\n",
    "&emsp;&emsp;[Exploring a tree's children \\(i.e. the nodes under a parent node\\)](#Exploring-a-tree's-children-\\(i.e.-the-nodes-under-a-parent-node\\))<br/>\n",
    "&emsp;&emsp;[Getting the common ancestor of a group of taxa](#Getting-the-common-ancestor-of-a-group-of-taxa)<br/>\n",
    "&emsp;&emsp;[Rooting or re-rooting a tree](#Rooting-or-re-rooting-a-tree)<br/>\n",
    "&emsp;&emsp;[Selecting a branch by name using &](#Selecting-a-branch-by-name-using-&)<br/>\n",
    "&emsp;&emsp;[Distances between nodes](#Distances-between-nodes)<br/>\n",
    "&emsp;&emsp;[Naming internal nodes](#Naming-internal-nodes)<br/>\n",
    "&emsp;&emsp;[Annotating a tree](#Annotating-a-tree)<br/>\n",
    "&emsp;&emsp;[Making a copy of a tree](#Making-a-copy-of-a-tree)<br/>\n",
    "&emsp;&emsp;[Deleting \\(detaching\\) leaves in a tree](#Deleting-\\(detaching\\)-leaves-in-a-tree)<br/>\n",
    "&emsp;&emsp;[Pruning a tree](#Pruning-a-tree)<br/>\n",
    "&emsp;&emsp;[Saving a tree to a Newick file](#Saving-a-tree-to-a-Newick-file)<br/>\n",
    "&emsp;&emsp;[Checking for monophyly](#Checking-for-monophyly)<br/>\n",
    "&emsp;&emsp;[Comparing trees](#Comparing-trees)<br/>\n",
    "&emsp;&emsp;[Creating and exporting tree images with ete3](#Creating-and-exporting-tree-images-with-ete3)<br/>\n",
    "&emsp;&emsp;[Exporting a tree image to a file](#Exporting-a-tree-image-to-a-file)<br/>\n",
    "&emsp;&emsp;[Colouring branches to match taxonomy](#Colouring-branches-to-match-taxonomy)<br/>\n",
    "&emsp;&emsp;[Advanced ETE tree styling](#Advanced-ETE-tree-styling)<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, let's make sure that we're using the `bioinfo` environment that we created in the `Py201` notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(If you're on Windows, remember that every line with `!my code` should be changed to `!wsl my code` and you should have an active [WSL installation](https://docs.microsoft.com/en-us/windows/wsl/install-win10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing required software via `conda`\n",
    "We have aligned sequences from the previous lesson. Here we want to align them. There are multiple packages to produce phylogenetic trees. Two very popular ones are `iqtree` and `fasttree`. Both of these are maximum-likelihood applications, and both are on `conda` (as long as you have the `bioconda` channel, which we added on an earlier lesson).\n",
    "\n",
    "FastTree does what it says on the tin, and it is great for exploration or when you need to produce hundreds or thousands of trees quickly. For more reliability, you may want to use IQ-Tree instead, which by default performs a very thorough exploration of a vast array of maximum-likelihood models, performs statistical tests on them, and then gives out the best (most likely) one. Here, however, we will use `fasttree`, merely because it's quicker.\n",
    "\n",
    "Once we have a tree, we will want to parse it. For that, we will use the Python Environment for Tree Exploration (better known as ETE) version 3.\n",
    "\n",
    "Let's have conda download and install `ete3`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now `fasttree`, if you're on Mac/Linux (won't work on Windows):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "⚠️ **Windows users:** ⚠️ `fasttree` is a shell application, so `conda` will not install it on Windows (even though technically it could, through the WSL). You could try to install it yourself following the instructions [here](http://www.microbesonline.org/fasttree/#Install), but this can get a little involved (I'd recommend installing the Linux versions on your Ubuntu console). Therefore, here I will simply ask you to download the trees instead of computing them yourself (sorry!).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I already have both packages installed, so it's only a few seconds while `conda` checks for updates and then tells me that `All requested packages already installed`. It may take a couple of minutes in your case if you don't have the packages installed or need to update."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing required modules\n",
    "Here we will need:\n",
    "\n",
    "Module             | Use\n",
    ":------------------|:-----------------------------------------\n",
    "**ete3.Tree**      | To read, parse, draw, and manipulate phylogenetic trees\n",
    "**ete3.TreeStyle** | To manipulate the visual style and overall tree looks\n",
    "**ete3.NodeStyle** | To manipulate the visual style of specific tree nodes\n",
    "\n",
    "We will also use the `fasttree` package of `conda`, but this is not a Python module and so we don't need to import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the alignment file\n",
    "We have an alignment from the previous lesson, let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the number of sequences by asking `grep -c` to find all the lines that start (`^`) with `>`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have `23` sequences, and as you can tell from the occassional dashes within the sequences, these sequences are aligned.\n",
    "\n",
    "Don't be confused with the `--` that separates each result. These are inserted by `grep` to make its output easier to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing a phylogenetic tree with FastTree\n",
    "Let's start by creating a file name based on the name of the alignment file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's compute the tree using `fasttree`. We use the `-nt` flag to let FastTree know that we're providing nucleotide sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastTree just made a file in the NEWICK format. We can try to take a look at it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...but as you can see, that's not particularly easy to interpret. To visualise this file, we can open it in a tree-viewing software such as [FigTree](http://tree.bio.ed.ac.uk/software/figtree/) (which unfortunately was last updated in Nov. 2020 and does not work well with macOS Big Sur) or upload it to [iTOL](https://itol.embl.de/upload.cgi) and see it on their website.\n",
    "\n",
    "Here's what I see on FigTree:\n",
    "![Screenshot of a tree of mammalian 12S rRNA sequences in FigTree](https://raw.githubusercontent.com/vsojo/Python_Workshop/DataAnalysis/Mammals_12S_rRNA.mafft.FastTree.nwk.jpg \"FastTree ML Tree of MAFFT alignment of DNA sequences encoding 12S rRNA from 23 mammals.\")\n",
    "\n",
    "This looks great... other than the fact that beavers (which are rodents) ended up being more related to humans than to house mice (another rodent)...\n",
    "\n",
    "Other than that weirdness with the mice refusing to join the beavers, some other valid interesting relationships do appear: the three marsupials are neatly together, the carnivores form a big joint clade, and the whale groups with the cow and pig.\n",
    "\n",
    "Something else worth noting: giant pandas and red pandas don't seem to be as closely related as their names would suggest. Instead, giant pandas are very close to bears, whereas the red panda clusters with the musteloids (raccoons, otters and so on). This matches what is commonly accepted in the literature at present.\n",
    "\n",
    "Why we got that weird mouse/beaver/human result is an interesting question that we won't solve here. Maybe we need more primate and rodent sequences; maybe just using the 12S rRNA gene does not give enough information to tell these lineages apart; maybe FastTree was too _fast_ for its own good; maybe there was something wrong with the default parameters we used for the alignment; or maybe we just picked the wrong evolutionary model.\n",
    "\n",
    "This is not a workshop on phylogeny, but I just wanted to show that:\n",
    "\n",
    "⚠️ **Just because you build a phylogenetic tree, it doesn't mean you'll get the _correct_ relationships. All trees (and multiple-sequence alignments) are hypotheses, and as such they can be right or they can be wrong.** ⚠️"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using ETE to parse and manipulate phylogenetic trees\n",
    "ETE is a great tool for processing, printing, analysing, and manipulating phylogenetic trees. We will only take a very superficial look at it. You are strongly encouraged to delve deep into its mighty powers on [their website](http://etetoolkit.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading a tree into ete3\n",
    "Let's start by loading the tree that we made above (whose file name is still stored in the variable `tree_file`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(you can of course name your tree anything you want; I named it `fstree` because we made it with FastTree, but many people just call their trees `t`)\n",
    "\n",
    "Typically a good idea to just print something to see what it holds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! ete3 prints the topology of the tree as an ASCII text. This can be very helpful on its own.\n",
    "\n",
    "You may think that this is not the same tree we saw on FigTree. It is; the difference is that in the image above I had re-rooted the tree around the marsupials (which I assumed should be an outgroup to all the other mammals that we have here. We can root and re-root trees with ete3 as well, and we will do so later in this lesson.\n",
    "\n",
    "But for now, let's explore our tree a bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the tree's leaves\n",
    "Following the analogy of a tree, the convention is to call the tips of a tree, i.e. the taxa, \"leaves\". We can easily explore these in ete3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring a tree's children (i.e. the nodes under a parent node)\n",
    "We can also explore the separate sub-trees of the tree, which in node jargon are called \"children\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we try to print the whole of `children`, we get something not so easy to interpret:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not too easy to make sense of, but it does hint at something important:\n",
    "\n",
    "⚠️ ETE children are all the **nodes** in a tree, whether they be _outer/leaf/tip nodes_ (the taxa), or _internal nodes_ (everythinig else, i.e. all the ancestral nodes, such as that node between PolarBear & BrownBear, and the one between their ancestor and BlackBear, and then the one between the common ancestor of all bears and GiantPanda).\n",
    "\n",
    "\n",
    "Instead of printing all of `children`, we can do a for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look again at the full tree, and you'll see that it indeed has three sub-trees.\n",
    "\n",
    "We can get the children of the children of the children... for as many sub-levels as there are in the tree. For example, to get the **Marsupials**, we would first need child `2` of the original tree (the last one printed above). Of this sub-tree, child `0` has the two cats, and the other child (`1`) has everything else, so we pick `1`. Then, child `1` has the big group with Human, NthAmBeaver, HouseMouse, and the Marsupials. Child `1` of that has marsupials and mouse, of whom child `0` has the marsupials:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Phew, that was a mouthful. Surely there must be a better way to get groups! There is, and we explore that next.\n",
    "\n",
    "(You'll note that we can extract our selections to a variable, which in this case we conveniently named `marsupials`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the common ancestor of a group of taxa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the same result as with the above method of inspecting the children visually, but this is of course much more reliable.\n",
    "\n",
    "The problem arises when the taxa that (we think) should form a group don't. For example, you'll remember that our two rodents don't seem to have any inclination to stick together. Let's see what happens if we get their common ancestor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Disaster... our rodents seem to be a little confused, and there isn't much we can do about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rooting or re-rooting a tree\n",
    "Let's see what the present root of the tree is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The root of the tree is the node that contains... well.. the whole tree. So we see no difference with printing the root versus printing the whole tree.\n",
    "\n",
    "⚠️ The fact that we can see 3 nodes (branches) at the highest (leftmost) level typically indicates that this tree is actually _unrooted_.\n",
    "\n",
    "We can define or redefine a tree's root to any node we wish. Here it would make evolutionary sense to root the tree at the marsupials, since they should be external to the placentals who evolved from marsupials quite a while back (somewhere between 66 and 90 Ma, if you care to know).\n",
    "\n",
    "Here's something important to remember: the `marsupial ancestor` variable that we defined above is not the three marsupial species, but instead their **ancestral node** and all its children (which in this case includes the three marsupial species in our tree). So, remember:\n",
    "\n",
    "⚠️ **almost everything in ETE is a _node_**. This means that when you print a tree or subtree, what you're actually printing is a node with all its children.\n",
    "\n",
    "Anyhow, setting the `marsupial_ancestor` as the outgroup makes evolutionary sense, so let's do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this is not a lesson on phylogeny, but that's a much better tree, with all carnivores together (we had them split before, some of them mixed with cows and pigs... never a good idea to mix carnivores with cows and pigs...).\n",
    "\n",
    "We still have that weirdness with mice refusing to relate to beavers, but there's nothing we can do about that without recomputing the alignment and/or tree with more data or different (better?) parameters.\n",
    "\n",
    "...actually, there is something we _could_ do about that weirdness with the mice... we could cut them out of the tree! Whether this is good science or not is another matter, but the point here is that we _can_ do it, and we'll learn how later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting a branch by name using `&`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we used Python's `type()` function to check what kind of object `redpanda` is. We see that it's a `TreeNode`, a kind of ete3 object used to contain nodes in a tree (either internal nodes or tip/leaf nodes).\n",
    "\n",
    "We can check if a node is a leaf very easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyhow. Once we have this `redpanda` node, we can do any manipulations we wish, such as exploring its children if it has any — which we can't do in this case because it's a leaf – or getting the node above it using `.up`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distances between nodes\n",
    "Wikipedia says the red panda is more closely related to the raccoon than to the giant panda (in spite of the names). Let's find out if that's true by measuring the distances between them.\n",
    "\n",
    "First, let's start by extracting a couple more branches (we already have the RedPanda one):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can easily measure the distances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, there is a much shorter evolutionary distance between red pandas and raccoons than between red pandas and giant pandas (at least according to this one gene). This combines well with the observation that the red panda and the raccoon have a much closer connection in the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naming internal nodes\n",
    "By default, typical NEWICK trees such as the one we've been using here contain names only for the leaf nodes, i.e. the taxa whose sequences we are using. But we can name internal nodes too. For example, let's give a name to that node containing all the marsupials:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we now print the tree again in ASCII form, asking it to show us the internal branch names, we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotating a tree\n",
    "A separate way to add information to a tree is with annotations. These can be anything you wish. For example, it is a well-known and universally accepted fact that arctic foxes, red pandas, raccoons, virginia opposums and river otters are some of the cutest animals on Earth, so let's create an attribute to show that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and that's it!\n",
    "\n",
    "We can now use that newly defined trait:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, we proabably want to do something more useful than just printing the names out, and we will later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also add some useful taxonomic information. For example, let's add the taxonomic Order.\n",
    "\n",
    "In a previous lesson we accessed the NCBI `Taxonomy` database, and this is what you would do here in a real analysis. This is a cheap version, so I'll just give the orders to you pre-baked in a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this to annotate our branches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And once again we can use this new attribute in our analyses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use this attribute for something more interesting later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a copy of a tree\n",
    "For normal variables you can just make a copy with `=`, but for ete3 trees, just like with lists, you need to use the `.copy()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the two are different objects, which we can confirm using Python's `id()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting (`detach`ing) leaves in a tree\n",
    "It's weird that mice ended up outside instead of being next to beavers. Let's take another look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's making our tree look ugly, so let's do a bit of great science (😉) and destroy the evidence by removing the mouse entirely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lovely. Now our tree looks prettier and we can publish it without fear of reviewers complaining... ... ... ...\n",
    "\n",
    "⚠️ **Important** ⚠️ I really do hope you're getting the sarcasm here, but just to be clear: you should **never** remove branches like we did above just because they show you something you don't want to see. In fact, this is the main case in which you _mustn't_ remove a branch – that is _not_ what `.detach()` is for. Instead, `.detach()` should be used only to remove uninformative nodes that don't change conclusions in the tree, and always with extreme care.\n",
    "\n",
    "If you have reasons to suspect that your alignment is wrong or your tree was poorly computed or you need more data to get an accurate tree, go ahead and do that; but you can't just remove a branch that you don't like! Well, you actually _can_, as we just saw, but that doesn't mean you _should_.\n",
    "\n",
    "Bottom line: use `detach()` only to remove uninformative branches or in analyses that require cutting out fragments for whatever other reason (never because they conflict with your preferred evolutionary model!)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pruning a tree\n",
    "A related operation to deleting (detaching) a leaf or internal node is _pruning_. This is done the opposite way, by specifying the names of the nodes that you want to keep.\n",
    "\n",
    "Let's say we want to keep only the the canids and the red panda (weird choice, I know, but this is just an example).\n",
    "\n",
    "We simply give all the names of the nodes we want to keep, in a `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether this is a good idea or not is another matter, but you can do it if you need to. Note that we set the `preserve_branch_length` attribute to `True`. Otherwise the branch lengths would have been altered. We can easily confirm that they weren't:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like with detaching nodes, it's a generally dangerous idea to prune a tree without extremely good reason. The tree you computed is a hypothesis about the relationships between the taxa you included, so, suddenly removing one from the final printed tree can be very deceiving (to yourself and your readers).\n",
    "\n",
    "Take home message: use pruning and detaching seldom and very carefully, only with good justification and, most importantly, making it very clear to your readers/reviewers/colleagues which nodes you removed and why."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving a tree to a Newick file\n",
    "Saving trees is easily done in ETE. The best thing is you have many choices of format ([adapted from their website](http://etetoolkit.org/docs/latest/tutorial/tutorial_trees.html#sec-newick-formats)):\n",
    "\n",
    "| Format | Description | Example\n",
    "|:--------------:|:--------------|:------------------:\n",
    "0 | flexible with support values | `((D:0.723274,F:0.567784)1.000000:0.067192,(B:0.279326,H:0.756049)1.000000:0.807788);`\n",
    "1 | flexible with internal node names | `((D:0.723274,F:0.567784)E:0.067192,(B:0.279326,H:0.756049)B:0.807788);`\n",
    "2 | all branches + leaf names + internal supports | `((D:0.723274,F:0.567784)1.000000:0.067192,(B:0.279326,H:0.756049)1.000000:0.807788);`\n",
    "3 | all branches + all names | `((D:0.723274,F:0.567784)E:0.067192,(B:0.279326,H:0.756049)B:0.807788);`\n",
    "4 | leaf branches + leaf names | `((D:0.723274,F:0.567784),(B:0.279326,H:0.756049));`\n",
    "5 | internal and leaf branches + leaf names | `((D:0.723274,F:0.567784):0.067192,(B:0.279326,H:0.756049):0.807788);`\n",
    "6 | internal branches + leaf names | `((D,F):0.067192,(B,H):0.807788);`\n",
    "7 | leaf branches + all names | `((D:0.723274,F:0.567784)E,(B:0.279326,H:0.756049)B);`\n",
    "8 | all names | `((D,F)E,(B,H)B);`\n",
    "9 | leaf names | `((D,F),(B,H));`\n",
    "100 | topology only | `((,),(,));`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the file we just made:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking for monophyly\n",
    "If you look at our original tree, you'll see that all the canids are together. That is, they are all within the same clade, and this ancestral node contains only canids and nothing else. In evolutionary jargon, this means the canids are \"monophyletic\". `ete3` allows us to test that directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a tuple with three elements:\n",
    "1. Whether the tree is or not monophyletic (`True` in this case).\n",
    "2. What kind of phylogenetic relationship the group has (`'monophyletic'`, `'paraphyletic'` or `'polyphyletic'`).\n",
    "3. A `set` with everything branch that was included in our selection but that isn't any of the branches we were looking for. Here that comes empty because only canids and nothing else is included between the four canids.\n",
    "\n",
    "The first two elements of the result are typically the most useful.\n",
    "\n",
    "We can also check for monophyly with a pre-defined list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that:\n",
    "1. The tree is _not_ monophyletic (`False`).\n",
    "2. It is actually `polyphyletic`.\n",
    "3. There's a bunch of stuff branching between and next to the two rodents.\n",
    "\n",
    "If you want to know more about what monophyletic, polyphletic and paraphyletic mean, check out [this Wikipedia article](https://en.wikipedia.org/wiki/Polyphyly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we created a `supercute` trait above and gave a to a few animals? Let's take a look at the tree with it printed out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to check whether all the `supercute` animals cluster together under the same clade,with nothing else inside it (i.e., they are monophyletic). If this were true, we would have support for the idea that supercuteness is a novel evolutionary trait inherited from a common ancestor of all the supercute animals (formally, a [synapomorphy](https://en.wikipedia.org/wiki/Synapomorphy_and_apomorphy)). If this is not true, we can deduce that supercuteness may have evolved independently by convergence (homoplasy):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, if you look at the tree, you'll notice that the supercute animals are spread all over. Supercuteness has therefore evolved convergently, this relation is therefore known as a **homoplasy** (i.e. they have pretty much nothing to do with each other, just like the winged flight of pterosaurs, insects, birds and bats)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing trees\n",
    "As you will have noticed by now, I was not happy with those unruly mice refusing to cluster with their fellow rodents, the beavers. So I went and recomputed the tree using IQ-Tree.<br/>\n",
    "(In case you care to know, I computed it this way: `! iqtree -s $aligned_file -nt 8 -B 1000`)\n",
    "\n",
    "Let's download it from my GitHub:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Remember that on Linux – or Linux-enabled Windows – you may have to do `wget` instead of `curl -O`. Otherwise just copy the link and download the file by hand)\n",
    "\n",
    "Let's load this tree file using `ete3`'s `Tree()` function, just as we did above with the FastTree one. We call it `iqtree` to distinguish it from the one we've been working with (the `fstree` that we created in FastTree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Beaver and mice are being more reasonable now.\n",
    "\n",
    "The tree must be rooted in order to be compared with the one using FastTree; so, just like we did with that one, let's root this one at the marsupials as well:<br>\n",
    "(⚠️Note: we cant use the `marsupial_ancestor` variable that we used above for the `fstree`, because that node belongs to the `fstree` object, not to this second `iqtree` object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tree looks much better overall. The sub-groups that should be together are all together. I'm less sure about the relationships between groups. The carnivores look pretty good overall, but humans next to the ungulates instead of to the rodents... less sure.\n",
    "\n",
    "In any case, it's a nice tree. But how does it compare to the other one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even explore which branches didn't match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, as you can see in #1 of the second comparison, the rodents (mouse+beaver) partition that we recovered in the IQ-Tree is not present in the FastTree version.\n",
    "\n",
    "⚠️ I'm talking about FastTree vs. IQ-Tree here, but in reality what we should be comparing are the **evolutionary models** used by these two softwares to compute the trees that we got. The focus here was merely in using ETE to compare trees, but if we were to publish this as a scientific analysis, the discussion should focus on the models, not merely the software (while certainly mentioning both)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating and exporting tree images with `ete3`\n",
    "As we mentioned above, FigTree is a very nice software for drawing and colouring trees. However, this software has not been updated since 2018, and I fear it may be abandoned. It doesn't seem to run on macOS BigSur, not easily anyway.\n",
    "\n",
    "Fear not, `ete3` can do that too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By all means, do change those values above at will, comment some of those instructions out, and see what happens.\n",
    "\n",
    "Note that we used `'%%inline'` as the \"name\" of the file. This tells Jupyter to put the image directly here in the notebook, but we can create files too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting a tree image to a file\n",
    "All we have to do to export the tree to an image instead of to the Jupyter screen is change the `'%%inline'` to a file name, whose extension (and type) can be `.pdf`, `.svg`, `.tiff`, `.jpg`, and more. ETE will judge from the extension which format it should produce.\n",
    "\n",
    "Let's create a PDF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go on take a look at the PDF. You may find that you need to change the style and width a bit, but this is a vector file anyway, so you can resize it at will in, say, Inkscape. And on that note:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note on exporting to Inkscape\n",
    "If your purpose is to do some aesthetic editing in Inkscape, I find that exporting to `PDF` and then importing that PDF into Inkscape works better than exporting to `SVG` (which is formally Inkscape's format)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Colouring branches to match taxonomy\n",
    "Recall that we have the taxonomic order of each mammal stored in the `.order` attribute of each leaf:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use this order to colour the branches of our tree. Here are my selected colours, which I took from a [list of named web colours](https://en.wikipedia.org/wiki/Web_colors#Extended_colors):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for the carnivores I actually entered a hexadecimal triplet. I just wanted to show you that you can if you wish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A function to apply colour to the branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A style to draw the trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This style can be reused to draw another tree (for example, the IQ-Tree one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Draw the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(I don't like that colour combination either. Feel free to change it if you're feeling artistic)\n",
    "\n",
    "It's now very easy to see that the rodents are behaving strangely, that artiodactyls, carnivores and marsupials are all monophyletic, and that the marsupials are our chosen outgroup.\n",
    "\n",
    "If we now wanted to render the IQ-Tree tree in the same way, we would only need to run the last two lines. However, this would fail because we never added the **order** to the leaves of that tree – we only did it to the FastTree tree (check out code cell #30 and #31 or thereabouts).\n",
    "\n",
    "**Homework:** add the **order** to the branches of the IQ-Tree tree and and plot it as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hint: each of those should only take two lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced ETE tree styling\n",
    "There are a lot more options to get super fancy trees using `TreeStyle()`... and by \"a lot\" I mean _a lot_! Take a look at this gorgeous example that I copied almost verbatim from the `ETE` website:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}