Merge pull request #2 from theabro/aws-cli

Additional scripts for making VPC and instances

README.md

@@ -1,18 +1,23 @@
 # aws-ukca
 
-Ansible playbook for running UKCA on an AWS EC2 instance. This is based on the Vagrant-based [Met Office Virtual Machine](https://github.com/metomi/metomi-vms) and previous work on using Ansible for this VM on the [JASMIN Unmanaged Cloud](https://github.com/theabro/ukca-playbook). 
+Ansible playbooks and AWS scripts for creating UKCA training VMs on an AWS EC2 instance. This is based on the Vagrant-based [Met Office Virtual Machine](https://github.com/metomi/metomi-vms) and previous work on using Ansible for this VM on the [JASMIN Unmanaged Cloud](https://github.com/theabro/ukca-playbook). 
 
-Based on instructions from [Dave Sugden (April 2020)](https://davelms.medium.com/use-ansible-to-create-and-configure-ec2-instances-on-aws-cfbb0ed019bf) and [Vivek Gite (February 2018)](https://www.cyberciti.biz/faq/how-to-create-aws-ec2-key-using-ansible/), as well as general Googling.
+The UKCA Tutorials, including information on how to connect to the VMs, can be found here:
+* https://www.ukca.ac.uk/wiki/index.php/UKCA_Chemistry_and_Aerosol_Tutorials_at_vn11.8 
 
-Tested on macOS 10.13.6. On macOS you may first need to
+Based on instructions from [Dave Sugden (April 2020)](https://davelms.medium.com/use-ansible-to-create-and-configure-ec2-instances-on-aws-cfbb0ed019bf) and [Vivek Gite (February 2018)](https://www.cyberciti.biz/faq/how-to-create-aws-ec2-key-using-ansible/), as well as general Googling. Many thanks to Courtney Waugh of AWS for help and advice with the AWS scripts.
 
-	export PATH="/Users/[YOUR USERNAME]/Library/Python/2.7/bin:$PATH"
+Tested on macOS 12.3.1. On macOS you may first need to
 
-This is a 3 stage process:
+	export PATH="/Users/[YOUR USERNAME]/Library/Python/3.8/bin:$PATH"
+
+This is a 5 stage process:
 
 1. Create a user on the AWS Console
 2. Create the EC2 instance.
-3. Provision the EC2 instance with the required packages and settings to run FCM, Rose, & Cylc and prepare it for UM-UKCA use. 
+3. Provision the EC2 instance with the required packages and settings to run FCM, Rose, & Cylc and prepare it for UM-UKCA use before saving it as an Amazon Machine Image (AMI)
+4. Use CloudFormation ton create a virtual private cloud (VPC) to host the training instances
+5. Use the AWS-CLI to create as many instances are required within the VPC, along with key files to connect to each instance
 
 ### Chose your region
 
@@ -108,6 +113,8 @@ If you stop the instance and then later restart it, the IP address may change. Y
 
 ### UM Install Commands
 
+Once your VM is provisioned you will need to stop it via the AWS web console and change its instance type to at least a **t2.large** to provide enough vCPUs and memory to run UKCA. Large and more powerful instance types are also available.
+
 **Note** that prior to UMvn11.1 the UM install won't work due to the `gfortran` compiler version used at Ubuntu 18.04. Post vn11.2 the setting 
 
 	grib_library: libgrib-api-dev 
@@ -121,9 +128,9 @@ must be made in `group_vars/all.yml` (this is the default). The current roles wi
     fcm checkout fcm:[email protected] UM11.8
     cd UM11.8
     rose stem --group=install,install_source -S CENTRAL_INSTALL=true -S UKCA=true
-    rose stem --group=kgo,ukca -S GENERATE_KGO=true
     rose stem --group=fcm_make --name=vn11.8_prebuilds -S MAKE_PREBUILDS=true
     rose stem -O offline --group=fcm_make --name=vn11.8_offline_prebuilds -S MAKE_PREBUILDS=true
+    rose stem --group=kgo,ukca -S GENERATE_KGO=true
 
 After the `um-setup` command you will need to close and re-open a terminal.
 
@@ -132,3 +139,66 @@ Availble suites for the VM can be found on the [UKCA Website](https://www.ukca.a
 To remove the existing MOSRS information, you need to delete the following file:
 
     rm .subversion/auth/svn.simple/2be6a67d04b1c8c6d879daafa52fd762
+
+## Turn this instance into an Amazon Machine Image
+
+Once you have finished all the steps above and the UM is installed and you have removed the MOSRS information, power the EC2 instance down using the AWS web console. Navigate to the **instance summary** and use the **instance state** drop-down menu to select **stop instance**.
+
+Once it is in a **stopped** state go to the **Actions** drop-down menu at the top right and select **images and templates** and then **create image**. 
+
+You will need to give it a short-ish descriptive name and a longer description. You can change the volume size if desired, and you may want to ensure that the *delete on termination* option is selected for the volume. This ensures that the volume is deleted when the instance is terminated. If this is not selected you will need to delete the volumes separately.
+
+When you are happy with the options, click the **create image** button on the bottom right. It will a short amount of time to create the image, and when complete the image can be found in the **Images** section of the left-hand naviagation menu, in the **AMIs** section. 
+
+You will need to make a note of the **AMI ID** assigned to your image, as you will need it in the next section.
+
+## Additional AWS scripts used to provision the EC2 instances for the students
+
+Once you have install the UM and the necessary files, you can then use the scripts contained within the [`src/`](src/) directory to create a virtual private cloud (VPC) in your AWS account to host the VMs, create a security group for them, and then create keys and instances to give to the students.
+
+### Using CloudFormation to create a VPC to host the training EC2 instances
+
+To create the network the EC2 instances will reside in, the file [UKCACloudFormationTemplate.json](src/UKCACloudFormationTemplate.json) should be used within [CloudFormation](https://aws.amazon.com/cloudformation/) to create the virtual private cloud, security group, and public subnets used by the [create-ec2.sh](src/create-ec2.sh) script. These settings have been adapted from the [VPC_With_Managed_NAT_And_Private_Subnet.yaml](https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/VPC/VPC_With_Managed_NAT_And_Private_Subnet.yaml) and then converted to `.json` using the CloudFormation designer.
+
+![Diagram showing the network layout within CloudFormation from the UKCACloudFormationTemplate.json](media/template1-designer.png?raw=true "Network layout within CloudFormation")
+
+### Using AWS-CLI to create keys and EC2 instances
+
+You will need the **AMI ID** from the image you created above and enter this in the `ami=` line in the [create-ec2.sh](src/create-ec2.sh) script.
+
+Whether keys and instances are created or not will depend on whether or not a `keys/ukca_key_trXX.pem` (for keys) and a `keys/ukca_vm_trXX.json` (for instances) exists. If both files exist for e.g. `XX=01` and 2 instances are requested to be created, in fact only the second one will be provision as the script assumes that the first already exists. If the `.pem` file exists and the `.json` file does not then the key will **not** be created but the instance will be (and vice versa). 
+
+It is therefore important to remember to delete any `ukca_key_trXX.pem` and `ukca_vm_trXX.json` files in the `keys/` directory, terminate any previous instances, and remove the corresponding keys from the AWS web console before creating new keys and instances, as they will not be created unless the corresponding files have been removed.
+
+**Note** here **NOT** to delete the `ukca_keypair.pem` file that is the key for the instance provisioned using the Ansible method described above.
+
+When CloudFormation has created the VPC, copy the relevant information from the console, e.g.
+
+    vpc='vpc-0ef173dcfd55cdc70'
+    sgid='sg-02ea177be98d549ff'
+    subnet0='subnet-0974c239cea07ef9e'
+    subnet1='subnet-0cdb50524da13885f'
+
+into the [create-ec2.sh](src/create-ec2.sh) script then `cd` into the `src/` directory and run
+
+    bash create-ec2.sh N 
+
+where `N` is the number of instances you want to create. This will then create corresponding files, `ukca_tr_XX.pem` and `ukca_vm_trXX.json`, in the `keys/` directory. You should specify both subnets and the script will distribute the instances between both of them. You may want to zip the key files up prior to distributing them, e.g.
+
+    zip ukca_key_tr01.zip ukca_key_tr01.pem
+
+However, the information in the .json files will not include the public IP address information at this time as these won't have been assigned. Once the EC2 instances have all started and are running, you can use the script [get-ec2-ip.sh](src/get-ec2-ip.sh) to return the IP address information, e.g.
+
+    bash get-ec2-ip.sh | sort
+    ukca_vm_tr01,ukca_key_tr01,3.8.192.184
+    ukca_vm_tr02,ukca_key_tr02,18.168.205.105
+    ukca_vm_tr03,ukca_key_tr03,35.177.89.5
+    ukca_vm_tr04,ukca_key_tr04,52.56.250.236
+    ukca_vm_tr05,ukca_key_tr05,3.8.139.243
+    ukca_vm_tr06,ukca_key_tr06,13.40.48.123
+
+This can then be copied and pasted into an Excel spreadsheet using the [Convert Text to Columns Wizard](https://support.microsoft.com/en-us/office/split-text-into-different-columns-with-the-convert-text-to-columns-wizard-30b14928-5550-41f5-97ca-7a3e9c363ed7).
+
+### EC2 Instance Connect
+
+The file [EC2InstanceConnectTraining.json](src/EC2InstanceConnectTraining.json) should be used as a template for a policy to allow demonstrators to connect to the training EC2 instances using [EC2 instance connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-using-EC2-Instance-Connect.html) when combined with the AmazonEC2ReadOnlyAccess policy. This means that they can see all instances but can only connect to the training ones (due to the tag Event:Training) under the ubuntu username. Here `AWSACCOUNTID` should be replaced with your AWS account ID without dashes.

roles/configure-os/tasks/main.yml

@@ -22,6 +22,14 @@
   apt:
     name: git
 
+#### Install additional packages, inc. ec2-instance-connect
+- name: Install packages
+  apt:
+    pkg:
+    - ec2-instance-connect
+    - gnuplot
+    - python3-pip
+
 # Download metomivms data from github
 - name: download metomi-vm-files from github
   git:

roles/download-xconv/tasks/main.yml

@@ -26,7 +26,7 @@
 
 - name: Download & install Xconv to UMDIR/bin
   unarchive:
-    src: http://cms.ncas.ac.uk/documents/xconv/_downloads/xconv1.94_linux_x86_64.tar.gz
+    src: https://gws-access.jasmin.ac.uk/public/ukca/xconv1.94_linux_x86_64.tar.gz
     dest: /home/{{ vagrant }}/umdir/bin
     remote_src: yes
   become: yes

src/EC2InstanceConnectTraining.json

@@ -0,0 +1,21 @@
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": "ec2-instance-connect:SendSSHPublicKey",
+            "Resource": "arn:aws:ec2:eu-west-2:AWSACCOUNTID:instance/*",
+            "Condition": {
+                "StringEquals": {
+                    "ec2:osuser": "ubuntu",
+                    "ec2:ResourceTag/Event": "Training"
+                }
+            }
+        },
+        {
+            "Effect": "Allow",
+            "Action": "ec2:DescribeInstances",
+            "Resource": "*"
+        }
+    ]
+}