These notes are a rough guide to getting xen set up with vTPM support and having keylime work with it.
clone the source tree and run the installer:
git clone keylime
./installer.sh
You must obtain the public EK and the EK certificate from outside of Xen when the kernel still has hardware tpm drivers take ownership first, then obtain pubek, and ekcert as follows
takeown -pwdo <owner_password>
getpubek -pwdo <owner-password>
nv_readvalue -pwdo <owner-password> -in 1000f000 -cert -of tpm_ekcert.der
the files pubek.pem and tpm_ekcert.der will pop out. You'll need these inside of the vtpmmgr domain
starting with fresh install of ubuntu 16.04 minimal server
setup proxy if needed setup passwordless sudo
update apt and install latest packages
install xen first https://help.ubuntu.com/community/Xen
sudo apt-get install xen-hypervisor-amd64
reboot
NAT for VM
create a bridge xenbr0
put in /etc/network/interfaces
auto xenbr0
iface xenbr0 inet static
address 10.0.0.1
netmask 255.255.255.0
bridge_ports eth22
we dont' want to bridge to internet, we'll nat instead. so use a non-existent interface for bridge_ports. it will bring up an empty bridge
add this to /etc/rc.local to turn on NAT
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eno1 -j MASQUERADE
now we've got to get tpm support out of the dom0 kernel
confirm that it is there by looking for /dev/tpm0 and initialization in dmesg
dmesg | grep -i tpm
we want this gone!
https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
mkdir ubuntu-kernel
cd ubuntu-kernel
apt-get source linux-image-$(uname -r)
sudo apt-get build-dep linux-image-$(uname -r)
cd linux-4.4.0
chmod a+x debian/rules
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*
fakeroot debian/rules clean
fakeroot debian/rules editconfigs
Edit the amd64 architecture when you do configs. Leave the others alone.
Turn off IMA: secure options->Integrity Subsystem->IMA
Turn off all TPM support: device-drivers->character devices->TPM Hardware support
It will complain about various things related to config-checks. these are for other arch's. just ignore
Now you need to add a local version extension to the name so that we know which one it is and can boot it
change the first line of debian.master/changelog to add -notpm like
linux (4.4.0-112.135+notpm) xenial; urgency=low
* linux: 4.4.0-112.135 -proposed tracker (LP: #1744244)
* CVE-2017-5715 // CVE-2017-5753
We need to remove mention of the tpm modules and symbols from the abi to get it not to complain. this is a bit of a hack.
go to debian.master/abi/4.4.0-111.134/amd64 and get rid of anything that says "tpm from generic or generic.modules
grep -v tpm generic > tmp && mv tmp generic
grep -v tpm generic.modules > tmp && tmp generic.modules
now it is time to build. Go back to linux 4.4.0 dir and kick it off. Took about 20 mins on my rather old machine.
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic binary-perarch
now in the parent directory ubuntu-kernel there should be a pkg for the new kernel
sudo dpkg -i ../linux-image-(VERSION).deb
NOTE that this doesn't seem to work since Meltdown and Spectre got patched. Try building the 16.04 kernel the ubuntu way above.
https://wiki.ubuntu.com/KernelTeam/GitKernelBuild
sudo apt-get install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
cp /boot/config-`uname -r` .config
yes '' | make oldconfig
make menuconfig
Turn off IMA: secure options->Integrity Subsystem->IMA
Turn off all TPM support: device-drivers->character devices->TPM Hardware support
make clean
make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-notpm
sudo dpkg -i ../linux-image-(VERSION).deb
if you used the ubuntu kernel build method (as opposed to git) and haven't updated your kernel recently, it should boot up in that kernel (as the latest) automatically.
set default kernel to the one we just built:
https://unix.stackexchange.com/questions/198003/set-default-kernel-in-grub
i used the third answer. linked: http://www.humans-enabled.com/2014/08/how-to-set-default-grub-kernel-boot.html
now reboot
Confirm that the correct kernel with (notpm) in the name got booted in Xen.
uname -r
and that we're still in xen:
sudo xl list
now we've got to build the vtpm and vtpmmgr subdomain in xen.
following pre-reqs here: https://wiki.xenproject.org/wiki/Compiling_Xen_From_Source
sudo -E apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
sudo -E apt-get install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial
sudo -E apt-get install make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch
sudo -E apt-get install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev
sudo -E apt-get install gettext libpixman-1-dev libaio-dev markdown pandoc
sudo -E apt-get install cmake
check out the source git clone git://xenbits.xen.org/xen.git
I had various build problems with stubdoms, xen version 4.8 seemed to work. Not a lot has changed in vtpm land in xen in a while, so mis-matching versions is ok.
cd xen
git checkout stable-4.8
# just need patched stubdoms, nothing else
./configure --enable-stubdom --disable-xen --disable-tools --disable-docs --disable-pv-grub --disable-xenstore-stubdom --disable-ioemu-stubdom
need to patch up stubdom/vtpmmgr/mgmt_authority.c
cd xen
wget https://raw.githubusercontent.com/mit-ll/keylime/master/patches/xen-vtpmmgr-patch.txt
patch -p1 < ../xen-vtpmmgr-patch.txt
make build-stubdom
this build may fail on something related to pv-grub, but it's usually ok
the unikernel elf files for vtpm and vtpmmgr are in xen/stubdom/mini-os-x86_64-vtpm and xen/stubdom/mini-os-x86_64-vtpmmgr
called mini-os.gz get those two gz files and rename them to something like vtpm.gz and vtpmmgr.gz
Build a vtpmmgr domain
see: https://xenbits.xen.org/docs/4.7-testing/misc/vtpmmgr.txt
also see http://xenbits.xen.org/docs/unstable/misc/vtpm-platforms.txt for example configs
mkdir -p /var/lib/xen/images/
cp the two gz's there
mkdir -p /var/lib/xen/disks
dd if=/dev/zero of=/var/lib/xen/disks/vtpmmgr.img bs=2M count=1
create the config file in /etc/xen/vtpmmgr.cfg
kernel="/var/lib/xen/images/vtpmmgr.gz"
memory=8
disk=["file:/var/lib/xen/disks/vtpmmgr.img,hda,w"]
name="vtpmmgr"
iomem=["fed40,5"]
extra="tpmlocality=0"
now start it up.
sudo xl create -c vtpmmgr.cfg
you should see "INFO[VTPM]: Waiting for commands from vTPM's:" if all went well
now it's time to build a vtpm domain. Build an empty disk image for it to put its stuff. You'll also need to give it a random UUID. we will change this later when configuring keylime
The config file will look like:
kernel="/var/lib/xen/images/vtpm.gz"
memory=8
disk=["file:/var/lib/xen/disks/vtpm0.img,hda,w"]
name="vtpm0"
extra="loglevel=debug"
vtpm=["backend=vtpmmgr,uuid=C7319AEA-F895-436C-A110-556C2DDC1EE2"]
start up with:
xl create -c vtpm0.cfg
it will say something like "tpm_startup.c:43: Info: TPM_Startup(1)" when it is started up.
You'll also see some log messages in the vtpmmgr showing that it talked to it: like TPM_GetRandom
now we need to create a linux machine to hook our first vtpm to.
http://www.virtuatopia.com/index.php/Building_a_Xen_Guest_Domain_using_Xen-Tools
sudo apt-get install xen-tools
we'll use xen-create-image.
cd /etc/xen/
xen-create-image --hostname xenial-linux-vtpmmgr --dist=xenial --ip 10.0.0.3 --netmask 255.255.255.0 --gateway=10.0.0.1 --nameserver="155.34.3.8" --dir /var/lib/xen/ --password=passwd
change whatever needs changing for this as needed for your environment. Note root password.
a cfg file will come out. we need to configure the vtpm we just made to talk to it. Note the name of the vtpm we just made. put it into the config at the bottom:
vtpm=["backend=vtpm0"]
now start it up
xl create -c xenial-linux-vtpmmgr.cfg
once up you can login, confirm that network is working, setup ssh/keys and then login via ssh. it's much better than the xen console.
bah, now we've got to patch up a kernel to talk to the vtpm in 16.04 https://patchwork.kernel.org/patch/9485637/
using the instructions above, but without the need to change the ABI or tpm config: Apply this patch, update the changelog with +vtpmpatch, and build the kernel.
You'll need to install both the linux-image deb and the linux-image-extras deb to get the kernel modules for the TPM driver.
You need to install the crda package before installing these two debs.
apt-get update
apt-get install crda
dpkg -i linux-image-*
reboot and confirm that you have the file /dev/tpm0 now.
Note that 14.04 (trusty) predates this bug. So, if you don't want to build a new kernel, you can try 14.04.
if the machine booted up and you didn't get the dreaded timeouts bug above, you should be ready to test the vtpm. Download and install keylime:
git clone keylime
./installer.sh
check that you can talk to the vtpm with:
getcapability -cap 1a
Result for capability 0x1a is :
major : 0x01
minor : 0x02
revMajor : 0x00
revMinor : 0x07
specLevel : 0x0002
errataRev : 0x01
VendorID : 45 54 48 5A
VendorID : ETHZ
[not displaying vendor specific information]
Note Vendor ID: ETHZ that means you're talking to the vtpm
now clone the vtpmmgr vm so that we have another one from which to test. I called this one xenial-linux-keylime
i just copied the config file in /etc/xen and the .img file dir in /var/lib/xen/domains/ update config to have new name, and redirect the swap and disk locations to the place you copied
Make another vtpm (follow procedure above including makeing a fresh disk image) and then hook it to the new machine.
we're going to run all of the keylime services on dom0 of the xen machine. Only client and cloud agent processes will run on the vms.
So, on dom0 of the xen box start up the following:
keylime_verifier
keylime_provider_registrar
keylime_registrar
Now on both vms: find and replace all 127.0.0.1 in /etc/keylime.conf with 10.0.0.1 this will point them to dom0 for all the services you just started
also make sure that vtpm_policy is somethign reasonable:
vtpm_policy = {"23":["ffffffffffffffffffffffffffffffffffffffff","0000000000000000000000000000000000000000"],"15":"0000000000000000000000000000000000000000"}
on the vtpmmgr domain (which must should have been brought up first before the vtpm attached to the clone xenial-linux-keylime.
first need to do platform init with the keys from dom0 from way back at the beginning of this guide. scp them over and then run:
keylime_provider_platform_init pubek.pem tpm_ekcert.der
you should see something like "keylime.provider_platform_init - INFO - Activated VTPM group"
Now you need to add a vtpm
keylime_provider_vtpm_add current_group.tpm
Note that current_group.tpm is a symlink to the latest initialized group that comes from platform_init.
log message like the following means that it worked: "keylime.platform-init - INFO - Registered new vTPM with UUID: 33C6AD2C-F20D-40B0-B3F6-BC0FB0627637
Copy this uuid. Shutdown the keylime vm and its vtpm. Put this UUID into the cfg file for the vtpm:
vtpm=["backend=vtpmmgr,uuid=33c6ad2c-f20d-40b0-b3f6-bc0fb0627637"]
Boot the vtpm back up. then boot up the linux machine. Now you must put this UUID into the keylime vm /etc/keylime.conf.
agent_uuid=33c6ad2c-f20d-40b0-b3f6-bc0fb0627637
Ok, now it is time to start up the agent process. (this is where it all comes together, so if this works, the rest likely will too)
watch for any 400 errors from registration.
if you see the following:
keylime.cloudagent - INFO - Starting Cloud Agent on port 9002 use to stop
then it's good.
Now lets actualy run through the entire process:
back on dom0 xen machine we want to run the tenant:
keylime_tenant -u 33c6ad2c-f20d-40b0-b3f6-bc0fb0627637 -t 10.0.0.4 --verify -f SOMEFILE.txt
where SOMEFILE.txt is any old file that you want delivered securely to the agent. The --verify option will poll the agent to make sure everything worked
if all goes well you should see: keylime.tenant - INFO - Key derivation successful
that's it folks!