Support volatile state migration / persistence (e.g: for VMs)

[daprilik]

I didn't see this being discussed anywhere, so I thought I'd start a thread to make sure there's visibility here...

---

The ability to save/restore volatile TPM state will be needed to support Virtual TPM (vTPM) scenarios, as unlike physical TPMs in physical machine, vTPMs in VMs need to support things such as snapshots (live save/restore), and live migration.

This is not a novel feature, and is something that's already supported by various open-source (e.g: swtpm) and proprietary (e.g: Hyper-V) C-based TPM implementations.

tpm-rs saved-state state should also be properly versioned, so as to support live-migration/updates from older to newer TPM revisions (which is one thing no other TPM implementation currently supports, as far as I'm aware).

---

This discussion might be a good place to start bike-shedding on specifics, such as:

• choice of protocol (bincode? protobuf? proprietary? json?)
• which Rust libraries to use (serde? something else?)
• how to architect saved-state support though the tpm-rs codebase
• how to detect saved-state regressions

[jettr]

This is an interesting problem. I am glad you brought it up! I am guessing that state here refers to volatile memory contents that would normally be stored in RAM, right?

Can a snapshot be taken in the middle of processing a request or can a snapshot only be taken while the TPM is idle? If it can be taken in the middle of processing, should that processing continue when the snapshot is restored?

how to architect saved-state support though the tpm-rs codebase

I like the idea of introducing a trait that allows access to structured state, maybe something like

pub enum ServerState {
    Idle,
    Processing,
    Running,
}


pub trait RuntimeState {
    fn get_server_state(&self) -> ServerState;
    fn set_server_state(&mut self, state: ServerState);
}

But the serialization and deserialization of state would be handled by something external. The type that implements RuntimeState could also implement some kind of serialization. Embedded implementation could implement RuntimeState with a simple struct and wouldn't have to worry about serialization. Virtual TPM implementations should implement RuntimeState with something more complex that also supported serializing/de-serializing and also different serialization version schemes.

What are your thoughts?

[daprilik]

I am guessing that state here refers to volatile memory contents that would normally be stored in RAM, right?

Yep, exactly.

Moreover - I would expect this saved state to be a superset of what is persisted to non-volatile storage, such that it's possible to regenerate an instance of the TPM without also having to do a (potentially expensive) read from non-volatile storage on the restore path.

Can a snapshot be taken in the middle of processing a request or can a snapshot only be taken while the TPM is idle?

Barring any strong objections from other folks, I think it's probably best to keep things simple, and assume that snapshots can only happen after any pending TPM operations have been resolved, and the TPM is back to idle. This is, of course, predicated on my working understanding that TPM operations are all relatively "quick" to complete, and that waiting for a request to resolve wouldn't noticeably affect snapshotting latency.

---

I'm not sure I have much input on the specific traits / structure of the inner plumbing here (at least at this stage in the project). I'm sure folks who are "in the trenches" here could come up with something far better than any sketch I write out here :)

---

But the serialization and deserialization of state would be handled by something external

Embedded implementation could implement RuntimeState with a simple struct and wouldn't have to worry about serialization. Virtual TPM implementations should implement RuntimeState with something more complex that also supported serializing/de-serializing and also different serialization version schemes.

Maybe I'm not totally following here, but is the suggestion here to expose an entirely-pub Rust structure as part of the TPM engine's save/restore API, and have individual implementations provide their own serialization / versioning schemes for it?

If so, I would strongly disagree with that approach.

This would mean consumer of the TPM library will also need to be familiar with the nitty-gritty internals of the TPM library, rather than just programming against the top-level, (presumably) SemVer stable API. While this would be a small improvement over how things are down today (i.e: the TPM core would be responsible for maintaining all the "plumbing" required to get-at this internal state), it wouldn't actually make it any easier for consumers to migrate between versions (without also becoming experts at navigating the TPM implementation internals themselves).

I would much rather see the TPM code to expose a top-level interface that looks something like this:

#[cfg(feature = "snapshot")] // machinery can be compiled out for embedded targets
impl TpmEngine {
    fn snapshot(&self, buf: &mut [u8]); // or something more "clever" than &mut [u8], e.g: `impl Writer` 
    fn restore_from_snapshot(&mut self, buf: &[u8]) -> Result<(), RestoreError>;
        // where RestoreError includes variants for corrupted data, version mismatch, etc...
}

That is to say: the snapshot is emitted as an opaque binary blob, which the consumer doesn't have to de/serialize themselves, with all details related to versioning, migration, etc... handled within tpm-rs itself.

Moreover, doing it this way also means that there can be in-tree migration testing, ensuring that it's always possible to jump between older and newer TPM revisions.

[jettr]

I think I have a better understand of what you are talking about now.

I have a few more questions about this:

• What would actually be calling snapshot and restore_from_snapshot? Is this a co-operative operation to take a snapshot from a vTPM?
• Is there really a use case for upgrading the TPM version without first shutting down the TPM and starting it back up again?
  • This requirement seems like it would cause a large testing permutation increase with many older versions of state and newer versions of implementations. This also might place large constraints on how we internally use state that might prevent certain refactors in the future.

Also in this save and restore scheme, we need to ensure that sensitive data (private key material etc) is not available in the serialized state.

But the serialization and deserialization of state would be handled by something external

I definitely see the value in something withing the tpm-rs codebase providing this serialization and migration. When I said external, I was thinking external to the main TpmEngine itself, but not necessarily the tpm-rs code base. Sorry for the confusion. The pub trait RuntimeState could have different implementation within the tpm-rs code based on the cfg(feature = ...) provided -- I think that would be a good way to go.

[daprilik]

What would actually be calling snapshot and restore_from_snapshot? Is this a co-operative operation to take a snapshot from a vTPM?

Yeah, it'd be a co-operative.

e.g: in a VM context, you might imagine a TPM emulated device with its own impl Snapshot API of some kind, whereupon it would save any of its own state, and then call-out to the TPM engine for its state as well.

Is there really a use case for upgrading the TPM version without first shutting down the TPM and starting it back up again?

Yes and no.

No, in the sense that there'd never be a case where we'd want to "magically" upgrade the user facing TPM revision out from under the user without having the TPM shutdown first.

Yes, in the sense that there are many cases where we'd want to "magically" upgrade the underlying TPM library out from under the user, without having the TPM shutdown first.

i.e: a common flow in the VM world is to perform "host updates" by first snapshotting any running VMs, updating the host, and then restoring VMs on-top of the host's newly updated virtualization stack.

i.e: in the case of VM migration between nodes (as might happen in a datacenter), different machines might be running slightly different versions of the virtualization stack. ideally, a VM spun up on an older node should be able to migrate over to a new node without having to be shut down first (so as to avoid any user-facing downtime)

Not having a way to migrate volatile saved-state between TPM versions would make these scenarios (and more) essentially impossible, outside of shipping N different copies of the tpm-rs code for each different TPM revision that needs to be supported.

This requirement seems like it would cause a large testing permutation increase with many older versions of state and newer versions of implementations.

I suspect it'd be reasonable to say that given TPM versions A, B, and C, going from A to C first requires migrating through B. Therefore, the permutations that would need to be tested / implemented would only be N, where N is the number of TPM releases.

This also might place large constraints on how we internally use state that might prevent certain refactors in the future.

The key thing would be to keep runtime data representations separate from the volatile saved state code, and have a transformation function between the two.

That way, you can tweak the true runtime internals however you want, so long as the corresponding transformation code between the runtime state and the "snapshot" state stays up-to-date.

(note that this is not a "novel" problem, and that this sort of snapshot saved-state / migration handling is present in just about every virtualization stack, in order to support migrating other kinds of virtual devices. vTPM would not be unique here.)

Also in this save and restore scheme, we need to ensure that sensitive data (private key material etc) is not available in the serialized state.

I'm not sure I agree with this. The security boundary I'm imagining is one where if you wanted to isolate the vTPM from the rest of a system, you wouldn't just isolate the tpm-rs code itself. Rather, you'd isolate a custom component that wraps the tpm-rs code in project-specific functionality, and said wrapper would be responsible for, say, encrypting/decrypting the resulting saved-state blob.

[daprilik]

By the way, there's some interesting discussion around vtpm migration over on the swtpm repo:

stefanberger/swtpm#710

[bradlitterell]

Resolution: Will include these requirements as optional functionality in the TPM Storage abstraction/module.

[daprilik]

Can you elaborate on this? What exactly does "optional" entail here?

[bradlitterell]

We didn't go into details, only acknowledged that we we want the interface/trait to the Storage crate/module to support something like this, and that a particular storage module might choose to only support shutdown/restart and not to support any runtime state other than the existing TPM state transitions - that's the optional part.