Skip to content

Guest Action Workflows

Jump to bottom
Manuel Mendez edited this page Jun 2, 2015 · 4 revisions

Introduction

This outlines three guest-related workflows: creation, deletion, and reboot. Reboot was chosen as it is representative of several other actions manipulating an existing guest - power-off, etc.

Tech stack

The job queue uses beanstalkd rather than trying to implement a queue on top of etcd. For now, a a single beanstalkd server is used for lochness. It is discovered/configured via etcd and ansible as other services. Later, "resiliency" can be added. The queue is used for ordering and easier notification for the worker daemons, though listing and looping over the jobs prefix in etcd would be possible.

Also, two additional daemons:

  • cplacerd - the "guest placement daemon". A single instance - protected by a cluster wide lock - is ran. It only selects hypervisors for new guests (and the metadata shuffling involved). It does not communicate with mistfy-agent hypervisors.
  • cworkerd - the "worker daemon". Multiple instances may be ran. This handles the actual communication with mistify-agent hypervisors and updating of metadata.

Tasks in the beanstalks are just IDs for jobs. Jobs are persisted to etcd and remain after completion. Jobs have a TTL of 24 hours.

cguestd is the "guest API" (not the agent), an HTTP API that end users interact with and is the "official" way to view jobs. The command line tool guest can be used for more convenient interaction with cguestd.

Workflows

Guest Creation

  1. End user makes request to cguestd to create guest with given configuration, directly or via guest
  2. cguestd processes request
    1. Basic validation of guest data
    2. Creates guest data stub in etcd, without hypervisor or ip specified
    3. Creates a job referencing the guest with an action for hypervisor selection
    4. Adds a task to the create beanstalk tube
    5. Returns guest object JSON and job ID to user. The rest of the process is asynchronous and the end user can request job status by ID via guest / cguestd
  3. cplacerd processes job
    1. Reserves task from the create tube
    2. Basic validation on job and guest
    3. Selects hypervisor for guest based on criteria
    4. Saves selected hypervisor in guest data
    5. Updates job with an action for creation
    6. Adds a task to the work beanstalk tube
  4. cworkerd begins processing job
    1. Reserves the task from the work tube
    2. Makes request to the hypervisor's mistify-agent to create guest
    3. Updates job with the remote job id (mistify-agent has a local job queue), start time, and working status
    4. Release/Requeue task in work beanstalk tube

The mistify-agent works on actually creating the guest.

  1. cworkerd monitors job
    1. Reserves task from worker tube
    2. Makes request to hypervisor's mistify-agent to check job status
    3. If incomplete:
      1. Requeue job into work tube (after brief delay) to check again later
    4. If complete/errored:
      1. Update any necessary metadata
      2. Update job with finish time and final status (and error if applicable)
      3. Delete task from beanstalk

Guest DELETE

  1. End user makes request to cguestd to delete guest, directly or via guest
  2. cguestd processes request
    1. Creates a job referencing the guest with an action for guest deletion
    2. Adds a task to the work beanstalk tube
    3. Returns guest object JSON and job ID to user. The rest of the process is asynchronous and the end user can request job status by ID via guest / cguestd.
  3. cworkerd begins processing job
    1. Reserves the task from the work tube
    2. Makes request to the hypervisor's mistify-agent to delete guest
    3. Updates job with the remote job id (mistify-agent has a local job queue), start time, and working status
    4. Requeue task in work beanstalk tube

The mistify-agent works on actually deleting the guest.

  1. cworkerd monitors job
    1. Reserves task from worker tube
    2. Makes request to hypervisor's mistify-agent to check job status
    3. If incomplete:
      1. Requeue in work beanstalk tube after brief delay to check again
    4. If complete/errored:
      1. Remove guest from etcd, free up IP, etc.
      2. Update job with finish time and final status (and error if applicable)
      3. Delete task from beanstalk

Reboot Guest

Reboot, power off, start, etc. can all be handled in the same way.

  1. End user makes request to cguestd to reboot guest, directly or via guest
  2. cguestd processes request
    1. Creates a job referencing the guest with an action for guest reboot
    2. Adds a task to the work beanstalk tube
    3. Returns guest object JSON and job ID to user. The rest of the process is asynchronous and the end user can request job status by ID via guest / cguestd
  3. cworkerd begins processing job
    1. Reserves the task from the work tube
    2. Makes request to the hypervisor's mistify-agent to reboot guest
    3. Updates job with the remote job id (mistify-agent has a local job queue), start time, and working status
    4. Requeue task in work beanstalk tube.

The mistify-agent works on actually rebooting the guest.

  1. cworkerd monitors job
    1. Reserves task from worker tube
    2. Makes request to hypervisor's mistify-agent to check job status
    3. If incomplete:
      1. Requeue in work beanstalk tube after brief delay to check again
    4. If complete/errored:
      1. Update job with finish time and final status (and error if applicable)
      2. Delete task from beanstalk
Clone this wiki locally