Rich file based discovery

[ripienaar]

Today we support the following discovery methods:

• network broadcast, the only one with compound filters
• puppetdb
• text file 1 node per line
• json file - an array with just nodes in it
• yaml file - an array with just nodes in it
• a previous request as JSON

We are busy adding project based configuration and I think it might be useful to also have project based inventory in that case, and just in general inventory files might be handy.

Compound Filters

We can support the same compound filters as -S, but without the future data plugin feature

Configuration

plugin.choria.discovery.inventory.source = ~/choria-inventory.yaml

In a per project choria.conf one can have:

plugin.choria.discovery.inventory.source = nodes.yaml

which will ensure the local nodes.yaml takes precedence.

Inventory format

$schema: io.choria.choria.discovery.v1.inventory
groups:
  - name: webservers
    filter:
      classes: [apache]
      
nodes:
  - name: example.net
    facts: {}
    classes: []
    collectives: [mcollective]
    agents:
      - name: foo
        version: 1.2.3

Here we define named groups as filters that get applied over the data in the file, effectively a pre-baked discovery query with a name.

CLI Usage

Normal discovery for class/agent/identity/fact/etc

$ choria req rpcutil ping --dm inventory -C apache

Reference a group:

$ choria req rpcutil ping --dm inventory -C group:webservers

Maintaining the file

Existing files can be validated against the schema:

$ choria tool inventory --validate inventory.yaml

The file can be created using another discovery method and the rpcutil#inventory agent and action:

$ choria tool inventory inventory.yaml --update -W country=mt
Discovering nodes .... 19

19 / 19    0s [====================================================================] 100%

Wrote 19 nodes to /home/user/inventory.yaml

Future Streaming Integration

Choria can publish all the above information as part of its registration feature, that data can go into our upcoming Streaming Server for analysis and query

One could create/update the above file like this:

$ choria tool inventory create --history 10m inventory.yaml --stream REGISTRATION --filter facts.client=acme

This will go back 10 minutes in time and any node seen in the last 10 minutes from the stream REGISTRATION created via registration that match an expr filter will be persisted to the file.

It can also build the file by simply listening to the registration stream passively,:

$ choria tool inventory create --subject choria.registration inventory.yaml --filter facts.client=acme

This will run until ^C is pressed, any node seen during this time will be persisted.

If it's an existing file the nodes will be updated, groups will be kept