Allow multiple private keys but use the latest one

[h2zh]

Overview

In issue #561, we want to rotate out old private key when the new one is provided. This PR mainly changed the functions chain starting from GetIssuerPrivateJWK(), in order to support multiple private key files in a new directory specified by a new config param IssuerKeysDirectory, along with a new goroutine to monitor the changes there every 5 mins.

New Key

In this PR, admin can create new private key through either of the following two ways:

1. Simply drop the private key into into the new directory as a .pem file (the location where config param IssuerKeysDirectory points to, which by default is /etc/pelican/issuer-keys), or modify any .pem there. Pelican program will pick up the last modified .pem file through a goroutine running every 5 mins, set the private key it contains as the active one, add it to an in-memory map, and update the public key in registry sqlite db.
2. Hit the API endpoint “/api/v1.0/origin_ui/newIssuerKey”. Then the new .pem file will be created. Then it will wait for the same goroutine to be loaded (note: The web ui for this API is not included)

Design thinking

Existing file hierarchy:

├── whatever-parent-dir/ (by default /etc/pelican)                             
│   ├── issuer.jwk            # Existing key location, specified by config param “IssuerKey”

To ensure backward compatibility, the new directory that stores the private keys (saved as .pem files) is mounted to a new config param IssuerKeysDirectory. User doesn’t need to change the value of IssuerKey, though all functions are not longer using it. The private key IssuerKey refers to will be migrated to the new directory specified by IssuerKeysDirectory. The new file hierarchy looks like this:

├── whatever-parent-dir/ (by default /etc/pelican)                             
│   └── issuer-keys/          # The new directory storing multiple private keys where “IssuerKeysDirectory” refers to
│       ├── pelican_generated_<timestamp_1>_<randomChars>.pem    # If no private key is provided, new .pem file will be created
│       ├── pelican_generated_<timestamp_2>_<randomChars>.pem         
│       ├── ...                 
│       ├── <system admin's fav name>.pem
│       └── migrated_key.pem    # previous key if it exists, migrated from the location where “IssuerKey” points to

Algorithm efficiency tradeoff

For a map using atomic.pointer, it requires to copy the entire map before update any key-value pair. In the loadPEMFiles func, because it runs every 5 minutes to update the in-memory key map from the file, which is not frequent, we think the memory use by a map using atomic.pointer is still acceptable.

Test

• No private key file
  TODO

• One existing private key file, adding a new private key
  a) an existing private key file at config param IssuerKey (usually .../issuer.jwk), adding a new private key manually
  b) an existing private key file at config param IssuerKey (usually .../issuer.jwk), adding a new private key via API
  c) an existing private key file at config param IssuerKeysDirectory (usually .../issuer-keys), adding a new private key manually
  d) an existing private key file at config param IssuerKeysDirectory (usually .../issuer-keys), adding a new private key via API
  --> result of any scenario in this section: in-memory active key: second; public key of this origin's namespaces in registry db: second

• Two existing private key files
  a) never run pelican before (empty registry db)
  --> in-memory active key: second; public key of this origin's namespaces in registry db: second
  b) first key is registered in registry db
  --> in-memory active key: second; public key of this origin's namespaces in registry db: second
  c) second key is registered in registry db
  --> in-memory active key: second; public key of this origin's namespaces in registry db: second
  d) second key is registered in registry db, adding the third key through API
  --> in-memory active key: third; public key of this origin's namespaces in registry db: third

Note: "first", "second" key are keys in chronological order of their private key file modification time.
In-memory active key can be copied at the top-right key icon in origin webUI; db public key of this namespace can be found at the registry.sqlite file, whose directory is specified by config param Registry.DbLocation, or in etc/pelican by default

Future work

At this moment, we use the most recent modified .pem file as the active private key. In the future, we want to allow multiple active private keys in use.

Eventually, when multiple origins have the same namespace, the admin of these origins can append a common private key to all. Then these origin servers will register the corresponding shared public key in the registry. Currently this operation needs to manually done by OSDF admin.