Introduce slog to replace env_logger and add journald logging

[mycognosist]

The env_logger library has proven very useful as a debugging and trouble-shooting tool during development. However, it is of limited use for introspecting and debugging applications on deployed systems.

I suggest we refactor the PeachCloud microservices, beginning with peach-stats, to utilize slog. The slog ecosystem includes a wide range of feature crates which provide various functionality, including terminal logging with environment variables and logging to journald.

Journald feels like a natural fit for PeachCloud since we are developing for Debian and using systemd for our application management. This route will provide us with a robust logging system with powerful filtering capabilities and log maintenance (deleting old logs and ensuring appropriate disk usage limits). The peach-probe diagnostic tool could leverage the journald logs to provide helpful information. Digital Ocean have an introductory article: How To Use Journalctl to View and Manipulate Systemd Logs.

A comparison of output for env_logger (currently in use) and slog-envlogger:

env_logger:

[2020-12-31T10:02:20Z INFO  peach_stats] Starting up.
[2020-12-31T10:02:20Z INFO  peach_stats] Creating JSON-RPC I/O handler.
[2020-12-31T10:02:20Z INFO  peach_stats] Starting JSON-RPC server on 127.0.0.1:5113.
[2020-12-31T10:02:20Z INFO  peach_stats] Listening for requests.
[2020-12-31T10:02:20Z DEBUG tokio_reactor] adding I/O source: 0

slog-envlogger:

version: 0.1.0
 Dec 31 10:04:44.364 INFO Logging ready!
 Dec 31 10:04:44.364 DEBG Testing the waters
 Dec 31 10:04:44.365 WARN Conditions are suboptimal
 Dec 31 10:04:44.365 ERRO It has all gone pear-shaped!, cause: cracked hull
 Dec 31 10:04:44.365 CRIT Abandon ship!

The slog version also features pretty colours in the terminal output and allows key-value pairs to be defined as part of the log statement (in the above example: cause: cracked hull).

And an example of a slog log entry in journald (showing date, time, user, process name, PID, and log message):

Dec 31 10:04:44 gnomad slog[8492]: Launching slog playground

In terms of the refactor, it only requires minor changes. A comparison of initialization and usage:

env_logger:

// initialize the logger
env_logger::init();
 
// log an error with a message
error!("Application error: speed wobble due to serialization error");

slog-envlogger:

// build the terminal decorator
let decorator = slog_term::TermDecorator::new().stderr().build();
// build the environment logger
let envlogger = slog_envlogger::new(slog_term::CompactFormat::new(decorator). build().fuse());
// define an asynchronous drain
let drain = slog_async::Async::default(envlogger);
// initialize the logger
let env_log = slog::Logger::root(drain.fuse(), o!("version" => "0.1.0"));

// log an error with a message and key-value
error!(env_log, "It has all gone pear-shaped!"; "cause" => "cracked hull");

slog-journald:

// initialize the logger
let jnl_log = slog::Logger::root(JournaldDrain.ignore_res(), o!());

// log info with a message and key-value
info!(jnl_log, "Launching slog playground"; "version" => "1.0.0");

I'm going to introduce this feature in peach-stats and we can use that as a test-case to decide if we're happy with the change and wish to introduce it to the rest of the PeachCloud ecosystem.

CC: @mhfowler