[RFE] pmlogger push protocol

[natoscott]

After yet another customer call where this topic came up, and
trying to shoe-horn existing approaches to solve this, I have
started pondering alternatives.

The issue revolves around modern corporate environments where
firewalls and just the scale of deployments in general make it
difficult to manage the current pmlogger pull model. Instead,
an optional, new model where individual hosts manage their own
local logging configuration and push over a socket that which
would usually be written to local disk is more desirable.

This removes the requirement of a remote pmlogger (farm) where
multiple pmloggers connect to the logged host and pull in all
the content that will be written to disk.

How might this work? A new pmlogger option (-R/--remote) that
specifies the remote host:port would activate new logic inside
pmlogger. This mode would effectively intercept those points
where usually we write to disk and instead send those over the
network using a new pmlogger-specific wire protocol. Now that
is pretty hand-wavy - there's alot to flesh out there (initial
log label, metadata, temporal index PDUs?, log volume sizing /
switching?, etc) but I think this should be feasible.

It'd be great if we could dovetail in with existing services -
so just the one new option to pmlogger for client-side setups,
and pmlogconf still doing configuration there, etc. Remotely,
it'd be excellent if pmproxy was the server catching these new
style connections/messages and logging to disk, and/or with an
option of going directly to a key server (Redis/Valkey) if no
PCP archive is needed for some deployment scenarios.

I think this tackles the issue being addressed by issue #1918
more elegantly since it a/ solves the client-side configuring
aspects more neatly, and b/ adds in PCP archives as the first
solution, with just optional direct-to-key-server mode.

[mallilinux1980]

Do you think we can get some traction here?

[natoscott]

I hope so, in due course. Please be aware that much of the work undertaken on PCP is unpaid, volunteer effort which you / your company get for free, so your question could be construed as a little bit entitled.

This RFE in particular is quite complex, and completing it will require some advanced skills from a very experienced software engineer. These skills are not cheap, and open source developers need to eat too. If this was a paid project I estimate a suitably skilled developer (not me) could implement this in 2-3 months, and based on typical annual salaries for such a suitably skilled developer, this feature would easily cost a company somewhere in the order of USD$50,000 to develop.

If you (or anyone) would like to make a financial contribution for this, and/or the other support you get here, please see the 'Sponsor' link at the top of the https://github.com/performancecopilot/pcp page.

If you need to specifically see this RFE implemented and have funding available, send me a private email and I'll put you in touch with a suitably skilled individual (again, not me) who may assist.