Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support remote log level #4413

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Support remote log level #4413

wants to merge 6 commits into from
+850,908 −13,322

Commits on Nov 28, 2024

  1. add support for remote log level to newlogd 

    As opposed to the default log level, which was setting which logs are
produced by EVE's microservices, the remote log level will set which
device logs are uploaded to the cloud. So it's assumed that the remote
log level is always higher than the default log level.

There are no changes in how newlogd collects the logs, only in how it
handles the log files.
For the logs that are uploaded:
- we create a separate file with prefix dev.log.upload in collect
- that file is gzipped when it reaches a certain size or by timer - the
  same as before
- once gzipped the file is moved to devUpload - the same as before
- once the file is uploaded successfully, it's deleted instead of being
  moved to keepSentQueue

For the logs that stay on the device:
- we create a separate file with prefix dev.log.keep in collect
- that file is gzipped when it reaches a certain size - no timer
- once gzipped the file is moved to keepSentQueue - the name of the
  folder is preserved

The commit also adds log levels "all" and "none" as well as remote log
levels for kernel and syslog logs:
as the names suggest, "all" will log everything and "none" will suppress
all logs. The levels can be set in the global configuration for
- debug.default.loglevel
- debug.default.remote.loglevel
- agent.*agentname*.debug.loglevel
- agent.*agentname*.debug.remote.loglevel
- debug.syslog.loglevel
- debug.syslog.remote.loglevel
- debug.kernel.loglevel
- debug.kernel.remote.loglevel

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    868be54 View commit details
    Browse the repository at this point in the history

  2. new metrics for newlogd 

    Add new metrics for newlogd:
- OldestSavedLog - the timestamp of the oldest log available on the
    device
- TotalSizeLogs - the total size of all logs on the device

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    9c08b1b View commit details
    Browse the repository at this point in the history

  3. define clear order of log file removal 

    In case the total size of gzipped logs exceeds the limit
they should be removed in the following order by directory:
- `keepSentQueue`
- `failedUpload`
- `devUpload`
- `appUpload`

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    5a3bdb7 View commit details
    Browse the repository at this point in the history

  4. devUpload should not be parsed by egde-view 

    Since keepSentQueue contains the same or more logs than
devUpload, it's enough to only use those logs when
searching with edge-view.

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    157b2aa View commit details
    Browse the repository at this point in the history

  5. update documentation on remote log level settings 

    Add a chapter to LOGGING.md explaining how different log levels are
handled in the system.

Updated CONFIG-PROPERTIES.md to clarify remote log level settings and
provide full list of log level parameters

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    3f7c35e View commit details
    Browse the repository at this point in the history

  6. remove logs longer than memlogd's max-line-len 

    Currently memlogd has set a max-line-len of 8192. If the log line is
longer than this, it will be truncated. Since we use structured logs
for EVE microservices, truncating logs will result in malformed JSON,
which is not parseable by newlogd.

This commit removes logs that have a potential to be longer than that
limit, because they log whole configs that tend to get very long.

In future developers should be aware of this limit and avoid logging
big chunks of data.

Signed-off-by: Paul Gaiduk <[email protected]>
    @europaul
    europaul committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    d622fc0 View commit details
    Browse the repository at this point in the history