Support remote log level #4413

europaul · 2024-10-31T20:33:09Z

This PR is a work in progress, but I appreciate if somebody could take a look at the code and the general approach and give me feedback.

Please see the commit messages (especially the second commit) for the overview of the changes.

Before merging this I need to update the pillar first to include the dependency in newlogd. I also need to update eve-api to include the new metrics.

I'm also planning to add more metrics as well as proper tests for newlogd to at least ensure that there is no regression.

europaul · 2024-10-31T20:37:01Z

the build is gonna break of course because the dependencies are missing in pillar

pkg/newlog/cmd/newlogd.go

pkg/pillar/types/newlogtypes.go

naiming-zededa

This patch in general looks good to me, I have some comments

in collect-info.sh for 'newlog' directory, we should skip the 'devUpload' directory, since those not yet uploaded files will be the duplicates for the keep directory
similar in the edgeview in walkLogDirs() function, we need to skip the 'devUpload' directory to avoid duplicates
in loguploader.go, there is 'failSendDir' and logic to move the failed to send to controller log files and keep them there. With this change, i don't think it is needed anymore, since everything will be in the keep directory. this can simplify some logic
when doing kibana search, i do search for 'Alpine' string for locate when the device has rebooted, now we may not have this. maybe to always log at least the first 5 device log entries?

pkg/newlog/cmd/newlogd.go

eriknordmark

Can you update the description in https://github.com/lf-edge/eve/blob/master/docs/LOGGING.md with the intended new state of things?

europaul · 2024-11-06T18:10:24Z

Can you update the description in https://github.com/lf-edge/eve/blob/master/docs/LOGGING.md with the intended new state of things?

@eriknordmark I made the changes to the documentation, please have a look. I'm not sure if it's an overkill with the debug.local.loglevel and is a feature that nobody wants that only makes the system harder to understand?

europaul · 2024-11-25T21:40:50Z

@naiming-zededa
Thank you very much for the detailed comments ❤️
I addressed this comment

* similar in the edgeview in walkLogDirs() function, we need to skip the 'devUpload' directory to avoid duplicates

but I would like to address the following in the next PR to not bloat the scope of this one.

* in collect-info.sh for 'newlog' directory, we should skip the 'devUpload' directory, since those not yet uploaded files will be the duplicates for the keep directory

* in loguploader.go, there is 'failSendDir' and logic to move the failed to send to controller log files and keep them there. With this change, i don't think it is needed anymore, since everything will be in the keep directory. this can simplify some logic

* when doing kibana search, i do search for 'Alpine' string for locate when the device has rebooted, now we may not have this. maybe to always log at least the first 5 device log entries?

europaul · 2024-11-25T21:43:07Z

@eriknordmark I think the architecture should be right from our discussions. Please give this PR another look now that I adapted the implementation accordingly.
I would like to include proper testing in another PR, as this will require big structural changes to newlog, which would make this PR harder to follow.

europaul · 2024-11-25T21:45:21Z

Btw before merging this I'll need to change the EVE API and pillar first, so that I can update the dependencies here. Please let me know if you are satisfied with the changes, then I'll prepare the respective PRs.

docs/CONFIG-PROPERTIES.md

docs/LOGGING.md

docs/CONFIG-PROPERTIES.md

eriknordmark

Approach looks fine to me so please proceeed generating the other PRs/commits.

@naiming-zededa can you take a look at the newlogd code changes?

uncleDecart · 2024-11-26T11:55:07Z

pkg/edgeview/src/edgeview_test.go

+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"


why do we want testify? Can't we use gomega for tests like we do in other golang tests in EVE repository?

#3435 (comment) ;-)

it's because I know how to use testify and I don't know how to use gomega 😄
but if you want I'll learn how to use gomega and switch to it. is it better than testify?

Brining go-chi was useful, because they have route parameter parser built-in, so we didn't have to build our own. The only reason why we left parts of Gorilla then was because we used ws communication (happy to restart the conversation of refactoring it). Rest was completely replaced by go-chi. Basically I'm asking the same question here: what's the benefit of testify over gomega and why do we want to have two testing frameworks in the repo

https://github.com/gorilla/mux/blob/main/README.md?plain=1#L303 - isn't that the same?

https://github.com/gorilla/mux/blob/main/README.md?plain=1#L303 - isn't that the same?

We didn't go for gorilla-mux, because at the time it didn't have maintainers, now they do have them. We could have just used standard http mux from golang, but then we would have to write our own route parameter parser

https://gorilla.github.io/blog/2023-07-17-project-status-update/

@uncleDecart rewrote the tests with gomega - it's not much of a difference, so it makes sense to me to stick with one testing framework

btw

grep -r "github.com/stretchr/testify/assert" pkg/ | grep "_test.go" | wc -l 26

Thank you very much for change! Missed the fact that you bring testify to edge_view, not pillar, didn't know we were using testify, I think it's a good raise on LF-Edge call

europaul · 2024-11-26T16:54:51Z

the changes to EVE API are here

europaul · 2024-11-26T17:17:32Z

the dependencies on pillar types are here

eriknordmark · 2024-11-26T18:43:23Z

docs/CONFIG-PROPERTIES.md

+| debug.kernel.loglevel | string | info | level of the produced kernel log messages. System default loglevel string representation should be used as described here ["https://man7.org/linux/man-pages/man3/syslog.3.html"]: emerg, alert, crit, err, warning, notice, info, debug. |
+| debug.kernel.remote.loglevel | string | info | level of the kernel log messages sent to the controller. System default loglevel string representation should be used as described here ["https://man7.org/linux/man-pages/man3/syslog.3.html"]: emerg, alert, crit, err, warning, notice, info, debug. |
+
+In addition, there can be per-agent settings to overwrite the default log level set for zedbox.


s/zedbox/EVE microservices/ (even though this only applies to some microservices like downloader and zedagent and not to others like wwan and wlan.

sorry, I meant to replace all of the occurrences and forgot that one

actually this applies to wwan as well
https://github.com/lf-edge/eve/blob/08d0583f68c9ce35747d2ea6566a9b30ad7404c6/pkg/wwan/mmagent/agent.go#L486

here is the full list of all "agents" that support log level settings (call agentlog.HandleGlobalConfig) right now:
newlogd
wwan
nodeagent
downloader
tpmmgr
client
vcomlink
executor
vaultmgr
baseosmgr
zedagent
verifier
wstunnelclient
zfsmanager
zedkube
ledmanager
faultinjection
zedmanager
nim
loguploader
watcher
volumemgr
zedrouter
msrv
domainmgr
diag

I added it also to CONFIG-PROPERTIES.md

eriknordmark · 2024-11-26T18:44:32Z

docs/CONFIG-PROPERTIES.md

+
+Additionally all log levels can be set to "none" to disable logging for the corresponding component or to "all" to enable all log levels.
+
+Furthermore, the "remote" log levels control, which subset of the produced logs will be sent to the controller.


FWIW I don't think the comma after "control" makes any sense. But I'm not a native english speaker.

you're right. copilot suggested it to me and I didn't dare to contradict it 🤖

uncleDecart

LGTM, FWIW you can tick "include copy of your diagram" in draw.io and then you'll have just one file my_diagram.drawio.png and if you can edit that one directly in draw.io

As opposed to the default log level, which was setting which logs are produced by EVE's microservices, the remote log level will set which device logs are uploaded to the cloud. So it's assumed that the remote log level is always higher than the default log level. There are no changes in how newlogd collects the logs, only in how it handles the log files. For the logs that are uploaded: - we create a separate file with prefix dev.log.upload in collect - that file is gzipped when it reaches a certain size or by timer - the same as before - once gzipped the file is moved to devUpload - the same as before - once the file is uploaded successfully, it's deleted instead of being moved to keepSentQueue For the logs that stay on the device: - we create a separate file with prefix dev.log.keep in collect - that file is gzipped when it reaches a certain size - no timer - once gzipped the file is moved to keepSentQueue - the name of the folder is preserved The commit also adds log levels "all" and "none" as well as remote log levels for kernel and syslog logs: as the names suggest, "all" will log everything and "none" will suppress all logs. The levels can be set in the global configuration for - debug.default.loglevel - debug.default.remote.loglevel - agent.*agentname*.debug.loglevel - agent.*agentname*.debug.remote.loglevel - debug.syslog.loglevel - debug.syslog.remote.loglevel - debug.kernel.loglevel - debug.kernel.remote.loglevel Signed-off-by: Paul Gaiduk <[email protected]>

Add new metrics for newlogd: - OldestSavedLog - the timestamp of the oldest log available on the device - TotalSizeLogs - the total size of all logs on the device Signed-off-by: Paul Gaiduk <[email protected]>

In case the total size of gzipped logs exceeds the limit they should be removed in the following order by directory: - `keepSentQueue` - `failedUpload` - `devUpload` - `appUpload` Signed-off-by: Paul Gaiduk <[email protected]>

codecov · 2024-11-27T19:34:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 20.93%. Comparing base (a73c78d) to head (4eeccfc).
Report is 8 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4413   +/-   ##
=======================================
  Coverage   20.93%   20.93%           
=======================================
  Files          13       13           
  Lines        2895     2895           
=======================================
  Hits          606      606           
  Misses       2163     2163           
  Partials      126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Since keepSentQueue contains the same or more logs than devUpload, it's enough to only use those logs when searching with edge-view. Signed-off-by: Paul Gaiduk <[email protected]>

Add a chapter to LOGGING.md explaining how different log levels are handled in the system. Updated CONFIG-PROPERTIES.md to clarify remote log level settings and provide full list of log level parameters Signed-off-by: Paul Gaiduk <[email protected]>

eriknordmark

LGTM

europaul · 2024-11-27T19:38:44Z

the last push just fixed some spelling found by local yetus run, since the one in CI doesn't work

europaul · 2024-11-27T19:45:21Z

thank you for your patience! in the final version I

reworked how we check the remote log level - instead of checking all log entries when writing to the temporary log file, we now check them when getting them from the source. This way it's easier to differentiate between the three main sources and log levels: EVE microservices, kernel and syslog
made sure the log levels coming from the global config are set and read atomically to prevent potential concurrency issues as they are set and read by different goroutines

europaul · 2024-11-27T20:06:58Z

could it be that yetus is failing because it doesn't exclude the changes to vendor dirs and fails to process the amount of files changed?

github-actions bot requested review from deitch, eriknordmark and naiming-zededa October 31, 2024 20:33

christoph-zededa reviewed Oct 31, 2024

View reviewed changes

pkg/newlog/cmd/newlogd.go Outdated Show resolved Hide resolved

uncleDecart reviewed Nov 1, 2024

View reviewed changes

pkg/newlog/cmd/newlogd.go Outdated Show resolved Hide resolved

naiming-zededa reviewed Nov 1, 2024

View reviewed changes

pkg/newlog/cmd/newlogd.go Outdated Show resolved Hide resolved

naiming-zededa reviewed Nov 1, 2024

View reviewed changes

pkg/pillar/types/newlogtypes.go Outdated Show resolved Hide resolved

naiming-zededa reviewed Nov 1, 2024

View reviewed changes

pkg/newlog/cmd/newlogd.go Outdated Show resolved Hide resolved

eriknordmark requested changes Nov 5, 2024

View reviewed changes

europaul force-pushed the support-remote-log-level branch from 580384e to c894bec Compare November 6, 2024 18:08

github-actions bot requested review from eriknordmark and naiming-zededa November 6, 2024 18:08

europaul force-pushed the support-remote-log-level branch from c894bec to 5b2cfcf Compare November 18, 2024 10:31

europaul force-pushed the support-remote-log-level branch from 5b2cfcf to 8e0fafa Compare November 25, 2024 21:36

europaul marked this pull request as ready for review November 25, 2024 21:37

github-actions bot requested a review from christoph-zededa November 25, 2024 21:38

eriknordmark reviewed Nov 26, 2024

View reviewed changes

docs/CONFIG-PROPERTIES.md Outdated Show resolved Hide resolved

eriknordmark reviewed Nov 26, 2024

View reviewed changes

docs/CONFIG-PROPERTIES.md Outdated Show resolved Hide resolved

eriknordmark reviewed Nov 26, 2024

View reviewed changes

docs/LOGGING.md Outdated Show resolved Hide resolved

eriknordmark reviewed Nov 26, 2024

View reviewed changes

docs/LOGGING.md Outdated Show resolved Hide resolved

eriknordmark reviewed Nov 26, 2024

View reviewed changes

docs/CONFIG-PROPERTIES.md Outdated Show resolved Hide resolved

eriknordmark approved these changes Nov 26, 2024

View reviewed changes

uncleDecart reviewed Nov 26, 2024

View reviewed changes

europaul force-pushed the support-remote-log-level branch from 8e0fafa to 846b26b Compare November 26, 2024 15:06

github-actions bot requested a review from eriknordmark November 26, 2024 15:08

europaul mentioned this pull request Nov 26, 2024

Add new types and functions to pillar needed by newlog and edge-view #4449

Merged

eriknordmark reviewed Nov 26, 2024

View reviewed changes

uncleDecart approved these changes Nov 27, 2024

View reviewed changes

europaul added 3 commits November 27, 2024 20:05

new metrics for newlogd

079e5a6

Add new metrics for newlogd: - OldestSavedLog - the timestamp of the oldest log available on the device - TotalSizeLogs - the total size of all logs on the device Signed-off-by: Paul Gaiduk <[email protected]>

define clear order of log file removal

49cd19d

In case the total size of gzipped logs exceeds the limit they should be removed in the following order by directory: - `keepSentQueue` - `failedUpload` - `devUpload` - `appUpload` Signed-off-by: Paul Gaiduk <[email protected]>

europaul force-pushed the support-remote-log-level branch from 846b26b to 8bf6b25 Compare November 27, 2024 19:14

europaul requested review from OhmSpectator, milan-zededa and rouming as code owners November 27, 2024 19:14

europaul force-pushed the support-remote-log-level branch from 8bf6b25 to 4eeccfc Compare November 27, 2024 19:16

github-actions bot requested review from eriknordmark and uncleDecart November 27, 2024 19:34

europaul added 2 commits November 27, 2024 20:36

devUpload should not be parsed by egde-view

78eda3e

Since keepSentQueue contains the same or more logs than devUpload, it's enough to only use those logs when searching with edge-view. Signed-off-by: Paul Gaiduk <[email protected]>

eriknordmark approved these changes Nov 27, 2024

View reviewed changes

github-actions bot requested a review from eriknordmark November 27, 2024 19:36

europaul force-pushed the support-remote-log-level branch from 4eeccfc to 95c5b13 Compare November 27, 2024 19:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support remote log level #4413

Support remote log level #4413

europaul commented Oct 31, 2024

europaul commented Oct 31, 2024

naiming-zededa left a comment

eriknordmark left a comment

europaul commented Nov 6, 2024

europaul commented Nov 25, 2024

europaul commented Nov 25, 2024

europaul commented Nov 25, 2024

eriknordmark left a comment

uncleDecart Nov 26, 2024

christoph-zededa Nov 26, 2024

europaul Nov 26, 2024

uncleDecart Nov 26, 2024

christoph-zededa Nov 26, 2024

uncleDecart Nov 26, 2024

christoph-zededa Nov 26, 2024

europaul Nov 26, 2024 •

edited

Loading

europaul Nov 26, 2024

uncleDecart Nov 27, 2024

europaul commented Nov 26, 2024

europaul commented Nov 26, 2024

eriknordmark Nov 26, 2024

europaul Nov 26, 2024

europaul Nov 26, 2024

europaul Nov 26, 2024

europaul Nov 26, 2024

eriknordmark Nov 26, 2024

europaul Nov 26, 2024

uncleDecart left a comment

codecov bot commented Nov 27, 2024

eriknordmark left a comment

europaul commented Nov 27, 2024

europaul commented Nov 27, 2024

europaul commented Nov 27, 2024


		Additionally all log levels can be set to "none" to disable logging for the corresponding component or to "all" to enable all log levels.

		Furthermore, the "remote" log levels control, which subset of the produced logs will be sent to the controller.

Support remote log level #4413

Are you sure you want to change the base?

Support remote log level #4413

Conversation

europaul commented Oct 31, 2024

europaul commented Oct 31, 2024

naiming-zededa left a comment

Choose a reason for hiding this comment

eriknordmark left a comment

Choose a reason for hiding this comment

europaul commented Nov 6, 2024

europaul commented Nov 25, 2024

europaul commented Nov 25, 2024

europaul commented Nov 25, 2024

eriknordmark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

europaul Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

europaul commented Nov 26, 2024

europaul commented Nov 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uncleDecart left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 27, 2024

Codecov Report

eriknordmark left a comment

Choose a reason for hiding this comment

europaul commented Nov 27, 2024

europaul commented Nov 27, 2024

europaul commented Nov 27, 2024

europaul Nov 26, 2024 •

edited

Loading