Jv integration tests for runtime config #1899

JoukoVirtanen · 2024-10-18T18:20:16Z

Description

Integration tests for runtime configuration. The tests create various versions of the runtime configuration file and check that the configuration introspection endpoint returns the correct expected configuration.

Checklist

Investigated and inspected CI test results
Updated documentation accordingly

Automated testing

Added unit tests
Added integration tests
Added regression tests

If any of these don't apply, please comment below.

Testing Performed

TODO(replace-me)
Use this space to explain how you tested your PR, or, if you didn't test it, why you did not do so. (Valid reasons include "CI is sufficient" or "No testable changes")
In addition to reviewing your code, reviewers must also review your testing instructions, and make sure they are sufficient.

For more details, ref the Confluence page about this section.

JoukoVirtanen · 2024-11-06T17:57:59Z

collector/lib/ConfigLoader.cpp

+  } catch (const YAML::Exception& e) {
+    CLOG(ERROR) << "Failed to parse the configuration file: " << config_file << ". Error: " << e.what();
+  } catch (const std::exception& e) {
+    CLOG(ERROR) << "An unknown error occured while loading the configuration file: " << config_file << ". Error: " << e.what();


Molter73

I'm less than half way through, I'll finish the review tomorrow.

collector/lib/ConfigLoader.cpp

Molter73 · 2024-11-06T18:05:56Z

collector/test/ConfigLoaderTest.cpp

@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
 }

 TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {


Please update this test name.

Molter73 · 2024-11-06T18:07:30Z

integration-tests/pkg/collector/collector_docker.go

@@ -41,6 +41,7 @@ func NewDockerCollectorManager(e executor.Executor, name string) *DockerCollecto
 		"/host/etc:ro":              "/etc",
 		"/host/usr/lib:ro":          "/usr/lib",
 		"/host/sys/kernel/debug:ro": "/sys/kernel/debug",
+		"/etc/stackrox:ro":          "/tmp",


Consider creating a separate directory for this like /tmp/collector-test and use that instead of the full /tmp directory.

Molter73 · 2024-11-06T18:16:26Z

integration-tests/pkg/collector/introspection.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		var response types.RuntimeConfig
+		err := json.Unmarshal(body, &response)
+		assert.NoError(t, err)
+
+		return response.Networking.ExternalIps.Enable == enable
+	})
+}
+
+func AssertNoRuntimeConfig(t *testing.T, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		return strings.TrimSpace(string(body)) == "{}"
+	})
+}
+
+// TODO: This should be in its own package
+func AssertRepeated(t *testing.T, condition func() bool) {
+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)
+
+	for {
+		select {
+		case <-tick.C:
+			if condition() {
+				// Condition has been met
+				return
+			}
+
+		case <-timer:
+			// TODO: This message should be passed in rather than hard coded here
+			t.Log("Timeout reached: Runtime configuration was not updated")
+			t.FailNow()
+		}
+	}
+}


We probably want to move all assertions to a separate pkg/assert or similar.

Molter73 · 2024-11-06T18:19:02Z

integration-tests/pkg/collector/introspection.go

+func QueryConfig(t *testing.T, collectorIP string) []byte {
+	log.Info("Querying: /state/config")
+	body, err := IntrospectionQuery(collectorIP, "/state/config")
+	assert.NoError(t, err)
+	log.Info("Response: %q", body)
+	return body
+}


I'm not 100% sold this is something we need, from a test we could just do:

body, err := IntrospectionQuery(s.Collector().IP(), "/state/config") s.Require().NoError(err)

This is simple enough and you can add the logging if you deem it necessary.

JoukoVirtanen · 2024-11-07T02:26:12Z

integration-tests/pkg/mock_sensor/expect_conn.go

@@ -33,7 +33,7 @@ loop:
 		case <-timer:
 			// we know they don't match at this point, but by using
 			// ElementsMatch we get much better logging about the differences
-			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for networks")
+			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for network connections")


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

JoukoVirtanen · 2024-11-07T02:26:25Z

integration-tests/pkg/mock_sensor/server.go

@@ -32,7 +32,6 @@ const (
 // us to use any comparable type as the key)
 type ProcessMap map[types.ProcessInfo]interface{}
 type LineageMap map[types.ProcessLineage]interface{}
-type ConnMap map[types.NetworkInfo]interface{}


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

Molter73 · 2024-11-07T12:17:08Z

integration-tests/pkg/assert/assert.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body, err := collector.IntrospectionQuery(collectorIP, "/state/config")


We don't need to do this here, but the IntrospectionQuery method needs to be part of a collector manager object, it would change this to be a lot simpler/clearer:

func AssertExternalIps(t *testing.T, enable bool, collector *CollectorManager) { AssertRepeated(t, func() bool { body, err := collector.IntrospectionQuery("/state/config")

I think it can be done in another PR. Maybe we should have an Introspection struct that could be a member of a Collector interface. The introspection object would get the IP address from the collector object.

integration-tests/pkg/types/runtime_config.go

integration-tests/suites/runtime_config_file.go

Molter73

Thanks for splitting the tests up, as silly as it may sound, it's a lot easier to understand what is going on now.

The comments I left now are mostly on moving some comments around to leave the code blocks on their own. This should help reduce clutter a bit more, since the description of what will be done is provided up front and the code is left as clean as possible.

integration-tests/suites/runtime_config_file.go

Molter73 · 2024-11-11T16:56:03Z

integration-tests/suites/runtime_config_file.go

+	// The runtime config file was deleted before starting collector so there should not be any config
+	assert.AssertNoRuntimeConfig(s.T(), collectorIP)
+	// Since there is no config the default is used, which means external IPs is disabled and we should
+	// expect a normalized connection


Suggested change

// The runtime config file was deleted before starting collector so there should not be any config

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

// Since there is no config the default is used, which means external IPs is disabled and we should

// expect a normalized connection

// The runtime config file was deleted before starting collector.

// Default configuration is external IPs disabled.

// We expect normalized connections.

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

This same change can be applied to the other 2 tests.

This was not applied to all tests.

It has now been applied to all tests.

integration-tests/suites/runtime_config_file.go

JoukoVirtanen · 2024-11-12T23:15:17Z

CI results

Run 1
Many Konflux integration tests failed. rhcos failed with

=== RUN   TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
    expect_conn.go:181: 
        	Error Trace:	/tests/pkg/mock_sensor/expect_conn.go:181
        	            				/tests/suites/procfs_scraper.go:72
        	Error:      	elements differ
        	            	
        	            	extra elements in list A:
        	            	([]interface ***) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listA:
        	            	([]types.EndpointInfo) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listB:
        	            	([]types.EndpointInfo) ***
        	            	***
        	Test:       	TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
        	Messages:   	timed out waiting for endpoints

The other konflux integration tests failed with either problems pulling the image. Both konflux and non-konflux cos integration tests failed with verifier errors.

Run 2:
Similar errors as above. No procfs test error. Collector also fails for fedora coreos.

Run 3:
rhcos had the following error

=== RUN   TestUdpNetworkFlow/TestUdpNetorkflow/sendto_recvmmsg
    udp_networkflow.go:274: 
        	Error Trace:	/tests/suites/udp_networkflow.go:274
        	            				/tests/suites/udp_networkflow.go:244
        	            				/tests/suites/udp_networkflow.go:105
        	            				/tests/suites/udp_networkflow.go:96
        	            				/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
        	Error:      	Received unexpected error:
        	            	Error response from daemon: container create: creating container storage: the container name "udp-server" is already in use by 0b88d4d870e77d5fed05bd5bad8a1dd1e5391d4e229d7ffb3d92a5bd57d0febf. You have to remove that container to be able to reuse that name: that name is already in use
        	            	create udp-server

I did not see a previous failure that caused this error. There were other failures unrelated to the changes in the PR.

Molter73

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

Stringy · 2024-11-13T10:36:01Z

integration-tests/pkg/assert/assert.go

+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)


Can these timeouts be configurable via some parameters?

JoukoVirtanen · 2024-11-14T04:02:00Z

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

I have addressed the PR review comments in the other PR.

integration-tests/pkg/assert/assert.go

integration-tests/pkg/mock_sensor/server.go

Molter73

LGTM, but holding approval until #1902 is merged, so this can be rebased on top of that.

e8bb38d Starting work 23ee1b0 Working with config introspection endpoint. No checks for connections yet 1b919ee Moved code for introspection endpoint from mock sensor to own directory 91ee35e Testing with an invalid config ae8bf37 Creates a container with an external connection and adds tests for it. Also made changes so that external connections can be checked for 34247e2 Mock server no longer maintains just a map of connections seen, but all connections edf7ca3 Should be able to handle cidr blocks and non-cidr blocks 4b227bd Able to handle the case when there is only port data dbed0a1 Brought in changes to server.go to handle external ips fd24fe8 Apply suggestions from code review 3eb5e48 Cleanup 13f8539 Able to control if connections should be ordered or not. Also added comments f9ea3fc Listening for new connection events instead of using a ticker 6d06416 Using ExpectSameElementsConnections instead of ExpectExactConnections 9c3f013 More error handling for reading the configuration. Using a volume mount a4fe63b Fixed lint error 8dce19c The runtime config file is only modified on the host machine 1c05a55 There is now only one source code file for runtime config introspection 7b77e23 Minor fixes for k8s e53509f Added comments and sleeps when config does not change so we get a scrape interval 2da34ae Increased the afterglow period ca6dd2c Update collector/lib/ConfigLoader.cpp 6aa60e0 Renamed unit test. Integration tests use a dedicated directory a6b0c57 Created a new assert package 949534d Apply suggestions from code review 09b13aa Split test into smaller tests 9346776 Apply suggestions from code review 3586964 Removed unneeded sort 5e95832 Using slices.ContainsFunc in HasConnection 76e244d Made timings in AssertRepeated configurable 5e94d5c Always sort the connections when comparing 12c04f3 Not using assertMismatch 129308e Updated comments 0b34466 Apply suggestions from code review e2fd23c Sorting connections inside Connections instead of SortedConnections a544a68 Created ElementsMatchFunc in assert pkg 6d1a737 Using spew for logging 2ae0101 Apply suggestions from code review 416b7f2 Importing logging package 9241f09 Simplified by removing checkIfConnectionsMatchExpected ba6babc Changed assert.True to assert.Fail

Molter73

LGTM!

Just a small comment on a TODO that I think we could solve quickly before merging, but we can also address it on the next PR that needs to use the helper function.

Molter73 · 2024-11-28T09:36:46Z

integration-tests/pkg/assert/assert.go

+			// TODO: This message should be passed in rather than hard coded here
+			log.Error("Timeout reached: Runtime configuration was not updated")


This TODO should be relatively simple to solve, add a format and args parameter to the function like this:

func AssertRepeated(t *testing.T, tickTime time.Duration, timeout time.Duration, condition func() bool, msg string, args ...interface{}) {

Then here you can just call log.Error like this:

log.Error("Timeout reached: " + msg, args...)

If this ends up being too verbose on the call sites with the function being defined inline, you can always define the function and assign it to a variable, then pass that in.

Molter73 · 2024-11-28T09:41:50Z

Also, there are some tests on the multiarch for konflux, but those are due to the integration test image not being rebuilt for those archs, this is not a blocker for this PR and I'm trying to address it here: #1974

openshift-ci bot added the do-not-merge/work-in-progress label Oct 18, 2024

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 316fecb to ae8bf37 Compare October 20, 2024 18:12

JoukoVirtanen mentioned this pull request Oct 22, 2024

ROX-26711: Collector mock server should handle external ips #1901

Merged

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 4b6cf76 to 4b227bd Compare October 22, 2024 20:43

JoukoVirtanen mentioned this pull request Oct 23, 2024

ROX-26726: Collector integration tests should record all connections #1902

Merged

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 6d06416 to 075a4df Compare November 1, 2024 18:59

JoukoVirtanen commented Nov 6, 2024

View reviewed changes

Molter73 reviewed Nov 6, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from b446ed0 to 347114d Compare November 6, 2024 19:21

JoukoVirtanen changed the base branch from master to jv-ROX-26726-collector-integration-tests-should-record-all-connections November 7, 2024 02:18

JoukoVirtanen changed the base branch from jv-ROX-26726-collector-integration-tests-should-record-all-connections to master November 7, 2024 02:23

JoukoVirtanen commented Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 7, 2024 03:30

Molter73 reviewed Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 11, 2024 04:05

Molter73 reviewed Nov 11, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 12, 2024 02:28

JoukoVirtanen marked this pull request as ready for review November 12, 2024 23:16

JoukoVirtanen requested a review from a team as a code owner November 12, 2024 23:16

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 12, 2024

Molter73 reviewed Nov 13, 2024

View reviewed changes

Stringy reviewed Nov 13, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 8420864 to 9b4f72e Compare November 14, 2024 01:54

JoukoVirtanen requested review from Molter73 and Stringy November 14, 2024 04:13

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from d734a7c to f092de4 Compare November 16, 2024 00:27

Molter73 reviewed Nov 20, 2024

View reviewed changes

integration-tests/pkg/assert/assert.go Outdated Show resolved Hide resolved

integration-tests/pkg/mock_sensor/server.go Outdated Show resolved Hide resolved

JoukoVirtanen requested a review from Molter73 November 21, 2024 03:27

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 4a03f7b to 9241f09 Compare November 21, 2024 05:20

Molter73 reviewed Nov 25, 2024

View reviewed changes

JoukoVirtanen added 3 commits November 27, 2024 10:48

X-Smart-Branch-Parent: master

4631d27

Cleanup after rebase

72fb150

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from ba6babc to 72fb150 Compare November 28, 2024 03:14

Molter73 approved these changes Nov 28, 2024

View reviewed changes

JoukoVirtanen added 2 commits December 1, 2024 18:31

Expected and observed connections are passed in the correct order

a079e91

Not hard coding error message in AssertRepeated

28db28a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jv integration tests for runtime config #1899

Jv integration tests for runtime config #1899

JoukoVirtanen commented Oct 18, 2024

JoukoVirtanen Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 left a comment

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

JoukoVirtanen Nov 7, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 7, 2024

JoukoVirtanen Nov 7, 2024

Molter73 left a comment

Molter73 Nov 11, 2024

Molter73 Nov 11, 2024

Molter73 Nov 13, 2024

JoukoVirtanen Nov 14, 2024

JoukoVirtanen commented Nov 12, 2024 •

edited

Loading

Molter73 left a comment

Stringy Nov 13, 2024

JoukoVirtanen Nov 14, 2024

JoukoVirtanen commented Nov 14, 2024

Molter73 left a comment

Molter73 left a comment

Molter73 Nov 28, 2024

Molter73 commented Nov 28, 2024

		@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
		}

		TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {

		tick := time.NewTicker(1 * time.Second)
		timer := time.After(3 * time.Minute)

		// TODO: This message should be passed in rather than hard coded here
		log.Error("Timeout reached: Runtime configuration was not updated")

Jv integration tests for runtime config #1899

Are you sure you want to change the base?

Jv integration tests for runtime config #1899

Conversation

JoukoVirtanen commented Oct 18, 2024

Description

Checklist

Testing Performed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 12, 2024 • edited Loading

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 14, 2024

Molter73 left a comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 commented Nov 28, 2024

JoukoVirtanen commented Nov 12, 2024 •

edited

Loading