🧑‍🌾 test_pub_sub, test_publisher and test_subscription failures

[Crola1702]

Description

These tests have been failing for a long time in Iron connext and fastrtps-dynamic.

Reference builds:

• Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64#155
• Ici__nightly-connext_ubuntu_jammy_amd64#158

Test regressions:

• test_tracetools.test_tracetools.test.test_subscription.TestSubscription.test_all
• test_tracetools.test_tracetools.test.test_publisher.TestPublisher.test_all
• test_tracetools.test_tracetools.test.test_pub_sub.TestPubSub.test_all
• projectroot.test_pub_sub

All three contain the same log output:

Log output:

11: =================================== FAILURES ===================================
11: __________________________ TestSubscription.test_all ___________________________
11: /tmp/ws/install_isolated/tracetools_test/lib/python3.10/site-packages/tracetools_test/case.py:110: in setUp
11:     self.assertSetEqual(all_event_names, set(self._event_names))
11: E   AssertionError: Items in the first set but not the second:
11: E   'ros2:rmw_subscription_init'
11: E   'ros2:rmw_take'
11: ----------------------------- Captured stdout call -----------------------------
11: [INFO] [launch]: All log files can be found below /home/buildfarm/.ros/log/2024-08-19-01-58-16-902022-a9e85b8b6b09-444099
11: [INFO] [launch]: Default logging verbosity is set to INFO
11: [INFO] [tracetools_launch.action]: Writing tracing session to: /tmp/session-test-subscription-20240819015816
11: [INFO] [test_ping-1]: process started with pid [444104]
11: [INFO] [test_pong-2]: process started with pid [444106]
11: [test_pong-2] [INFO] [1724061498.067881993] [test_pong]: [output] some random ping string
11: [test_ping-1] [INFO] [1724061498.068240921] [test_ping]: [output] some random pong string
11: [test_pong-2] RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
11: [test_pong-2] Expires on 00-jan-00 See www.rti.com for more information.
11: [test_pong-2] spinning
11: [INFO] [test_pong-2]: process has finished cleanly [pid 444106]
11: [test_ping-1] RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
11: [test_ping-1] Expires on 00-jan-00 See www.rti.com for more information.
11: [test_ping-1] spinning
11: [INFO] [test_ping-1]: process has finished cleanly [pid 444104]
11: TRACE DIRECTORY: /tmp/session-test-subscription-20240819015816
11: - generated xml file: /tmp/ws/test_results/test_tracetools/test_subscription.xunit.xml -
11: 
11: ---------- coverage: platform linux, python 3.10.12-final-0 ----------
11: 
11: =========================== short test summary info ============================
11: FAILED test/test_subscription.py::TestSubscription::test_all - AssertionError...
11: ============================== 1 failed in 2.52s ===============================
11: -- run_test.py: return code 1
11: -- run_test.py: verify result file '/tmp/ws/test_results/test_tracetools/test_subscription.xunit.xml'
3/3 Test #11: test_subscription ................***Failed    3.22 sec



The following tests FAILED:
	  8 - test_pub_sub (Failed)
	  9 - test_publisher (Failed)
	 11 - test_subscription (Failed)

[christophebedard]

Do we know roughly when these tests started failing?

[christophebedard]

Ah I see why they're failing. They probably never passed.

The test_tracetools tests in Iron are simply run against the default rmw implementation. Each test is run once, without RMW_IMPLEMENTATION being set explicitly:

ros2_tracing/test_tracetools/CMakeLists.txt

Lines 133 to 157 in b723505

	# Only run tracing tests if instrumentation and tracepoints are included
	if(NOT TRACETOOLS_TRACEPOINTS_EXCLUDED)
	find_package(ament_cmake_pytest REQUIRED)
	set(_test_tracetools_pytest_tests
	test/test_buffer.py
	test/test_executor.py
	test/test_intra.py
	test/test_intra_pub_sub.py
	test/test_lifecycle_node.py
	test/test_node.py
	test/test_pub_sub.py
	test/test_publisher.py
	test/test_service.py
	test/test_subscription.py
	test/test_timer.py
	)
	foreach(_test_path ${_test_tracetools_pytest_tests})
	get_filename_component(_test_name ${_test_path} NAME_WE)
	ament_add_pytest_test(${_test_name} ${_test_path}
	APPEND_ENV PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}
	TIMEOUT 60
	WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
	)
	endforeach()
	endif()

. The other rmw implementations are not fully instrumented in Iron and the default rmw implementation for Iron (and Jazzy) is rmw_fastrtps_cpp, so running tests against the other rmw implementations will fail (except maybe for rmw_cyclonedds_cpp).

The simplest fix for Iron here is to skip the tests if the "current" rmw implementation is not rmw_fastrtps_cpp.

[christophebedard]

See #132 for Iron.

I'll need to do the same for rmw_fastrtps_dynamic_cpp in Jazzy, see Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64#43. rmw_connextdds in Jazzy is instrumented.

[christophebedard]

This was fixed for Iron in #132 and Jazzy in #134, so I'm closing this.