Make more tests robust and independent #546

glatterf42 · 2024-09-26T08:23:26Z

Unfortunately, despite our recent improvements in #543, we still see some apparent flakiness. Its main source seems to be the setup-graphviz action on the windows runners, which sometimes fails like this:

C:\ProgramData\Chocolatey\bin\choco.exe install graphviz
Chocolatey v2.3.0
Installing the following packages:
graphviz
By installing, you accept licenses for the packages.
Downloading package from source 'https://community.chocolatey.org/api/v2/'
[NuGet] Error downloading 'Graphviz.12.1.1 : chocolatey-core.extension [1.3.5, )' from 'https://community.chocolatey.org/api/v2/package/Graphviz/12.1.1'.
[NuGet] Response status code does not indicate success: 503 (Service Unavailable: Back-end server is at capacity).
[NuGet] Error downloading 'Graphviz.12.1.1 : chocolatey-core.extension [1.3.5, )' from 'https://community.chocolatey.org/api/v2/package/Graphviz/12.1.1'.
[NuGet] Response status code does not indicate success: 503 (Service Unavailable: Back-end server is at capacity).
Graphviz not installed. An error occurred during installation:
 Error downloading 'Graphviz.12.1.1 : chocolatey-core.extension [1.3.5, )' from 'https://community.chocolatey.org/api/v2/package/Graphviz/12.1.1'.
Graphviz package files install failed with exit code 1. Performing other installation steps.
The install of Graphviz was NOT successful.
Graphviz not installed. An error occurred during installation:
 Error downloading 'Graphviz.12.1.1 : chocolatey-core.extension [1.3.5, )' from 'https://community.chocolatey.org/api/v2/package/Graphviz/12.1.1'.

Chocolatey installed 0/1 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - Graphviz (exited 1) - Graphviz not installed. An error occurred during installation:
 Error downloading 'Graphviz.12.1.1 : chocolatey-core.extension [1.3.5, )' from 'https://community.chocolatey.org/api/v2/package/Graphviz/12.1.1'.

In addition, there was a new flaky test today:

test_connect_message::AssertionError

      ____________________________ test_connect_message _____________________________
[gw0] win32 -- Python 3.9.13 C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe

capfd = <_pytest.capture.CaptureFixture object at 0x000001C064A052B0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x000001C065035A30>
request = <FixtureRequest for <Function test_connect_message>>

    def test_connect_message(capfd, caplog, request):
        msg = (
            f"connected to database 'jdbc:hsqldb:mem://{request.node.name}_0' "
            "(user: ixmp)..."
        )
    
        ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url=f"jdbc:hsqldb:mem://{request.node.name}_0",
            log_level="INFO",
        )
    
        # Java code via JPype does not log to the standard Python logger
        assert not any(msg in m for m in caplog.messages)
    
        # NB cannot inspect capfd here. Depending on the order in which the tests were run,
        #    a previous run may have left the Java log level higher than INFO, in which
        #    case the Java Platform object would not write to stderr before set_log_level()
        #    in the above call. Try again now that the level is INFO:
        msg = (
            f"connected to database 'jdbc:hsqldb:mem://{request.node.name}_1' "
            "(user: ixmp)..."
        )
        ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url=f"jdbc:hsqldb:mem://{request.node.name}_1",
        )
    
        # Instead, log messages are printed to stdout
        captured = capfd.readouterr()
>       assert msg in captured.out
E       assert "connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)..." in ''
E        +  where '' = CaptureResult(out='', err="2024-09-26 05:51:51,552  INFO at.ac.iiasa.ixmp.Platform:182 - closed the connection to database 'jdbc:hsqldb:mem://test_connect_message_0'\r\n2024-09-26 05:51:51,708  INFO at.ac.iiasa.ixmp.Platform:165 - Welcome to the IX modeling platform!\r\n2024-09-26 05:51:51,708  INFO at.ac.iiasa.ixmp.Platform:166 -  connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)...\r\n").out

ixmp\tests\backend\test_jdbc.py:277: AssertionError
------------------------------ Captured log call ------------------------------
INFO     ixmp.backend.jdbc:jdbc.py:288 launching ixmp.Platform connected to jdbc:hsqldb:mem://test_connect_message_1

Which almost seems like out and err got mixed up in the CaptureResult. If this originates from one test changing the logging level without changing it back again, we should figure out a way to change the level just for this one test, without affecting the whole environment.

The text was updated successfully, but these errors were encountered:

glatterf42 · 2024-09-27T08:00:51Z

test_R_transport::CellTimeoutError

FAILED ixmp/tests/test_tutorials.py::test_R_transport@Darwin3.8.18-1 - nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 30 seconds.
The message was: Cell execution timed out.
Here is a preview of the cell contents:
-------------------
# Load reticulate, used to access the Python API from R
library(reticulate)

# Import ixmp and message_ix, just as in Python
ixmp <- import("ixmp")
-------------------

This reappeared today on macos-13-py3.8, unfortunately.

glatterf42 · 2024-09-30T05:35:58Z

The same error at test_connect_message happened today on Windows-latest-py3.11, though with slightly different error and captured output:

test_connect_message::AssertionError-2

>       assert msg in captured.out
E       assert "connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)..." in ''
E        +  where '' = CaptureResult(out='', err='').out

ixmp\tests\backend\test_jdbc.py:277: AssertionError
 ---------------------------- Captured stdout call -----------------------------
2024-09-30 05:17:54,257  INFO at.ac.iiasa.ixmp.Platform:182 - closed the connection to database 'jdbc:hsqldb:mem://test_connect_message_0'

2024-09-30 05:17:54,445  INFO at.ac.iiasa.ixmp.Platform:165 - Welcome to the IX modeling platform!

2024-09-30 05:17:54,445  INFO at.ac.iiasa.ixmp.Platform:166 -  connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)...

------------------------------ Captured log call ------------------------------
INFO     ixmp.backend.jdbc:jdbc.py:288 launching ixmp.Platform connected to jdbc:hsqldb:mem://test_connect_message_1

khaeru · 2024-10-01T08:15:25Z

Thanks for continuing to track these.

I'll remove the bug label, because I think we should reserve that for cases when we confirm that the code per se does not do what it is expected or advertised to do in user or downstream applications. These flaky tests do pass, meaning the code itself and the tests work as expected/advertised, but only under certain conditions specifically in the GitHub Actions environment do they occasionally fail.

If particular users report failures of the code in real-world usage that correspond to the flaky tests on GHA, then those are definitely bugs and we can open separate issues for each one.

glatterf42 · 2024-10-10T06:53:51Z

test_close_increased_logging::AssertionError

________________________ test_close_increased_logging _________________________
[gw1] win32 -- Python 3.9.13 C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe

test_mp_f = <ixmp.core.platform.Platform object at 0x0000015C13BA03A0>
capfd = <_pytest.capture.CaptureFixture object at 0x0000015C13BA0EB0>

    def test_close_increased_logging(test_mp_f, capfd):
        """Platform.close_db() doesn't throw needless exceptions."""
        # Use the session-scoped fixture to avoid affecting other tests in this file
        mp = test_mp_f
  
        # Close once
        mp.close_db()
  
        # Set higher log level INFO
        level = mp.get_log_level()
        mp.set_log_level(logging.INFO)
  
        # Close again, once already closed
        # With logging.INFO, a message is printed
        mp.close_db()
        captured = capfd.readouterr()
        msg = "Database connection could not be closed or was already closed"
        try:
>           assert msg in captured.out
E           AssertionError: assert 'Database connection could not be closed or was already closed' in ''
E            +  where '' = CaptureResult(out='', err='').out

ixmp\tests\backend\test_jdbc.py:83: AssertionError

glatterf42 · 2024-10-22T06:19:26Z

test_default_version::AssertionError

______________________ TestScenario.test_default_version ______________________
[gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

self = <ixmp.tests.core.test_scenario.TestScenario object at 0x0000023A4553EC60>
mp = <ixmp.core.platform.Platform object at 0x0000023A454B05F0>

    def test_default_version(self, mp):
        scen = ixmp.Scenario(mp, **models["dantzig"])
>       assert scen.version == 2
E       assert 5 == 2
E        +  where 5 = <ixmp.core.scenario.Scenario object at 0x0000023A518462D0>.version

ixmp\tests\core\test_scenario.py:82: AssertionError
---------------------------- Captured stdout setup ----------------------------
--- Warning: The GAMS version [43.4.1] differs from the API version [24.8.3].

--- Warning: The GAMS version [43.4.1] differs from the API version [24.8.3].

test_from_url::AssertionError

_________________________ TestScenario.test_from_url __________________________
[gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

self = <ixmp.tests.core.test_scenario.TestScenario object at 0x0000023A4553EF00>
mp = <ixmp.core.platform.Platform object at 0x0000023A50746FF0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000023A506D43E0>

    def test_from_url(self, mp, caplog):
        url = f"ixmp://{mp.name}/Douglas Adams/Hitchhiker"
  
        # Default version is loaded
        scen, mp = ixmp.Scenario.from_url(url)
>       assert scen.version == 1
E       assert 2 == 1
E        +  where 2 = <ixmp.core.scenario.Scenario object at 0x0000023A506D7BF0>.version

ixmp\tests\core\test_scenario.py:89: AssertionError

test_add_timeseries_with_extra_col[TimeSeries-long]::AssertionError

_____ TestTimeSeries.test_add_timeseries_with_extra_col[TimeSeries-long] ______
[gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

self = <ixmp.tests.core.test_timeseries.TestTimeSeries object at 0x0000023A45580E00>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000023A507EED20>
ts = <ixmp.core.timeseries.TimeSeries object at 0x0000023A507ED190>
format = 'long'

    @pytest.mark.parametrize("format", ["long", "wide"])
    def test_add_timeseries_with_extra_col(self, caplog, ts, format):
        _data = DATA[0].copy()
        _data["climate_model"] = [0, 1]
        data = _data if format == "long" else wide(_data)
  
        # check that extra column wasn't dropped by `wide(_data)
        assert "climate_model" in data.columns
  
        # Data added
        ts.add_timeseries(data)
        # TODO: add check that warning message is displayed
        ts.commit("")
>       assert [
            "Dropped extra column(s) ['climate_model'] from data"
        ] == caplog.messages
E       assert ["Dropped ext...'] from data"] == []
E         
E         Left contains one more item: "Dropped extra column(s) ['climate_model'] from data"
E         
E         Full diff:
E         - []
E         + [
E         +     "Dropped extra column(s) ['climate_model'] from data",
E         + ]

ixmp\tests\core\test_timeseries.py:423: AssertionError

test_add_timeseries_with_extra_col[Scenario-long]::AssertionError
```python ______ TestTimeSeries.test_add_timeseries_with_extra_col[Scenario-long] _______ [gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe
self = <ixmp.tests.core.test_timeseries.TestTimeSeries object at 0x0000023A45580FE0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000023A518F1C70>
ts = <ixmp.core.scenario.Scenario object at 0x0000023A518F1850>, format = 'long'
```
@pytest.mark.parametrize("format", ["long", "wide"])
def test_add_timeseries_with_extra_col(self, caplog, ts, format):
    _data = DATA[0].copy()
    _data["climate_model"] = [0, 1]
    data = _data if format == "long" else wide(_data)

    # check that extra column wasn't dropped by `wide(_data)
    assert "climate_model" in data.columns

    # Data added
    ts.add_timeseries(data)
    # TODO: add check that warning message is displayed
    ts.commit("")
```
```
  assert [
```
```
        "Dropped extra column(s) ['climate_model'] from data"
    ] == caplog.messages
```
E assert ["Dropped ext...'] from data"] == []
E
E Left contains one more item: "Dropped extra column(s) ['climate_model'] from data"
E
E Full diff:
E - []
E + [
E + "Dropped extra column(s) ['climate_model'] from data",
E + ]

ixmp\tests\core\test_timeseries.py:423: AssertionError
```
</details>
```

test_add_timeseries_with_extra_col[Scenario-wide]::AssertionError

______ TestTimeSeries.test_add_timeseries_with_extra_col[Scenario-wide] _______
[gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

self = <ixmp.tests.core.test_timeseries.TestTimeSeries object at 0x0000023A455810A0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000023A518F2B40>
ts = <ixmp.core.scenario.Scenario object at 0x0000023A518F2C60>, format = 'wide'

    @pytest.mark.parametrize("format", ["long", "wide"])
    def test_add_timeseries_with_extra_col(self, caplog, ts, format):
        _data = DATA[0].copy()
        _data["climate_model"] = [0, 1]
        data = _data if format == "long" else wide(_data)
  
        # check that extra column wasn't dropped by `wide(_data)
        assert "climate_model" in data.columns
  
        # Data added
        ts.add_timeseries(data)
        # TODO: add check that warning message is displayed
        ts.commit("")
>       assert [
            "Dropped extra column(s) ['climate_model'] from data"
        ] == caplog.messages
E       assert ["Dropped ext...'] from data"] == []
E         
E         Left contains one more item: "Dropped extra column(s) ['climate_model'] from data"
E         
E         Full diff:
E         - []
E         + [
E         +     "Dropped extra column(s) ['climate_model'] from data",
E         + ]

ixmp\tests\core\test_timeseries.py:423: AssertionError

test_store_ts[attrseries]::IndexError

__________________________ test_store_ts[attrseries] __________________________
[gw1] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

request = <FixtureRequest for <Function test_store_ts[attrseries]>>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000023A518F7FE0>
test_mp = <ixmp.core.platform.Platform object at 0x0000023A454B05F0>

    def test_store_ts(request, caplog, test_mp) -> None:
        # Computer and target scenario
        c = Computer()
   
        # Target scenario
        model_name = __name__
        scenario_name = "test scenario"
        scen = Scenario(test_mp, model_name, scenario_name, version="new")
        scen.commit("Empty scenario")
        c.add("target", scen)
  
        # Add test data to the Computer: a pd.DataFrame
        input_1 = test_data[0].assign(variable="Foo")
        c.add("input 1", input_1)
  
        # A pyam.IamDataFrame
        input_2 = test_data[2050].assign(variable="Bar")
        c.add("input 2", pyam.IamDataFrame(input_2))
  
        # Expected results: same as input, but with the `model` and `scenario` columns
        # filled automatically.
        expected_1 = input_1.assign(model=model_name, scenario=scenario_name)
        expected_2 = input_2.assign(model=model_name, scenario=scenario_name)
  
        # Task to update the scenario with the data
        c.add("test 1", store_ts, "target", "input 1", "input 2")
  
        # Scenario starts empty of time series data
        assert 0 == len(scen.timeseries())
  
        # The computation runs successfully
        c.get("test 1")
  
        # All rows from both inputs are present
        assert len(input_1) + len(input_2) == len(scen.timeseries())
  
        # Input is stored exactly
        assert_frame_equal(expected_1, scen.timeseries(variable="Foo"))
        assert_frame_equal(expected_2, scen.timeseries(variable="Bar"))
  
        # Data with an unregistered region name
        c.add("input 3", test_data[0].assign(variable="Foo", region="Moon"))
        c.add("test 2", store_ts, "target", "input 3")
  
        # Succeeds with default strict=False
        caplog.clear()
        c.get("test 2")
  
        # A message is logged
>       r = caplog.record_tuples[-1]
E       IndexError: list index out of range

ixmp\tests\report\test_operator.py:213: IndexError

glatterf42 · 2024-11-04T06:15:16Z

test_platform_copy::AssertionError

  ______________________________ test_platform_copy ______________________________
[gw3] darwin -- Python 3.9.20 /Users/runner/hostedtoolcache/Python/3.9.20/x64/bin/python

ixmp_cli = <ixmp.testing.ixmp_cli.<locals>.Runner object at 0x15e23b640>
tmp_path = PosixPath('/private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0')
request = <FixtureRequest for <Function test_platform_copy>>

    def test_platform_copy(ixmp_cli, tmp_path, request):
        """Test 'platform' command."""
        test_specific_name = request.node.name
  
        def call(*args, exit_0=True):
            result = ixmp_cli.invoke(["platform"] + list(map(str, args)))
            assert not exit_0 or result.exit_code == 0, result.output
            return result
  
        # Add some temporary platform configuration
        call(
            "add",
            f"p1-{test_specific_name}",
            "jdbc",
            "oracle",
            "HOSTNAME",
            "USER",
            "PASSWORD",
        )
        call(
            "add",
            f"p2-{test_specific_name}",
            "jdbc",
            "hsqldb",
            tmp_path.joinpath(f"p2-{test_specific_name}"),
        )
        # Force connection to p2 so that files are created
        ixmp_cli.invoke([f"--platform=p2-{test_specific_name}", "list"])
  
        # Dry-run produces expected output
        r = call("copy", f"p2-{test_specific_name}", f"p3-{test_specific_name}")
        assert re.search(
            f"Copy .*p2-{test_specific_name}.script → .*p3-{test_specific_name}.script",
            r.output,
        )
        with pytest.raises(ValueError):
            # New platform configuration is not saved
            ixmp.config.get_platform_info(f"p3-{test_specific_name}")
  
        # --go actually copies files, saves new platform config
>       r = call("copy", "--go", f"p2-{test_specific_name}", f"p3-{test_specific_name}")

ixmp/tests/test_cli.py:208: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

exit_0 = True
args = ('copy', '--go', 'p2-test_platform_copy', 'p3-test_platform_copy')
result = <Result FileNotFoundError(2, 'No such file or directory')>
@py_assert1 = ['not True', "1\n{1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code\n} == 0"]

    def call(*args, exit_0=True):
        result = ixmp_cli.invoke(["platform"] + list(map(str, args)))
>       assert not exit_0 or result.exit_code == 0, result.output
E       AssertionError: Copy /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p2-test_platform_copy.script.new → /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p3-test_platform_copy.new
E         Copy /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p2-test_platform_copy.script → /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p3-test_platform_copy.script
E         Copy /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p2-test_platform_copy.properties.new → /private/var/folders/nj/wh528zms06j9t8y7bmlvpmjm0000gn/T/pytest-of-runner/pytest-0/popen-gw3/test_platform_copy0/p3-test_platform_copy.new
E         
E       assert (not True or 1 == 0)
E        +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code

ixmp/tests/test_cli.py:174: AssertionError

glatterf42 · 2024-11-06T07:52:23Z

setup of test_check_single_model_access and test_check_multi_model_access::ConnectionRefusedError

_______________ ERROR at setup of test_check_single_model_access _______________
[gw3] darwin -- Python 3.9.20 /Users/runner/hostedtoolcache/Python/3.9.20/x64/bin/python

server = None

    @pytest.fixture(scope="function")
    def mock(server):
        # Create the mock server
>       httpmock = HTTPMock("localhost", 8000)

ixmp/tests/test_access.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/http.py:49: in __init__
    super(HTTPMock, self).__init__(host, port, timeout, name)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:102: in __init__
    self.pretender_details = self._request_mock_access()
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:152: in _request_mock_access
    response, data = self.boss_access.http('POST',
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:39: in http
    response = self._get_response(method, *args, **kwargs)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:27: in _get_response
    self.connection.request(method=method, *args, **kwargs)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1285: in request
    self._send_request(method, url, body, headers, encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1331: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1280: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1040: in _send_output
    self.send(msg)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:980: in send
    self.connect()
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:946: in connect
    self.sock = self._create_connection(
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/socket.py:856: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('localhost', 8000), timeout = <object object at 0x1097e53a0>
source_address = None

    def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                          source_address=None):
        """Connect to *address* and return the socket object.
  
        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        A host of '' or port 0 tells the OS to use the default.
        """
  
        host, port = address
        err = None
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket(af, socktype, proto)
                if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               ConnectionRefusedError: [Errno 61] Connection refused

../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/socket.py:844: ConnectionRefusedError
---------------------------- Captured stderr setup -----------------------------
2024-11-06 05:10:16,187 DEBUG:pretenders.server.server:44: Setting pretender timeout: 120
2024-11-06 05:10:16,187 DEBUG:pretenders.server.server:16: Starting maintainer process
_______________ ERROR at setup of test_check_multi_model_access ________________
[gw3] darwin -- Python 3.9.20 /Users/runner/hostedtoolcache/Python/3.9.20/x64/bin/python

server = None

    @pytest.fixture(scope="function")
    def mock(server):
        # Create the mock server
>       httpmock = HTTPMock("localhost", 8000)

ixmp/tests/test_access.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/http.py:49: in __init__
    super(HTTPMock, self).__init__(host, port, timeout, name)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:102: in __init__
    self.pretender_details = self._request_mock_access()
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:152: in _request_mock_access
    response, data = self.boss_access.http('POST',
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:39: in http
    response = self._get_response(method, *args, **kwargs)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/pretenders/client/__init__.py:27: in _get_response
    self.connection.request(method=method, *args, **kwargs)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1285: in request
    self._send_request(method, url, body, headers, encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1331: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1280: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:1040: in _send_output
    self.send(msg)
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:980: in send
    self.connect()
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/http/client.py:946: in connect
    self.sock = self._create_connection(
../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/socket.py:856: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('localhost', 8000), timeout = <object object at 0x1097e53a0>
source_address = None

    def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                          source_address=None):
        """Connect to *address* and return the socket object.
  
        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        A host of '' or port 0 tells the OS to use the default.
        """
  
        host, port = address
        err = None
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket(af, socktype, proto)
                if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               ConnectionRefusedError: [Errno 61] Connection refused

../../../hostedtoolcache/Python/3.9.20/x64/lib/python3.9/socket.py:844: ConnectionRefusedError

glatterf42 · 2024-11-18T07:24:16Z

test_del_ts::AssertionError

_________________________________ test_del_ts _________________________________
[gw0] win32 -- Python 3.12.7 C:\hostedtoolcache\windows\Python\3.12.7\x64\python.exe

request = <FixtureRequest for <Function test_del_ts>>

    @pytest.mark.flaky(
        reruns=5,
        rerun_delay=2,
        condition="GITHUB_ACTIONS" in os.environ and platform.system() == "Windows",
        reason="Flaky; see iiasa/ixmp#543",
    )
    def test_del_ts(request):
        mp = ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url="jdbc:hsqldb:mem:test_del_ts",
        )
  
        backend: ixmp.backend.jdbc.JDBCBackend = mp._backend  # type: ignore
  
        # Number of Java objects referenced by the JDBCBackend
        N_obj = len(backend.jindex)
  
        # Create a list of some Scenario objects
        N = 8
        scenarios = [make_dantzig(mp, request=request)]
        for i in range(1, N):
            scenarios.append(scenarios[0].clone(scenario=f"{request.node.name} clone {i}"))
  
        # Number of referenced objects has increased by 8
        assert len(backend.jindex) == N_obj + N
  
        # Pop and free the objects
        for i in range(N):
            s = scenarios.pop(0)
  
            message = "\n".join(map(str, gc.get_referrers(s)))
  
            # The variable 's' is the only reference to this Scenario object
            assert 1 == getrefcount(s) - 1, (message, gc.garbage)
  
            # ID of the Scenario object
            s_id = id(s)
  
            # Underlying Java object
            s_jobj = backend.jindex[s]
  
            # Now delete the Scenario object
            # del s # should work, but doesn't always resolve to s.__del__()
            backend.del_ts(s)
  
            # Number of referenced objects decreases by 1
>           assert len(backend.jindex) == N_obj + N - (i + 1)
E           assert 7 == ((8 + 8) - (0 + 1))
E            +  where 7 = len(<WeakKeyDictionary at 0x1c6f48ff410>)
E            +    where <WeakKeyDictionary at 0x1c6f48ff410> = <ixmp.backend.jdbc.JDBCBackend object at 0x000001C69DF1D760>.jindex

ixmp\tests\backend\test_jdbc.py:431: AssertionError

glatterf42 · 2024-11-25T07:33:33Z

test_del_ts::AssertionError 2

_________________________________ test_del_ts _________________________________
[gw0] win32 -- Python 3.13.0 C:\hostedtoolcache\windows\Python\3.13.0\x64\python.exe

request = <FixtureRequest for <Function test_del_ts>>

    @pytest.mark.flaky(
        reruns=5,
        rerun_delay=2,
        condition="GITHUB_ACTIONS" in os.environ and platform.system() == "Windows",
        reason="Flaky; see iiasa/ixmp#543",
    )
    def test_del_ts(request):
        mp = ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url="jdbc:hsqldb:mem:test_del_ts",
        )
  
        backend: ixmp.backend.jdbc.JDBCBackend = mp._backend  # type: ignore
  
        # Number of Java objects referenced by the JDBCBackend
        N_obj = len(backend.jindex)
  
        # Create a list of some Scenario objects
        N = 8
        scenarios = [make_dantzig(mp, request=request)]
        for i in range(1, N):
            scenarios.append(scenarios[0].clone(scenario=f"{request.node.name} clone {i}"))
  
        # Number of referenced objects has increased by 8
        assert len(backend.jindex) == N_obj + N
  
        # Pop and free the objects
        for i in range(N):
            s = scenarios.pop(0)
  
            message = "\n".join(map(str, gc.get_referrers(s)))
  
            # The variable 's' is the only reference to this Scenario object
>           assert 1 == getrefcount(s) - 1, (message, gc.garbage)
E           AssertionError: ("<frame at 0x00000138A94FA0A0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\backend\\\\jdbc.py', line 734, code check_out>
E             <frame at 0x00000138A94CEC50, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\backend\\\\base.py', line 44, code __call__>
E             <frame at 0x00000138A94FA180, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\timeseries.py', line 109, code _backend>
E             <frame at 0x00000138A9442B50, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\timeseries.py', line 186, code check_out>
E             <frame at 0x00000138A9442740, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\scenario.py', line 110, code check_out>
E             <frame at 0x00000138A94428E0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\util\\\\__init__.py', line 254, code maybe_check_out>
E             <bound method Scenario.list_items of <ixmp.core.scenario.Scenario object at 0x00000138A9477EE0>>
E             {'__self__': <ixmp.core.scenario.Scenario object at 0x00000138A9477EE0>}
E             <frame at 0x00000138A94984C0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\model\\\\base.py', line 186, code initialize_items>
E             <frame at 0x00000138A94F9FC0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\backend\\\\jdbc.py', line 734, code check_out>
E             <frame at 0x00000138A94CEA70, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\backend\\\\base.py', line 44, code __call__>
E             <frame at 0x00000138A94F9EE0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\timeseries.py', line 109, code _backend>
E             <frame at 0x00000138A9440D40, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\timeseries.py', line 186, code check_out>
E             <frame at 0x00000138A9440E10, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\scenario.py', line 110, code check_out>
E             <frame at 0x00000138A9441E50, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\util\\\\__init__.py', line 254, code maybe_check_out>
E             <frame at 0x00000138A94CEB60, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\model\\\\dantzig.py', line 108, code initialize>
E             <frame at 0x0000013887A2D210, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\core\\\\scenario.py', line 89, code __init__>
E             <frame at 0x00000138A95387B0, file 'D:\\\\a\\\\ixmp\\\\ixmp\\\\ixmp\\\\testing\\\\data.py', line 250, code make_dantzig>", [])
E           assert 1 == (20 - 1)
E            +  where 20 = getrefcount(<ixmp.core.scenario.Scenario object at 0x00000138A9477EE0>)

ixmp\tests\backend\test_jdbc.py:417: AssertionError

glatterf42 · 2024-11-25T07:38:17Z

test_del_ts::AssertionError 3

_________________________________ test_del_ts _________________________________
[gw1] win32 -- Python 3.13.0 C:\hostedtoolcache\windows\Python\3.13.0\x64\python.exe

request = <FixtureRequest for <Function test_del_ts>>

    @pytest.mark.flaky(
        reruns=5,
        rerun_delay=2,
        condition="GITHUB_ACTIONS" in os.environ and platform.system() == "Windows",
        reason="Flaky; see iiasa/ixmp#543",
    )
    def test_del_ts(request):
        mp = ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url="jdbc:hsqldb:mem:test_del_ts",
        )
    
        backend: ixmp.backend.jdbc.JDBCBackend = mp._backend  # type: ignore
    
        # Number of Java objects referenced by the JDBCBackend
        N_obj = len(backend.jindex)
    
        # Create a list of some Scenario objects
        N = 8
        scenarios = [make_dantzig(mp, request=request)]
        for i in range(1, N):
            scenarios.append(scenarios[0].clone(scenario=f"{request.node.name} clone {i}"))
    
        # Number of referenced objects has increased by 8
>       assert len(backend.jindex) == N_obj + N
E       assert 8 == (16 + 8)
E        +  where 8 = len(<WeakKeyDictionary at 0x2bebfe587d0>)
E        +    where <WeakKeyDictionary at 0x2bebfe587d0> = <ixmp.backend.jdbc.JDBCBackend object at 0x000002BEC5F46E10>.jindex

ixmp\tests\backend\test_jdbc.py:408: AssertionError

glatterf42 added bug ci Continuous integration labels Sep 26, 2024

khaeru removed the bug label Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make more tests robust and independent #546

Make more tests robust and independent #546

glatterf42 commented Sep 26, 2024 •

edited

Loading

glatterf42 commented Sep 27, 2024 •

edited

Loading

glatterf42 commented Sep 30, 2024 •

edited

Loading

khaeru commented Oct 1, 2024

glatterf42 commented Oct 10, 2024 •

edited

Loading

glatterf42 commented Oct 22, 2024 •

edited

Loading

glatterf42 commented Nov 4, 2024 •

edited

Loading

glatterf42 commented Nov 6, 2024 •

edited

Loading

glatterf42 commented Nov 18, 2024 •

edited

Loading

glatterf42 commented Nov 25, 2024

glatterf42 commented Nov 25, 2024

Make more tests robust and independent #546

Make more tests robust and independent #546

Comments

glatterf42 commented Sep 26, 2024 • edited Loading

glatterf42 commented Sep 27, 2024 • edited Loading

glatterf42 commented Sep 30, 2024 • edited Loading

khaeru commented Oct 1, 2024

glatterf42 commented Oct 10, 2024 • edited Loading

glatterf42 commented Oct 22, 2024 • edited Loading

glatterf42 commented Nov 4, 2024 • edited Loading

glatterf42 commented Nov 6, 2024 • edited Loading

glatterf42 commented Nov 18, 2024 • edited Loading

glatterf42 commented Nov 25, 2024

glatterf42 commented Nov 25, 2024

glatterf42 commented Sep 26, 2024 •

edited

Loading

glatterf42 commented Sep 27, 2024 •

edited

Loading

glatterf42 commented Sep 30, 2024 •

edited

Loading

glatterf42 commented Oct 10, 2024 •

edited

Loading

glatterf42 commented Oct 22, 2024 •

edited

Loading

glatterf42 commented Nov 4, 2024 •

edited

Loading

glatterf42 commented Nov 6, 2024 •

edited

Loading

glatterf42 commented Nov 18, 2024 •

edited

Loading