Modernize testing to pytest #1312

jmeyers314 · 2024-09-17T05:54:06Z

pytest 8 broke our tests by changing the behavior of the setup() function inside test files. This function used to be run ahead of each individual test to do things like set up common variables. I've rewritten the tests to instead use pytest fixtures, which can accomplish the same goal.

I also figured out how to forward arbitrary arguments given to python test_blah.py --option --option2 -o3 ... to the pytest runner, so you can now do things like

python test_optics.py --durations=0

and the optics test will run, (including the slow run-only-in-main variations), and also output the durations via the pytest-durations plugin. So the main difference now between python test_blah.py and pytest test_blah.py is just whether or not the slow variations get run by default, and even this can be overridden with the --run_slow argument.

rmjarvis · 2024-09-17T18:43:33Z

tests/time_all_tests

-#time run_all_tests | grep time | sort -n -k 5 | gawk '{print; n++; sum+=$5} END { print "total time for",n,"tests = ",sum }'
-
-# Another option...
-time nosetests -s | grep time | sort -n -k 5 | gawk '{print; n++; sum+=$5} END { print "total time for",n,"tests = ",sum }'


I use this occasionally to look for tests that are kind of slow. Can we just switch it from nosetests to pytest. I think the rest can be the same.

Why not pytest --durations=0 ?

OK. That works. I didn't know about that feature. I'll try to remember that next time I want to do this.

rmjarvis · 2024-09-17T18:48:03Z

tests/galsim_test_helpers.py

-        if args.k is None or args.k in testfn.__name__:
-            testfn()
-
+    pytest.main(pytest_args)


Oh, actually, I don't like this. When developing code, I want the tests to be run as regular functions, not via pytest. I very much don't like the pytest reporting when I'm in that mode. Can we revert this bit here, so python test_blah.py runs the normal way?

Or does that mess up all the run_slow stuff? I guess it's hard to determine here whether a test function needs the extra argument or not.

It would certainly complicate the run_slow stuff. I think I might be able to work around that. Can I ask though, what is it about the pytest reporting you don't like? Maybe that's easier to address than refactoring the run_slow stuff.

Multiple things. First, it puts the stdout last, which means the error is often off the top of the screen. It also quotes the whole test function, which is rather excessive at times. And often thinks the error is in some contextlib, although it also gives the right place as well, albeit even farther up the scrollback.

Compare: (I added an intentional error somewhere in the config/value.py:

$ python test_config_value.py ======================================= test session starts ======================================== platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0 rootdir: /Users/Mike configfile: pytest.ini plugins: nbval-0.10.0, anyio-4.4.0, timeout-2.3.1, xdist-3.6.1 collected 11 items test_config_value.py ........F.. [100%] ============================================= FAILURES ============================================= ____________________________________________ test_eval _____________________________________________ @timer def test_eval(): """Test various ways that we evaluate a string as a function or value """ config = { # The basic calculation 'eval1' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * 1.8**2)' }, # Different ways to get variables 'eval2' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * x**2)', 'fx' : 1.8 }, 'eval3' : { 'type' : 'Eval', 'str' : 'np.exp(-y**2 / two) if maybe else 0.' }, # Make sure to use all valid letter prefixes here... 'eval_variables' : { 'fy' : 1.8, 'bmaybe' : True, 'itwo' : 2, 'shalf' : '0.5', 'atheta' : 1.8 * galsim.radians, 'ppos' : galsim.PositionD(1.8,0), 'ccoord' : galsim.CelestialCoord(1.8*galsim.radians,0*galsim.radians), 'gshear' : galsim.Shear(g1=0.5, g2=0), 'ttable' : galsim.LookupTable([0,1,2,3], [0,1.8,1.8,0], interpolant='linear'), 'ddct' : { 'a' : 1, 'b' : 2 }, 'llst' : [ 1.5, 1.0, 0.5 ], 'xlit_two' : 2, 'qlength' : 1.8*u.m, 'upint' : u.imperial.pint, }, # Shorthand notation with $ 'eval4' : '$np.exp(-0.5 * y**2)', # math and numpy should also work 'eval5' : '$numpy.exp(-0.5 * y**2)', 'eval6' : '$math.exp(-0.5 * y**2)', # Use variables that are automatically defined 'eval7' : '$np.exp(-0.5 * image_pos.x**2)', 'eval8' : '$np.exp(-0.5 * world_pos.y**2)', 'eval9' : '$np.exp(-0.5 * pixel_scale**2)', 'eval10' : '$np.exp(-0.5 * (image_xsize / 100.)**2)', 'eval11' : '$np.exp(-0.5 * (image_ysize / 200.)**2)', 'eval12' : '$np.exp(-0.5 * (stamp_xsize / 20.)**2)', 'eval13' : '$np.exp(-0.5 * (stamp_ysize / 20.)**2)', 'eval14' : '$np.exp(-0.5 * (image_bounds.xmax / 100.)**2)', 'eval15' : '$np.exp(-0.5 * ((image_center.y-0.5) / 100.)**2)', 'eval16' : '$np.exp(-0.5 * wcs.scale**2)', # Shorthand notation with @ 'psf' : { 'type' : 'Gaussian', 'sigma' : 1.8 }, 'eval17' : '$np.exp([email protected]**2 / @eval_variables.itwo)', # A couple more to cover the other various letter prefixes. 'eval18' : { 'type' : 'Eval', 'str' : 'np.exp(-eval(half) * theta.rad**lit_two)' }, 'eval19' : { 'type' : 'Eval', 'str' : 'np.exp(-shear.g1 * pos.x * coord.ra.rad)' }, 'eval20' : { 'type' : 'Eval', 'str' : 'np.exp(-lst[2] * table(1.5)**dct["b"])' }, # Can access the input object as a current. 'eval21' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * ((@input.catalog).nobjects*0.6)**2)' }, 'eval22' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * (@input.dict["f"]*18)**2)' }, 'eval23' : { 'type' : 'Eval', 'str' : 'np.exp(-pint/u.imperial.quart * length.to_value(u.m)**2)' }, # Some that raise exceptions 'bad1' : { 'type' : 'Eval', 'str' : 'npexp(-0.5)' }, 'bad2' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * x**2)', 'x' : 1.8 }, 'bad3' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * x**2)', 'wx' : 1.8 }, 'bad4' : { 'type' : 'Eval', 'str' : 'np.exp(-0.5 * q**2)', 'fx' : 1.8 }, 'bad5' : { 'type' : 'Eval', 'eval_str' : 'np.exp(-0.5 * x**2)', 'fx' : 1.8 }, # Check that a list can be made using Eval 'list0' : [0,1,2,3,4,5], 'list1' : '$np.arange(6)', 'list2' : (0,1,2,3,4,5), 'list3' : '$(0,1,2,3,4,5)', # Check that a dict can be made using Eval 'dict0' : {0:'h', 1:'e', 2:'l', 3:'l', 4:'o'}, 'dict1' : dict(enumerate("hello")), 'dict2' : '$dict(enumerate("hello"))', 'dict3' : '${ k:v for k,v in zip(np.arange(5), "hello") }', # These would be set by config in real runs, but just add them here for the tests. 'image_pos' : galsim.PositionD(1.8,13), 'world_pos' : galsim.PositionD(7.2,1.8), 'uv_pos' : galsim.PositionD(7.2,1.8), 'pixel_scale' : 1.8, 'image_xsize' : 180, 'image_ysize' : 360, 'stamp_xsize' : 36, 'stamp_ysize' : 36, 'image_bounds' : galsim.BoundsI(1,180,1,360), 'image_center' : galsim.PositionD(90.5, 180.5), 'wcs' : galsim.PixelScale(1.8), 'input' : { 'catalog' : { 'dir' : 'config_input', 'file_name' : 'catalog.txt' }, 'dict' : { 'dir' : 'config_input', 'file_name' : 'dict.yaml' }, }, } galsim.config.ProcessInput(config) true_val = np.exp(-0.5 * 1.8**2) # All of these should equal this value. for i in range(1,24): test_val = galsim.config.ParseValue(config, 'eval%d'%i, config, float)[0] print('i = ',i, 'val = ',test_val,true_val) np.testing.assert_almost_equal(test_val, true_val) # Doing it again uses saved _value and _fn galsim.config.RemoveCurrent(config) for i in range(1,24): test_val = galsim.config.ParseValue(config, 'eval%d'%i, config, float)[0] print('i = ',i, 'val = ',test_val,true_val) np.testing.assert_almost_equal(test_val, true_val) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'bad1',config, float) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'bad2',config, float) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'bad3',config, float) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'bad4',config, float) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'bad5',config, float) config['eval_variables'] = 'itwo' config = galsim.config.CleanConfig(config) with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'eval3',config, float) del config['eval_variables'] with assert_raises(galsim.GalSimConfigError): galsim.config.ParseValue(config,'eval3',config, float) # Check ways of making a list for i in range(4): test_list = galsim.config.ParseValue(config, 'list%d'%i, config, list)[0] print(test_list) np.testing.assert_array_equal(test_list, np.arange(6)) # Check ways of making a dict for i in range(4): test_dict = galsim.config.ParseValue(config, 'dict%d'%i, config, dict)[0] print(test_dict) > np.testing.assert_array_equal(test_dict, dict(enumerate('hello'))) test_config_value.py:1800: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = (<built-in function eq>, 'h', {0: 'h', 1: 'e', 2: 'l', 3: 'l', ...}) kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True} @wraps(func) def inner(*args, **kwds): with self._recreate_cm(): > return func(*args, **kwds) E AssertionError: E Arrays are not equal E E Mismatched elements: 1 / 1 (100%) E x: array('h', dtype='<U1') E y: array({0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o'}, dtype=object) ../../miniforge3/envs/py3.10/lib/python3.10/contextlib.py:79: AssertionError --------------------------------------- Captured stdout call --------------------------------------- i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] h ===================================== short test summary info ====================================== FAILED test_config_value.py::test_eval - AssertionError: =================================== 1 failed, 10 passed in 0.37s ===================================

vs.

$ python test_config_value.py [.. snip output from other functions than the one that failed ...] i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] h Traceback (most recent call last): File "/Users/Mike/GalSim/tests/test_config_value.py", line 2029, in <module> testfn() File "/Users/Mike/GalSim/galsim/utilities.py", line 1756, in f2 result = f(*args, **kwargs) File "/Users/Mike/GalSim/tests/test_config_value.py", line 1800, in test_eval np.testing.assert_array_equal(test_dict, dict(enumerate('hello'))) File "/Users/Mike/miniforge3/envs/py3.10/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 920, in assert_array_equal assert_array_compare(operator.__eq__, x, y, err_msg=err_msg, File "/Users/Mike/miniforge3/envs/py3.10/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/Users/Mike/miniforge3/envs/py3.10/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal Mismatched elements: 1 / 1 (100%) x: array('h', dtype='<U1') y: array({0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o'}, dtype=object)

In the latter case, the error and the line number I need to know are right there on the screen. With pytest, I have to scroll up to see what the error was. When I see the error, it's giving me the wrong function. And I have to scroll farther up to get the correct line number.

BTW, I think if you just make the run_slow arguments have a default value of True, the old way of running the functions would work fine. Then pytest would still have a default run_slow=False and override them when it runs.

Adding --tb=short -s isn't perfect, but certainly improves things. (Possibly a bit too short of a traceback unfortunately). Curious what you think:

(lsst-scipipe-9.0.0) [jmeyers3@PC102602 src/GalSim/tests (pytest|✚1)]$ python test_config_value.py --tb=short -s ========================================================================== test session starts =========================================================================== platform darwin -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 rootdir: /Users/jmeyers3/src/GalSim configfile: pyproject.toml plugins: cov-5.0.0, time-machine-2.15.0, vcr-1.0.2, doctestplus-1.2.1, flake8-0.0.0, typeguard-4.3.0, anyio-4.4.0, session2file-0.1.11, openfiles-0.5.0, subtests-0.13.1, asdf-3.4.0, xdist-3.6.1 collected 11 items test_config_value.py weak lensing mag = 1.9035493122199765 strong lensing mag = 249.37404979716405 GetRNG for obj_num No index_key_rng. Use base[rng] image 5: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=21, center=galsim.PositionD(0.0,0.0), interpolant=linear, variance=None) obj 4: Warning: NFWHalo mu = 249.374050 means strong lensing. Using mu = 25.000000 obj 4: NFWHalo mu = 25.0 very strong lensing mag = -163.63184593277163 GetRNG for obj_num No index_key_rng. Use base[rng] image 5: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=21, center=galsim.PositionD(0.0,0.0), interpolant=linear, variance=None) obj 4: Warning: NFWHalo mu = -163.631846 means strong lensing. Using mu = 3000.000000 obj 4: NFWHalo mu = 3000.0 ps mag = 1.109567906785996 strong lensing mag = -4.335136823543759 file 5: Start ProcessInput file 5: Process input key catalog file 5: Build input type catalog file 5: catalog kwargs = {'dir': 'config_input', 'file_name': 'catalog.txt'} file 5: Built input object catalog 0 file 5: file_name = catalog.txt Input catalog has 3 objects file 5: Cleared current vals for items with type Catalog file 5: Build input type catalog file 5: catalog kwargs = {'dir': 'config_input', 'file_name': 'catalog.fits'} file 5: Built input object catalog 1 file 5: file_name = catalog.fits Input catalog has 3 objects file 5: Cleared current vals for items with type Catalog file 5: Process input key dict file 5: Build input type dict file 5: dict kwargs = {'dir': 'config_input', 'file_name': 'dict.p'} file 5: Built input object dict 0 file 5: file_name = dict.p file 5: Cleared current vals for items with type Dict file 5: Build input type dict file 5: dict kwargs = {'dir': 'config_input', 'file_name': 'dict.json'} file 5: Built input object dict 1 file 5: file_name = dict.json file 5: Cleared current vals for items with type Dict file 5: Build input type dict file 5: dict kwargs = {'dir': 'config_input', 'file_name': 'dict.yaml'} file 5: Built input object dict 2 file 5: file_name = dict.yaml file 5: Cleared current vals for items with type Dict file 5: Process input key nfw_halo file 5: Build input type nfw_halo file 5: nfw_halo kwargs = {'conc': 4.0, 'mass': 100000000000000.0, 'redshift': 0.3} file 5: Built input object nfw_halo 0 file 5: Cleared current vals for items with type NFWHaloShear file 5: Cleared current vals for items with type NFWHaloMagnification file 5: Process input key power_spectrum file 5: Build input type power_spectrum file 5: power_spectrum kwargs = {'e_power_function': '2000 * np.exp(-k**0.2)'} file 5: Built input object power_spectrum 0 file 5: Cleared current vals for items with type PowerSpectrumShear file 5: Cleared current vals for items with type PowerSpectrumMagnification file 5: Process input key fits_header file 5: Build input type fits_header file 5: fits_header kwargs = {'dir': 'fits_files', 'file_name': 'tpv.fits'} file 5: Built input object fits_header 0 file 5: file_name = tpv.fits file 5: Cleared current vals for items with type FitsHeader GetRNG for obj_num No index_key_rng. Use base[rng] image 5: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=21, center=galsim.PositionD(0.0,0.0), interpolant=linear, variance=None) obj 4: Warning: PowerSpectrum mu = -4.335137 means strong lensing. Using mu=25.000000 obj 4: PowerSpectrum mu = 25.0 GetRNG for obj_num No index_key_rng. Use base[rng] image 5: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=21, center=galsim.PositionD(0.0,0.0), interpolant=linear, variance=None) obj 4: Warning: PowerSpectrum mu = 26.746296 means strong lensing. Using mu=25.000000 obj 4: PowerSpectrum mu = 25.0 GetRNG for obj_num No index_key_rng. Use base[rng] image 5: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=21, center=galsim.PositionD(0.0,0.0), interpolant=linear, variance=None) obj 4: Extrapolating beyond input range. galsim.PositionD(x=1000.0, y=2000.0) not in galsim.BoundsD(xmin=-100.00000000000011, xmax=100.00000000000011, ymin=-100.00000000000011, ymax=100.00000000000011) obj 4: PowerSpectrum mu = 1.0 time for test_float_value = 0.20 .time for test_int_value = 0.01 .time for test_bool_value = 0.01 .time for test_str_value = 0.01 .time for test_angle_value = 0.01 .nfw1a = galsim.Shear(g1=0.03270863078408739,g2=-0.11214387697401387) nfw1b = (0.03270863078408739, -0.11214387697401387) strong lensing shear = (1.148773104222187, -1.5316974722962495) GetRNG for obj_num No index_key_rng. Use base[rng] No base['rng'] available for PowerSpectrum. Using /dev/urandom. image 0: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=40.0, center=galsim.PositionD(5.0,5.0), interpolant=linear, variance=None) obj 4: Warning: NFWHalo shear (g1=1.148773, g2=-1.531697) is invalid. Using shear = 0. obj 4: NFWHalo shear = galsim.Shear(g1=0.0,g2=0.0) ps shear= (0.08539587992379115, -0.02000648400195712) strong lensing shear = (-0.8944456487409024, -0.5208919551577306) ps shear= (-0.8944456487409024, -0.5208919551577306) GetRNG for obj_num No index_key_rng. Use base[rng] image 0: PowerSpectrum buildGrid(grid_spacing=10.0, ngrid=40.0, center=galsim.PositionD(5.0,5.0), interpolant=linear, variance=None) obj 4: Extrapolating beyond input range. galsim.PositionD(x=1000.0, y=2000.0) not in galsim.BoundsD(xmin=-190.00000000000023, xmax=200.00000000000023, ymin=-190.00000000000023, ymax=200.00000000000023) obj 4: PowerSpectrum shear = galsim.Shear(g1=0.0,g2=0.0) time for test_shear_value = 0.01 .time for test_pos_value = 0.00 .time for test_table_value = 0.00 .i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 i = 1 val = 0.19789869908361465 0.19789869908361465 i = 2 val = 0.19789869908361465 0.19789869908361465 i = 3 val = 0.19789869908361465 0.19789869908361465 i = 4 val = 0.19789869908361465 0.19789869908361465 i = 5 val = 0.19789869908361465 0.19789869908361465 i = 6 val = 0.19789869908361465 0.19789869908361465 i = 7 val = 0.19789869908361465 0.19789869908361465 i = 8 val = 0.19789869908361465 0.19789869908361465 i = 9 val = 0.19789869908361465 0.19789869908361465 i = 10 val = 0.19789869908361465 0.19789869908361465 i = 11 val = 0.19789869908361465 0.19789869908361465 i = 12 val = 0.19789869908361465 0.19789869908361465 i = 13 val = 0.19789869908361465 0.19789869908361465 i = 14 val = 0.19789869908361465 0.19789869908361465 i = 15 val = 0.19789869908361465 0.19789869908361465 i = 16 val = 0.19789869908361465 0.19789869908361465 i = 17 val = 0.19789869908361465 0.19789869908361465 i = 18 val = 0.19789869908361465 0.19789869908361465 i = 19 val = 0.19789869908361465 0.19789869908361465 i = 20 val = 0.19789869908361465 0.19789869908361465 i = 21 val = 0.19789869908361474 0.19789869908361465 i = 22 val = 0.19789869908361465 0.19789869908361465 i = 23 val = 0.1978986990836147 0.19789869908361465 [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5] {0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o'} F.. ================================================================================ FAILURES ================================================================================ _______________________________________________________________________________ test_eval ________________________________________________________________________________ test_config_value.py:1800: in test_eval np.testing.assert_array_equal(test_dict, dict(enumerate('hell'))) ../../lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/contextlib.py:81: in inner return func(*args, **kwds) E AssertionError: E Arrays are not equal E E Mismatched elements: 1 / 1 (100%) E x: array({0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o'}, dtype=object) E y: array({0: 'h', 1: 'e', 2: 'l', 3: 'l'}, dtype=object) ============================================================================ warnings summary ============================================================================ ../../lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/_pytest/config/__init__.py:1273 /Users/jmeyers3/src/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/_pytest/config/__init__.py:1273: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: asdf self._mark_plugins_for_rewrite(hook) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================================================================== short test summary info ========================================================================= FAILED test_config_value.py::test_eval - AssertionError: ================================================================ 1 failed, 10 passed, 1 warning in 0.34s =================================================================

I miss nosetests... :) But sure, I can probably get used to this. Can we make these options the default somehow when running via python?

Thanks. Making these the default is what that last commit was trying to do. I'm not sure now what happens when there's more than one failure though... Looking into adding maxfail=1 now...

Here's what 2 failures looks like. I don't really mind this output, but let me know if you'd like to add --maxfail=1, which would halt at the first error.

(lsst-scipipe-9.0.0) [jmeyers3@PC102602 src/GalSim/tests (pytest|✚1)]$ python test_airy.py ======================================================== test session starts ========================================================= platform darwin -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 rootdir: /Users/jmeyers3/src/GalSim configfile: pyproject.toml plugins: cov-5.0.0, time-machine-2.15.0, vcr-1.0.2, doctestplus-1.2.1, flake8-0.0.0, typeguard-4.3.0, anyio-4.4.0, session2file-0.1.11, openfiles-0.5.0, subtests-0.13.1, asdf-3.4.0, xdist-3.6.1 collected 5 items test_airy.py Try pickling galsim.Airy(lam_over_diam=1.25, obscuration=0.1) Testing Airy obscuration=0.1 flux: 17.9 17.820211169081006 17.820211169081006 maxsb: 8.907546146282357 8.907546 x: i,j = 2 3 0.14315423 0.14315423865255583 x: i,j = -4 1 0.031676114 0.03167611449513994 x: i,j = 0 -5 0.018037206 0.018037204861788034 x: i,j = -3 -3 0.0154736005 0.015473600392803334 8.354096827917735 [8.354096827917735, 8.354096827917735, 8.354096827917735, 8.354096827917735, 8.354096827917735] 0.16179046502244443 [0.16179046502244443, 0.16179046502244443, 0.16179046502244443, 0.16179046502244443, 0.1617904650224442] 1.8375221443104758e-05 [1.8375221443104758e-05, 1.8375221443104758e-05, 1.8375221443104758e-05, 1.8375221443104867e-05, 1.8375221443104867e-05] nyquist_scale, stepk, maxk = 0.625 0.055811298024539685 5.026548245743669 kimage scale,bounds = 0.055811298024539685 galsim.BoundsI(-91,91,-91,91) k flux: 17.9 (17.899999999999995+0j) (17.899999999999995+0j) k: i,j = 2 3 (16.88898885373816+0j) (16.88898885373816+0j) k: i,j = -4 1 (16.744870906057052+0j) (16.744870906057052+0j) k: i,j = 0 -5 (16.501702269711792+0j) (16.501702269711792+0j) k: i,j = -3 -3 (16.711643459763444+0j) (16.711643459763444+0j) (16.89513667046745+0j) [(16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j)] (11.83234933756301+0j) [(11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j)] 0j [0j, 0j, 0j, 0j, 0j] Testing Rotated Airy obscuration=0.1 flux: 17.9 17.82021116908669 17.82021116908669 maxsb: 8.907546146282357 8.907546 x: i,j = 2 3 0.14315423 0.14315423865255583 x: i,j = -4 1 0.031676114 0.03167611449513994 x: i,j = 0 -5 0.018037206 0.018037204861788034 x: i,j = -3 -3 0.0154736005 0.015473600392803334 8.354096827917735 [8.354096827917735, 8.354096827917735, 8.354096827917735, 8.354096827917735, 8.354096827917735] 0.16179046502244443 [0.16179046502244443, 0.16179046502244443, 0.16179046502244443, 0.16179046502244443, 0.16179046502244443] 1.8375221443104758e-05 [1.8375221443104758e-05, 1.8375221443104758e-05, 1.8375221443104707e-05, 1.8375221443104758e-05, 1.8375221443104758e-05] nyquist_scale, stepk, maxk = 0.625 0.055811298024539685 5.026548245743669 kimage scale,bounds = 0.055811298024539685 galsim.BoundsI(-91,91,-91,91) k flux: 17.9 (17.899999999999995+0j) (17.899999999999995+0j) k: i,j = 2 3 (16.88898885373797+0j) (16.88898885373816+0j) k: i,j = -4 1 (16.74487090605722+0j) (16.744870906057052+0j) k: i,j = 0 -5 (16.501702269711892+0j) (16.501702269711792+0j) k: i,j = -3 -3 (16.711643459763625+0j) (16.711643459763444+0j) (16.89513667046745+0j) [(16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j), (16.89513667046745+0j)] (11.83234933756301+0j) [(11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j), (11.83234933756301+0j)] 0j [0j, 0j, 0j, 0j, 0j] Testing Airy obscuration=0.0 flux: 17.9 17.820345854354567 17.820345854354567 maxsb: 6.888727291159018 6.888727 x: i,j = 2 3 0.09253469 0.09253469279837165 x: i,j = -4 1 0.016914546 0.016914546939777286 x: i,j = 0 -5 0.01993737 0.019937370440713805 x: i,j = -3 -3 0.006969283 0.006969283001836318 6.5622189642042175 [6.5622189642042175, 6.5622189642042175, 6.5622189642042175, 6.5622189642042175, 6.5622189642042175] 0.5122844307682973 [0.5122844307682973, 0.5122844307682973, 0.5122844307682973, 0.5122844307682973, 0.5122844307682968] 7.148358373819705e-06 [7.148358373819705e-06, 7.148358373819705e-06, 7.148358373819705e-06, 7.148358373820165e-06, 7.148358373820165e-06] nyquist_scale, stepk, maxk = 0.7142857142857143 0.05426098419052468 4.39822971502571 kimage scale,bounds = 0.05426098419052468 galsim.BoundsI(-82,82,-82,82) k flux: 17.9 (17.9+0j) (17.9+0j) k: i,j = 2 3 (16.8865523953315+0j) (16.8865523953315+0j) k: i,j = -4 1 (16.741196038643636+0j) (16.741196038643636+0j) k: i,j = 0 -5 (16.495029358331767+0j) (16.4950293583318+0j) k: i,j = -3 -3 (16.707630845826152+0j) (16.707630845826152+0j) (16.8639863111466+0j) [(16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j)] (11.26300228130922+0j) [(11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j)] 0j [0j, 0j, 0j, 0j, 0j] Testing Rotated Airy obscuration=0.0 flux: 17.9 17.820345854354567 17.820345854354567 maxsb: 6.888727291159018 6.888727 x: i,j = 2 3 0.09253469 0.0925346927983717 x: i,j = -4 1 0.016914546 0.01691454693977724 x: i,j = 0 -5 0.01993737 0.019937370440713805 x: i,j = -3 -3 0.006969283 0.006969283001836318 6.5622189642042175 [6.5622189642042175, 6.5622189642042175, 6.5622189642042175, 6.5622189642042175, 6.5622189642042175] 0.5122844307682973 [0.5122844307682973, 0.5122844307682973, 0.5122844307682973, 0.5122844307682973, 0.5122844307682973] 7.148358373819705e-06 [7.148358373819705e-06, 7.148358373819705e-06, 7.148358373818789e-06, 7.148358373819705e-06, 7.148358373819705e-06] nyquist_scale, stepk, maxk = 0.7142857142857143 0.05426098419052468 4.39822971502571 kimage scale,bounds = 0.05426098419052468 galsim.BoundsI(-82,82,-82,82) k flux: 17.9 (17.9+0j) (17.9+0j) k: i,j = 2 3 (16.886552395331613+0j) (16.8865523953315+0j) k: i,j = -4 1 (16.741196038643537+0j) (16.741196038643636+0j) k: i,j = 0 -5 (16.495029358331724+0j) (16.4950293583318+0j) k: i,j = -3 -3 (16.707630845826056+0j) (16.707630845826152+0j) (16.8639863111466+0j) [(16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j), (16.8639863111466+0j)] (11.26300228130922+0j) [(11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j), (11.26300228130922+0j)] 0j [0j, 0j, 0j, 0j, 0j] Start do_shoot prof.flux = 17.9 flux_max = 0.2668303 flux_tot = 17.8466637258445 nphot = 167209.87120439534 img2.sum => 17.814910886743746 img2.max = 0.26205868 New, saved array sizes: (564, 564) (564, 564) Sum of values: 17.814910886743746 17.8466637258445 Minimum image value: 0.0 4.5835616e-08 Maximum image value: 0.26205868 0.2668303 Peak location: 159329 158765 Moments Mx, My, Mxx, Myy, Mxy for new array: -0.479401 -0.4720518 234.00658 234.72241 2.5134168 Moments Mx, My, Mxx, Myy, Mxy for saved array: -0.50000012 -0.50000012 275.22526 275.22526 -0.00070296743 stepk, maxk = 0.05426098419052468 4.39822971502571 img.sum = 1.7999999999999994 cf. 1.8 nphot -> 1700.9550675209055 img.sum = 1.7968253968253962 cf. 1.8 img.max = 0.0349206349206349 cf. 0.027708847204661977 ratio = 1.260270218486735 img.sum = -1.7989417989417984 cf. -1.8 Start do_shoot prof.flux = 17.9 flux_max = 0.3415164 flux_tot = 17.84805671717487 nphot = 130652.99673669062 img2.sum => 17.815458959375974 img2.max = 0.34866986 New, saved array sizes: (564, 564) (564, 564) Sum of values: 17.815458959375974 17.84805671717487 Minimum image value: 0.0 4.1880675e-08 Maximum image value: 0.34866986 0.3415164 Peak location: 158766 158765 Moments Mx, My, Mxx, Myy, Mxy for new array: -0.44624961 -0.51099303 222.65452 221.32405 -1.4555396 Moments Mx, My, Mxx, Myy, Mxy for saved array: -0.50000024 -0.50000024 267.36748 267.36748 1.4893235e-05 stepk, maxk = 0.055811298024539685 5.026548245743669 img.sum = 1.8000000000000014 cf. 1.8 nphot -> 1328.8700422267066 img.sum = 1.797291196388262 cf. 1.8 img.max = 0.04063205417607221 cf. 0.03582923589566089 ratio = 1.1340474659966995 img.sum = -1.7986455981941312 cf. -1.8 New, saved array sizes: (564, 564) (564, 564) Sum of values: 17.81724958008166 17.846663156042922 Minimum image value: 7.3694053e-16 4.079516e-08 Maximum image value: 0.2689529 0.268953 Peak location: 158765 158765 Moments Mx, My, Mxx, Myy, Mxy for new array: -0.5 -0.5 230.01797 230.01797 9.1149298e-12 Moments Mx, My, Mxx, Myy, Mxy for saved array: -0.50000018 -0.50000018 275.13983 275.13983 -3.3654001e-05 New, saved array sizes: (564, 564) (564, 564) Sum of values: 17.81945074368817 17.848056148180728 Minimum image value: 8.7616294e-16 3.4983664e-08 Maximum image value: 0.34508568 0.3450858 Peak location: 158765 158765 Moments Mx, My, Mxx, Myy, Mxy for new array: -0.5 -0.5 223.38174 223.38174 -1.7191667e-12 Moments Mx, My, Mxx, Myy, Mxy for saved array: -0.50000021 -0.50000021 267.2819 267.2819 -0.0005806891 Try pickling galsim.Airy(lam_over_diam=1.4285714285714286, flux=17.9) Try pickling galsim.Airy(lam_over_diam=1.4285714285714286, flux=17.9) Try pickling galsim.Airy(lam_over_diam=1.25, obscuration=0.1, flux=17.9) Try pickling galsim.Airy(lam_over_diam=1.25, obscuration=0.1, flux=17.9) lam = 1818.0513041607599 Try pickling galsim.Airy(lam_over_diam=1.25, obscuration=0.1, flux=1.7) Try pickling galsim.Airy(lam_over_diam=1.25, obscuration=0.1, flux=1.7) Try pickling galsim.Airy(lam_over_diam=1.2500000000000002, obscuration=0.1, flux=1.7) Try pickling galsim.Airy(lam_over_diam=1.2500000000000002, obscuration=0.1, flux=1.7) time for test_airy = 0.69 .hlr_sum = 0.5000000000596438 fwhm ratio = 0.5000000000000002 time for test_airy_radii = 0.00 .time for test_airy_flux_scaling = 0.00 .FF ============================================================== FAILURES ============================================================== __________________________________________________________ test_airy_shoot ___________________________________________________________ test_airy.py:229: in test_airy_shoot 1/0 E ZeroDivisionError: division by zero ______________________________________________________________ test_ne _______________________________________________________________ test_airy.py:267: in test_ne 1/0 E ZeroDivisionError: division by zero ========================================================== warnings summary ========================================================== ../../lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/_pytest/config/__init__.py:1273 /Users/jmeyers3/src/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/_pytest/config/__init__.py:1273: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: asdf self._mark_plugins_for_rewrite(hook) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ====================================================== short test summary info ======================================================= FAILED test_airy.py::test_airy_shoot - ZeroDivisionError: division by zero FAILED test_airy.py::test_ne - ZeroDivisionError: division by zero =============================================== 2 failed, 3 passed, 1 warning in 0.72s ===============================================

let me know if you'd like to add --maxfail=1

Either way is fine. I don't mind getting all the failures at once.
Thanks for accommodating my peculiar preferences. :)

jmeyers314 requested a review from rmjarvis September 17, 2024 05:57

jmeyers314 force-pushed the pytest branch from d4e1b78 to 4f28486 Compare September 17, 2024 16:18

rmjarvis approved these changes Sep 17, 2024

View reviewed changes

rmjarvis reviewed Sep 17, 2024

View reviewed changes

jmeyers314 force-pushed the pytest branch from 4f28486 to bb148a6 Compare September 17, 2024 20:04

jmeyers314 force-pushed the quantity branch 2 times, most recently from dc14a00 to 35de439 Compare September 18, 2024 16:00

Base automatically changed from quantity to main September 18, 2024 16:18

jmeyers314 added 4 commits September 18, 2024 09:22

Modernize testing to pytest

d451718

Remove doc mentions of nosetests

4b6991b

See if we can really excise nosetests

21216f2

Improve python test_blah.py output

59d8550

jmeyers314 force-pushed the pytest branch from 94f906b to 59d8550 Compare September 18, 2024 16:24

rmjarvis merged commit b25d099 into main Sep 18, 2024
10 checks passed

rmjarvis deleted the pytest branch September 18, 2024 17:33

rmjarvis added admin Administrative stuff, rather than code tests Related to the test suite labels Sep 18, 2024

rmjarvis added this to the v2.6 milestone Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernize testing to pytest #1312

Modernize testing to pytest #1312

jmeyers314 commented Sep 17, 2024

rmjarvis Sep 17, 2024

jmeyers314 Sep 17, 2024

rmjarvis Sep 17, 2024

rmjarvis Sep 17, 2024 •

edited

Loading

rmjarvis Sep 17, 2024

jmeyers314 Sep 17, 2024

rmjarvis Sep 17, 2024 •

edited

Loading

rmjarvis Sep 17, 2024

jmeyers314 Sep 17, 2024

rmjarvis Sep 18, 2024

jmeyers314 Sep 18, 2024

jmeyers314 Sep 18, 2024

rmjarvis Sep 18, 2024

Modernize testing to pytest #1312

Modernize testing to pytest #1312

Conversation

jmeyers314 commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmjarvis Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmjarvis Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmjarvis Sep 17, 2024 •

edited

Loading

rmjarvis Sep 17, 2024 •

edited

Loading