Test suite fails

[strzibny]

Hi, I am about to package sprite-factory for fedora, but I am getting a few errors during the build:

+ ruby -rminitest/autorun -rrubygems -Ilib:test -
MiniTest::Unit::TestCase is now Minitest::Test. From /builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/test_case.rb:5:in `<module:SpriteFactory>'
MiniTest::Unit::TestCase is now Minitest::Test. From /builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/library_test.rb:3:in `<top (required)>'
MiniTest::Unit::TestCase is now Minitest::Test. From /builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/integration_test.rb:4:in `<module:SpriteFactory>'
MiniTest::Unit::TestCase is now Minitest::Test. From /builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/style_test.rb:4:in `<module:SpriteFactory>'
MiniTest::Unit::TestCase is now Minitest::Test. From /builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/runner_test.rb:4:in `<module:SpriteFactory>'
Run options: --seed 53921
# Running:
.......FFwhich: no pngcrush in (/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin:/builddir/.local/bin:/builddir/bin)
............................F.....
Finished in 1.929278s, 22.2881 runs/s, 109.3674 assertions/s.
  1) Failure:
SpriteFactory::LibraryTest#test_load_irregular_using_image_magick [/builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/library_test.rb:22]:
 - unexpected width at index 0.
Expected: 60
  Actual: 0
  2) Failure:
SpriteFactory::LibraryTest#test_create_using_chunkypng [/builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/library_test.rb:37]:
generated image does not match pregenerated reference:
 actual:   test/images/regular.horizontal.png
 expected: test/images/reference/regular.horizontal.png.
Expected: 0.0
  Actual: 3.2189052864267654e-20
  3) Failure:
SpriteFactory::RunnerTest#test_invalid_config [/builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/runner_test.rb:133]:
[RuntimeError] exception expected, not
Class: <LocalJumpError>
Message: <"no block given (yield)">
---Backtrace---
-:11:in `assert_raise'
/builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/test_case.rb:81:in `assert_runtime_error'
/builddir/build/BUILD/sprite-factory-1.6.0/usr/share/gems/gems/sprite-factory-1.6.0/test/runner_test.rb:133:in `test_invalid_config'
---------------
43 runs, 211 assertions, 3 failures, 0 errors, 0 skips

Note that we have minitest5 in Fedora so I run the test suite as:

# To run the test suite against minitest 5
ruby -rminitest/autorun -rrubygems -Ilib:test - << \EOF
  module Kernel
    alias orig_require require
    remove_method :require

    def require path
      orig_require path unless path == 'test/unit'
    end
  end
  module MiniTest::Assertions
    def assert_raise(*exp)
      assert_raises(*exp)
    end
  end
  Test = Minitest
  Dir.glob "./test/*_test.rb", &method(:require)
EOF

[jakesgordon]

I've switched to minitest in the latest version (1.7) - is this still an issue?

[strzibny]

So, here is the build using the new version. The first and last reported errors are gone, but I got 10 similar failures as the one from the last time:

+ ruby -rminitest/autorun -rrubygems -Ilib:test -e 'Dir.glob("./test/*_test.rb").sort.each { |t| require t }'
Run options: --seed 41789
# Running:
F....FF....F.FFF.F....FF.......................
Finished in 3.442323s, 13.6536 runs/s, 79.3069 assertions/s.
  1) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_sassy_style [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.sassy.png
 expected: test/images/reference/irregular.sassy.png.
Expected: 0.0
  Actual: 0.0040291457193674125
  2) Failure:
SpriteFactory::IntegrationTest#test_generate_packed_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.packed.png
 expected: test/images/reference/irregular.packed.png.
Expected: 0.0
  Actual: 0.001224956373697395
  3) Failure:
SpriteFactory::IntegrationTest#test_generate_sprite_from_other_formats [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/formats.png
 expected: test/images/reference/formats.png.
Expected: 0.0
  Actual: 0.000380501456122673
  4) Failure:
SpriteFactory::IntegrationTest#test_generate_custom_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/custom.png
 expected: test/images/reference/custom.png.
Expected: 0.0
  Actual: 0.01838716278377367
  5) Failure:
SpriteFactory::IntegrationTest#test_generate_vertical_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.vertical.png
 expected: test/images/reference/irregular.vertical.png.
Expected: 0.0
  Actual: 0.0050426118205579885
  6) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_fixed_size [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.fixed.png
 expected: test/images/reference/irregular.fixed.png.
Expected: 0.0
  Actual: 0.0009863348721011426
  7) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_margin [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.margin.png
 expected: test/images/reference/irregular.margin.png.
Expected: 0.0
  Actual: 0.0020278266284974146
  8) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_padding [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.padded.png
 expected: test/images/reference/irregular.padded.png.
Expected: 0.0
  Actual: 0.0020278266284974146
  9) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.png
 expected: test/images/reference/irregular.png.
Expected: 0.0
  Actual: 0.0040291457193674125
 10) Failure:
SpriteFactory::IntegrationTest#test_generate_horizontal_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.horizontal.png
 expected: test/images/reference/irregular.horizontal.png.
Expected: 0.0
  Actual: 0.0040291457193674125
47 runs, 273 assertions, 10 failures, 0 errors, 0 skips

Note that when I run tests locally I don't get this issues, but get the first one:

$ bundle exec rake test
/usr/bin/ruby -I"lib" -I"/home/strzibny/.gem/ruby/gems/rake-10.4.2/lib" "/home/strzibny/.gem/ruby/gems/rake-10.4.2/lib/rake/rake_test_loader.rb" "test/integration_test.rb" "test/layout/horizontal_test.rb" "test/layout/packed_test.rb" "test/layout/vertical_test.rb" "test/library_test.rb" "test/runner_test.rb" "test/style_test.rb" 
Run options: --seed 12915

# Running:

..........................................F......................................

Finished in 0.862435s, 93.9201 runs/s, 1836.6604 assertions/s.

  1) Failure:
SpriteFactory::LibraryTest#test_load_irregular_using_image_magick [/home/strzibny/fedora/rubygem-sprite-factory/sprite-factory/test/library_test.rb:22]:
 - unexpected width at index 0.
Expected: 60
  Actual: 0

81 runs, 1584 assertions, 1 failures, 0 errors, 0 skips

Here are the components used in build https://kojipkgs.fedoraproject.org//work/tasks/9163/10729163/root.log. One apparent difference is the version of ImageMagick (ImageMagick-6.8.8.10-9.fc22.x86_64 on my system vs 6.9.1.3-0.beta.3.fc23.2).

[jakesgordon]

Looks like different version of ImageMagick is causing a tiny variance in the generated image,

The assert_equal in assert_reference_image should probably be changed to an assert_in_delta to handle tiny variations a little more gracefully

[pvalena]

No more errors with ImageMagick-6.9.9.27-1.fc28.x86_64.

[pvalena]

No more errors with ImageMagick-6.9.9.27-1.fc28.x86_64.

Correction: that's for local execution only. I still get the failures when run in our build infrastructure:

+ ruby -rminitest/autorun -rrubygems -Ilib:test -e 'Dir.glob("./test/*_test.rb").sort.each { |t| require t }'
Run options: --seed 7439
# Running:
......................F....FF.F..F.FF.F.....FF.
Finished in 2.671778s, 17.5913 runs/s, 102.1792 assertions/s.
  1) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_margin [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.margin.png
 expected: test/images/reference/irregular.margin.png.
Expected: 0.0
  Actual: 2.567960561584528e-19
  2) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_sassy_style [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.sassy.png
 expected: test/images/reference/irregular.sassy.png.
Expected: 0.0
  Actual: 5.102353011252918e-19
  3) Failure:
SpriteFactory::IntegrationTest#test_generate_horizontal_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.horizontal.png
 expected: test/images/reference/irregular.horizontal.png.
Expected: 0.0
  Actual: 5.102353011252918e-19
  4) Failure:
SpriteFactory::IntegrationTest#test_generate_vertical_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.vertical.png
 expected: test/images/reference/irregular.vertical.png.
Expected: 0.0
  Actual: 6.385766958868686e-19
  5) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_fixed_size [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.fixed.png
 expected: test/images/reference/irregular.fixed.png.
Expected: 0.0
  Actual: 1.2490560171547147e-19
  6) Failure:
SpriteFactory::IntegrationTest#test_generate_sprite_from_other_formats [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/formats.png
 expected: test/images/reference/formats.png.
Expected: 0.0
  Actual: 7.806913177762553e-08
  7) Failure:
SpriteFactory::IntegrationTest#test_generate_custom_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/custom.png
 expected: test/images/reference/custom.png.
Expected: 0.0
  Actual: 1.2465802849832276e-18
  8) Failure:
SpriteFactory::IntegrationTest#test_generate_packed_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.packed.png
 expected: test/images/reference/irregular.packed.png.
Expected: 0.0
  Actual: 1.551236981066462e-19
  9) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite_with_padding [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.padded.png
 expected: test/images/reference/irregular.padded.png.
Expected: 0.0
  Actual: 2.567960561584528e-19
 10) Failure:
SpriteFactory::IntegrationTest#test_generate_irregular_sprite [/builddir/build/BUILD/sprite-factory-1.7/usr/share/gems/gems/sprite-factory-1.7/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/irregular.png
 expected: test/images/reference/irregular.png.
Expected: 0.0
  Actual: 5.102353011252918e-19
47 runs, 273 assertions, 10 failures, 0 errors, 0 skips

ImageMagick version is the same. Not sure what makes the difference.

[lucaskanashiro]

Most of these failures are fixed now but I still get one test failure:

$ bundle exec rake test
/usr/bin/ruby2.5 -w -I"lib"  "/usr/lib/ruby/vendor_ruby/rake/rake_test_loader.rb" "test/integration_test.rb" "test/layout/horizontal_test.rb" "test/layout/packed_test.rb" "test/layout/vertical_test.rb" "test/library_test.rb" "test/runner_test.rb" "test/style_test.rb" 
/home/kanashiro/projects/ruby/sprite-factory/test/integration_test.rb:178: warning: assigned but unused variable - input
/home/kanashiro/projects/ruby/sprite-factory/test/test_case.rb:109: warning: assigned but unused variable - img
/home/kanashiro/projects/ruby/sprite-factory/test/layout/packed_test.rb:275: warning: assigned but unused variable - result
/home/kanashiro/projects/ruby/sprite-factory/vendor/ruby/2.5.0/gems/rmagick-2.13.4/lib/rmagick_internal.rb:1826: warning: assigned but unused variable - current
/home/kanashiro/projects/ruby/sprite-factory/vendor/ruby/2.5.0/gems/chunky_png-1.3.0/lib/chunky_png/canvas/png_decoding.rb:395: warning: assigned but unused variable - line_no
/home/kanashiro/projects/ruby/sprite-factory/vendor/ruby/2.5.0/gems/chunky_png-1.3.0/lib/chunky_png/canvas/operations.rb:246: warning: assigned but unused variable - rotated
/home/kanashiro/projects/ruby/sprite-factory/vendor/ruby/2.5.0/gems/chunky_png-1.3.0/lib/chunky_png/canvas/drawing.rb:285: warning: shadowing outer local variable - y
Run options: --seed 23682

# Running:

........../home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/layout/packed.rb:116: warning: mismatched indentations at 'end' with 'module' at 3
.........................../home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/layout/horizontal.rb:12: warning: `*' after local variable or literal is interpreted as binary operator
/home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/layout/horizontal.rb:12: warning: even though it seems like argument prefix
/home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/test/integration_test.rb:182: warning: File.exists? is a deprecated name, use File.exist? instead
/home/kanashiro/projects/ruby/sprite-factory/test/integration_test.rb:183: warning: File.exists? is a deprecated name, use File.exist? instead
/home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
/home/kanashiro/projects/ruby/sprite-factory/test/integration_test.rb:187: warning: File.exists? is a deprecated name, use File.exist? instead
/home/kanashiro/projects/ruby/sprite-factory/test/integration_test.rb:188: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
F/home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
./home/kanashiro/projects/ruby/sprite-factory/lib/sprite_factory/runner.rb:62: warning: File.exists? is a deprecated name, use File.exist? instead
.....................

Finished in 0.727233s, 115.5063 runs/s, 2222.1213 assertions/s.

  1) Failure:
SpriteFactory::IntegrationTest#test_generate_sprite_from_other_formats [/home/kanashiro/projects/ruby/sprite-factory/test/test_case.rb:57]:
generated image does not match pregenerated reference:
 actual:   test/images/formats.png
 expected: test/images/reference/formats.png
 expected: test/images/reference/formats.png.
Expected |0.0 - 6.093295735239461e-05| (6.093295735239461e-05) to be <= 1.0e-07.

84 runs, 1616 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -w -I"lib"  "/usr/lib/ruby/vendor_ruby/rake/rake_test_loader.rb" "test/integration_test.rb" "test/layout/horizontal_test.rb" "test/layout/packed_test.rb" "test/layout/vertical_test.rb" "test/library_test.rb" "test/runner_test.rb" "test/style_test.rb" ]

Tasks: TOP => test
(See full trace by running task with --trace)

I am using imagemagick version 8:6.9.10.23+dfsg-2 in a Debian unstable system. However, I did not install gems from Debian repository but from rubygems.org using bundle (then I have all the dependencies in the versions specified in Gemfile). Any idea how to fix it @jakesgordon ?

[pvalena]

SpriteFactory::IntegrationTest#test_generate_sprite_from_other_formats [/builddir/build/BUILD/sprite-factory-1.7.1/usr/share/gems/gems/sprite-factory-1.7.1/test/test_case.rb:57]:

Same situation on Fedora Rawhide. Thanks for solving the rest!