Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: add downgrade testing #300

Merged
merged 1 commit into from
Nov 22, 2024
Merged

ci: add downgrade testing #300

merged 1 commit into from
Nov 22, 2024

Conversation

avik-pal
Copy link
Collaborator

fixes #271

@avik-pal avik-pal force-pushed the ap/downgrade branch 2 times, most recently from f04ad50 to 5b70284 Compare November 22, 2024 19:47
github-actions[bot]
github-actions bot reviewed Nov 22, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reactant.jl Benchmarks

Benchmark suite Current: 61957cf Previous: 1ef3992 Ratio
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 7293664373 ns 5413804517 ns 1.35
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant 5268804296 ns 6129080328 ns 0.86
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 5951875029 ns 6885334947 ns 0.86
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 7375603401 ns 7488256383 ns 0.98
ViT base (256 x 256 x 3 x 32)/forward/CPU/Lux 36623229816 ns 33764137689 ns 1.08
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1575403751 ns 1572509918 ns 1.00
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant 1552837782 ns 1540574169 ns 1.01
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1549283585 ns 1551622605 ns 1.00
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3420499811 ns 3307639474 ns 1.03
ViT small (256 x 256 x 3 x 4)/forward/CPU/Lux 2636545482 ns 2559807136 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 2196164262 ns 2130919371 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant 2200572871 ns 2129693004 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 2192395656 ns 2133876774 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 4111309298 ns 3968926719 ns 1.04
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Lux 6827295845 ns 6168170173.5 ns 1.11
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1457862254 ns 1426118988 ns 1.02
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant 1451599725 ns 1426378204 ns 1.02
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1430962374 ns 1409132409 ns 1.02
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3178896734 ns 3150696061 ns 1.01
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Lux 1296940504.5 ns 1640702004.5 ns 0.79
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 1740810857 ns 1694201241 ns 1.03
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant 1744103115 ns 1694859743 ns 1.03
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 1736439286 ns 1692209732 ns 1.03
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 3512868471 ns 3438162145 ns 1.02
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Lux 2930194791 ns 3437779789.5 ns 0.85
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 2217871504 ns 2125368450 ns 1.04
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant 2224869156 ns 2180362988 ns 1.02
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 2233027920 ns 2113675356 ns 1.06
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 4038740386 ns 3901836386 ns 1.04
ViT small (256 x 256 x 3 x 16)/forward/CPU/Lux 5579088056 ns 6764812162 ns 0.82
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 3077513163 ns 2977712820 ns 1.03
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant 3155492787 ns 3003667317 ns 1.05
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 3006823417 ns 3046943792 ns 0.99
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 4999168792 ns 4857405620 ns 1.03
ViT small (256 x 256 x 3 x 32)/forward/CPU/Lux 9317663113 ns 9041023986 ns 1.03
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 3195383104 ns 3109229306 ns 1.03
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant 3274792378 ns 3163244884 ns 1.04
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 3236218326 ns 3152278733 ns 1.03
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 5224927381 ns 5077029212 ns 1.03
ViT base (256 x 256 x 3 x 16)/forward/CPU/Lux 12249997056 ns 15973751896 ns 0.77
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1911110728 ns 1845679933 ns 1.04
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant 1882630739 ns 1847427891 ns 1.02
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1886973340 ns 1823520802 ns 1.03
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3644504340 ns 3578977105 ns 1.02
ViT base (256 x 256 x 3 x 4)/forward/CPU/Lux 3435605891.5 ns 3805640188.5 ns 0.90

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions GitHub Actions
Copy link
Contributor

Benchmark Results

main 61957cf... main/61957cfe294702...
comptime/NN/ViT base (optimize = :after_enzyme) 9.77 s 9.73 s 1
comptime/NN/ViT base (optimize = :all) 9.92 s 9.72 s 1.02
comptime/NN/ViT base (optimize = :before_enzyme) 9.77 s 9.77 s 1
comptime/NN/ViT base (optimize = :only_enzyme) 12.2 s 10.2 s 1.19
comptime/NN/ViT tiny (optimize = :after_enzyme) 7.33 s 7.36 s 0.996
comptime/NN/ViT tiny (optimize = :all) 7.48 s 7.44 s 1.01
comptime/NN/ViT tiny (optimize = :before_enzyme) 7.42 s 7.3 s 1.02
comptime/NN/ViT tiny (optimize = :only_enzyme) 7.55 s 7.44 s 1.02
comptime/NN/vgg11 bn=false (optimize = :after_enzyme) 1.21 ± 0.031 s 1.2 ± 0.034 s 1.01
comptime/NN/vgg11 bn=false (optimize = :all) 1.08 ± 0.015 s 1.12 ± 0.0039 s 0.97
comptime/NN/vgg11 bn=false (optimize = :before_enzyme) 1.08 ± 0.021 s 1.07 ± 0.021 s 1.01
comptime/NN/vgg11 bn=false (optimize = :only_enzyme) 1.11 ± 0.04 s 1.07 ± 0.009 s 1.04
comptime/NN/vgg11 bn=true (optimize = :after_enzyme) 1.86 ± 0.01 s 1.91 ± 0.031 s 0.977
comptime/NN/vgg11 bn=true (optimize = :all) 1.84 ± 0.0072 s 1.93 ± 0.073 s 0.949
comptime/NN/vgg11 bn=true (optimize = :before_enzyme) 2.07 ± 0.044 s 1.94 ± 0.041 s 1.07
comptime/NN/vgg11 bn=true (optimize = :only_enzyme) 1.92 ± 0.058 s 1.91 ± 0.054 s 1.01
comptime/NN/vgg13 bn=false (optimize = :after_enzyme) 1.49 ± 0.082 s 1.52 ± 0.044 s 0.981
comptime/NN/vgg13 bn=false (optimize = :all) 1.68 ± 0.1 s 1.47 ± 0.081 s 1.14
comptime/NN/vgg13 bn=false (optimize = :before_enzyme) 1.62 ± 0.023 s 1.5 ± 0.081 s 1.08
comptime/NN/vgg13 bn=false (optimize = :only_enzyme) 1.66 ± 0.09 s 1.62 ± 0.082 s 1.02
comptime/NN/vgg13 bn=true (optimize = :after_enzyme) 2.61 s 2.5 s 1.04
comptime/NN/vgg13 bn=true (optimize = :all) 2.51 s 2.47 s 1.02
comptime/NN/vgg13 bn=true (optimize = :before_enzyme) 2.54 s 2.5 s 1.01
comptime/NN/vgg13 bn=true (optimize = :only_enzyme) 2.73 s 2.81 s 0.974
comptime/NN/vgg16 bn=false (optimize = :after_enzyme) 2.07 ± 0.0042 s 2.14 ± 0.069 s 0.966
comptime/NN/vgg16 bn=false (optimize = :all) 2.12 ± 0.094 s 1.82 ± 0.0062 s 1.16
comptime/NN/vgg16 bn=false (optimize = :before_enzyme) 1.82 ± 0.01 s 1.77 ± 0.0016 s 1.03
comptime/NN/vgg16 bn=false (optimize = :only_enzyme) 2.23 ± 0.016 s 1.77 ± 0.036 s 1.26
comptime/NN/vgg16 bn=true (optimize = :after_enzyme) 3.19 s 3.06 s 1.04
comptime/NN/vgg16 bn=true (optimize = :all) 3.1 s 2.96 s 1.05
comptime/NN/vgg16 bn=true (optimize = :before_enzyme) 3.27 s 3.18 s 1.03
comptime/NN/vgg16 bn=true (optimize = :only_enzyme) 3.49 s 3.1 s 1.13
comptime/NN/vgg19 bn=false (optimize = :after_enzyme) 2.17 ± 0.012 s 2.02 ± 0.026 s 1.07
comptime/NN/vgg19 bn=false (optimize = :all) 2.36 ± 0.074 s 1.98 ± 0.012 s 1.19
comptime/NN/vgg19 bn=false (optimize = :before_enzyme) 2.27 ± 0.019 s 2.19 ± 0.19 s 1.04
comptime/NN/vgg19 bn=false (optimize = :only_enzyme) 2.08 ± 0.041 s 2.14 ± 0.1 s 0.973
comptime/NN/vgg19 bn=true (optimize = :after_enzyme) 3.9 s 3.9 s 1
comptime/NN/vgg19 bn=true (optimize = :all) 3.9 s 3.48 s 1.12
comptime/NN/vgg19 bn=true (optimize = :before_enzyme) 4.26 s 3.47 s 1.23
comptime/NN/vgg19 bn=true (optimize = :only_enzyme) 3.85 s 3.77 s 1.02
comptime/basics/2D sum (optimize = :after_enzyme) 27.2 ± 0.73 ms 27.9 ± 1 ms 0.975
comptime/basics/2D sum (optimize = :all) 0.032 ± 0.0015 s 0.0317 ± 0.0021 s 1.01
comptime/basics/2D sum (optimize = :before_enzyme) 28.8 ± 1.2 ms 30.2 ± 1.7 ms 0.952
comptime/basics/2D sum (optimize = :only_enzyme) 24.4 ± 1.4 ms 26.1 ± 1.9 ms 0.936
comptime/basics/cos.(x) (optimize = :after_enzyme) 0.0351 ± 0.00092 s 0.0351 ± 0.0014 s 1
comptime/basics/cos.(x) (optimize = :all) 0.0356 ± 0.0012 s 0.0364 ± 0.0034 s 0.976
comptime/basics/cos.(x) (optimize = :before_enzyme) 0.0366 ± 0.0012 s 0.0367 ± 0.0016 s 0.996
comptime/basics/cos.(x) (optimize = :only_enzyme) 0.0321 ± 0.0011 s 0.0327 ± 0.002 s 0.98
comptime/basics/∇cos (optimize = :all) 0.0528 ± 0.0016 s 0.0537 ± 0.0036 s 0.984
runtime/NN/ViT base (optimize = :after_enzyme) 6.31 s 6.31 s 1
runtime/NN/ViT base (optimize = :all) 6.31 s 6.22 s 1.01
runtime/NN/ViT base (optimize = :before_enzyme) 6.28 s 6.25 s 1.01
runtime/NN/ViT base (optimize = :only_enzyme) 7.52 s 7.53 s 0.998
runtime/NN/ViT tiny (optimize = :after_enzyme) 1.64 s 1.62 s 1.01
runtime/NN/ViT tiny (optimize = :all) 1.62 s 1.66 s 0.975
runtime/NN/ViT tiny (optimize = :before_enzyme) 1.64 s 1.64 s 0.998
runtime/NN/ViT tiny (optimize = :only_enzyme) 2.61 s 2.61 s 1
runtime/NN/vgg11 bn=false (optimize = :after_enzyme) 2.19 s 2.17 s 1.01
runtime/NN/vgg11 bn=false (optimize = :all) 2.1 s 2.18 s 0.967
runtime/NN/vgg11 bn=false (optimize = :before_enzyme) 2.11 s 2.12 s 0.995
runtime/NN/vgg11 bn=false (optimize = :only_enzyme) 1.92 s 1.96 s 0.982
runtime/NN/vgg11 bn=true (optimize = :after_enzyme) 2.31 s 2.36 s 0.981
runtime/NN/vgg11 bn=true (optimize = :all) 2.31 s 2.28 s 1.01
runtime/NN/vgg11 bn=true (optimize = :before_enzyme) 2.28 s 2.28 s 1
runtime/NN/vgg11 bn=true (optimize = :only_enzyme) 2.34 s 2.29 s 1.02
runtime/NN/vgg13 bn=false (optimize = :after_enzyme) 2.98 s 3.03 s 0.984
runtime/NN/vgg13 bn=false (optimize = :all) 3.04 s 2.99 s 1.02
runtime/NN/vgg13 bn=false (optimize = :before_enzyme) 2.91 s 3.11 s 0.934
runtime/NN/vgg13 bn=false (optimize = :only_enzyme) 2.8 s 2.76 s 1.01
runtime/NN/vgg13 bn=true (optimize = :after_enzyme) 3.26 s 3.24 s 1.01
runtime/NN/vgg13 bn=true (optimize = :all) 3.17 s 3.22 s 0.983
runtime/NN/vgg13 bn=true (optimize = :before_enzyme) 3.25 s 3.25 s 1
runtime/NN/vgg13 bn=true (optimize = :only_enzyme) 3.32 s 3.3 s 1.01
runtime/NN/vgg16 bn=false (optimize = :after_enzyme) 3.78 s 3.75 s 1.01
runtime/NN/vgg16 bn=false (optimize = :all) 3.75 s 3.65 s 1.03
runtime/NN/vgg16 bn=false (optimize = :before_enzyme) 3.76 s 3.73 s 1.01
runtime/NN/vgg16 bn=false (optimize = :only_enzyme) 3.66 s 3.54 s 1.03
runtime/NN/vgg16 bn=true (optimize = :after_enzyme) 4.2 s 3.89 s 1.08
runtime/NN/vgg16 bn=true (optimize = :all) 4.09 s 4.15 s 0.983
runtime/NN/vgg16 bn=true (optimize = :before_enzyme) 4.14 s 4.06 s 1.02
runtime/NN/vgg16 bn=true (optimize = :only_enzyme) 4.34 s 4.54 s 0.955
runtime/NN/vgg19 bn=false (optimize = :after_enzyme) 4.57 s 4.52 s 1.01
runtime/NN/vgg19 bn=false (optimize = :all) 4.54 s 4.56 s 0.997
runtime/NN/vgg19 bn=false (optimize = :before_enzyme) 4.54 s 4.57 s 0.993
runtime/NN/vgg19 bn=false (optimize = :only_enzyme) 4.35 s 4.45 s 0.978
runtime/NN/vgg19 bn=true (optimize = :after_enzyme) 4.98 s 5.02 s 0.993
runtime/NN/vgg19 bn=true (optimize = :all) 5.06 s 5.04 s 1.01
runtime/NN/vgg19 bn=true (optimize = :before_enzyme) 5.04 s 4.88 s 1.03
runtime/NN/vgg19 bn=true (optimize = :only_enzyme) 5.54 s 5.43 s 1.02
time_to_load 1.99 ± 0.037 s 1.98 ± 0.021 s 1

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

@avik-pal avik-pal merged commit 97bf7ad into main Nov 22, 2024
27 of 39 checks passed
@avik-pal avik-pal deleted the ap/downgrade branch November 22, 2024 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@wsmoses wsmoses wsmoses approved these changes

@mofeing mofeing Awaiting requested review from mofeing

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add downgrade testing CI
2 participants
@avik-pal @wsmoses