Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TEST04 takes reciprocal of scores for SingleStream/MultiStream #1433

Open
psyhtest opened this issue Jul 6, 2023 · 0 comments · May be fixed by #1491
Open

TEST04 takes reciprocal of scores for SingleStream/MultiStream #1433

psyhtest opened this issue Jul 6, 2023 · 0 comments · May be fixed by #1491

Comments

@psyhtest
Copy link
Contributor

psyhtest commented Jul 6, 2023 •
edited
Loading

It is rather peculiar and confusing that for TEST04 (only applicable to ResNet50), the comparisons for SingleStream and MultiStream are performed on score reciprocals (more precisely, 1e9 / float(score)) rather than on scores directly like for TEST01 and TEST05 e.g.:

  • SingleStream:
resnet50/singlestream/TEST04/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST04/verify_performance.txt-reference score = 1569.1221859754996
resnet50/singlestream/TEST04/verify_performance.txt-test score = 1571.568441019822
resnet50/singlestream/TEST04/verify_performance.txt-TEST PASS
--
resnet50/singlestream/TEST05/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST05/verify_performance.txt-reference score = 637299
resnet50/singlestream/TEST05/verify_performance.txt-test score = 638180
resnet50/singlestream/TEST05/verify_performance.txt-TEST PASS
--
resnet50/singlestream/TEST01/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST01/verify_performance.txt-reference score = 637299
resnet50/singlestream/TEST01/verify_performance.txt-test score = 638024
resnet50/singlestream/TEST01/verify_performance.txt-TEST PASS
  • MultiStream:
resnet50/multistream/TEST04/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST04/verify_performance.txt-reference score = 460.6963794332237
resnet50/multistream/TEST04/verify_performance.txt-test score = 469.96500640562306
resnet50/multistream/TEST04/verify_performance.txt-TEST PASS
--
resnet50/multistream/TEST05/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST05/verify_performance.txt-reference score = 2170627
resnet50/multistream/TEST05/verify_performance.txt-test score = 2216725
resnet50/multistream/TEST05/verify_performance.txt-TEST PASS
--
resnet50/multistream/TEST01/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST01/verify_performance.txt-reference score = 2170627
resnet50/multistream/TEST01/verify_performance.txt-test score = 2184797
resnet50/multistream/TEST01/verify_performance.txt-TEST PASS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix SingleStream and MultiStream reciprocal score comparison for TEST04 krai/inference
1 participant
@psyhtest