Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shp benchmarks in borealis failed #593

Open
lslusarczyk opened this issue Oct 25, 2023 · 2 comments · Fixed by #614 or #618
Open

shp benchmarks in borealis failed #593

lslusarczyk opened this issue Oct 25, 2023 · 2 comments · Fixed by #614 or #618
Assignees
Labels
bug Something isn't working

Comments

@lslusarczyk
Copy link
Contributor

https://github.com/intel-sandbox/libraries.runtimes.hpc.dds.dr-ci/actions/runs/6637463788

check it, if newly enabled benchmarks revealed some problem fix it if it is easy or at least comment out appropriate benchmark in shp with comment pointing to this issue

@lslusarczyk lslusarczyk added the bug Something isn't working label Oct 25, 2023
@lslusarczyk lslusarczyk moved this to 🏗 In progress in Distributed-Ranges Project Oct 27, 2023
@lslusarczyk
Copy link
Contributor Author

analysing the failure, currently on Borealis shp-benhc times out, on devcloud my account expired, running locally - out-of-mem (seems shp-bench ignores in some cases vector-size - fixing it...)

in progress...

@lslusarczyk
Copy link
Contributor Author

ExclusiveScan benchmark in shp fails. See: https://github.com/intel-sandbox/libraries.runtimes.hpc.dds.dr-ci/actions/runs/6703645628

Exact command:

ONEAPI_DEVICE_SELECTOR='level_zero:gpu;ext_oneapi_cuda:gpu' \
KMP_AFFINITY=compact shp/shp-bench --vector-size 2000000000 --reps 50\
 --benchmark_out_format=json --context device:GPU --context model:SHP --context runtime:SYCL\
 --context target:SHP_SYCL_GPU --v=3 --benchmark_out=dr-bench-adc021a8e9a64a6c86da243e79fcb338.json\
 --benchmark_filter=.*Sort_DR\|Gemm_DR\|^DotProduct_DR\|^Exclusive_Scan_DR\|^Inclusive_Scan_DR\|^Reduce_DR --num-devices 6

output with failure:

- LOG(2): Running Exclusive_Scan_DR/min_time:0.100/min_warmup_time:0.100/real_time for 1
free(): invalid next size (fast)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 🏗 In progress
2 participants