include build info in out directory

[brendangregg]

I've found that perf-map-agent compiled for OracleJDK can crash OpenJDK (usually on the second invocation). I'm changing my tooling so that we have builds for each. This ticket is to suggest that build info is included in the out directory, so that other tooling can double check what it was built against. Eg, just a:

echo JAVA_HOME=$JAVA_HOME > out/build.txt

From the build process would be handy.

For reference, here's stack trace from one of the crashes:

Core was generated by `java -Xms24952m -Xmx24952m -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemory'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe7253fb700 (LWP 5078))]
Installing openjdk unwinder
(gdb) bt
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fe7a6d0e02a in __GI_abort () at abort.c:89
#2  0x00007fe7a6012db9 in os::abort(bool) (dump_core=<optimized out>) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
#3  0x00007fe7a61c6897 in VMError::report_and_die() (this=this@entry=0x7fe7253f7e20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
#4  0x00007fe7a61c6fb7 in crash_handler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8070, ucVoid=0x7fe7253f7f40)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/vmError_linux.cpp:106
#5  0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007fe7a6e1a287 in __GI__dl_addr (symbolp=0x0, mapp=0x0, info=0x7fe7253f83f0, match=0x7fe6f0001800, addr=140630735942332) at dl-addr.c:61
#7  0x00007fe7a6e1a287 in __GI__dl_addr (address=0x7fe7250f72bc, info=0x7fe7253f83f0, mapp=0x0, symbolp=0x0) at dl-addr.c:137
#8  0x00007fe7a6018b49 in os::find(unsigned char*, outputStream*) (addr=addr@entry=0x7fe7250f72bc "\205W\255Ox\361\030\233\067O\200\262\266\067\071_\321\357\335+\204\360\344_\223_", st=st@entry=0x7fe7253f8b90) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:5166
#9  0x00007fe7a600d61d in os::print_location(outputStream*, long, bool) (st=0x7fe7253f8b90, x=140630735942332, verbose=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:1030
#10 0x00007fe7a601cfbb in os::print_register_info(outputStream*, void*) (st=0x7fe7253f8b90, context=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:852
#11 0x00007fe7a61c557a in VMError::report(outputStream*) (this=0x7fe7253f8c90, st=st@entry=0x7fe7253f8b90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:542
#12 0x00007fe7a61c63eb in VMError::report_and_die() (this=this@entry=0x7fe7253f8c90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:971
#13 0x00007fe7a601c29f in JVM_handle_linux_signal(int, siginfo_t*, void*, int) (sig=sig@entry=11, info=info@entry=0x7fe7253f8f30, ucVoid=ucVoid@entry=0x7fe7253f8e00, abort_if_unrecognized=abort_if_unrecognized@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#14 0x00007fe7a600fef8 in signalHandler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8f30, uc=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4346
#15 0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#16 0x00007fe7a72b9cda in check_match (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", ref=ref@entry=0x0, version=version@entry=0x0, flags=flags@entry=2, type_class=type_class@entry=0, sym=sym@entry=0x6f8, symidx=44, strtab=0x7a0 <error: Cannot access memory at address 0x7a0>, map=0x7fe6f0001800, versioned_sym=0x7fe7253f9338, num_versions=0x7fe7253f9334) at dl-lookup.c:92
#17 0x00007fe7a72ba64b in do_lookup_x (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", new_hash=new_hash@entry=1336760197, old_hash=old_hash@entry=0x7fe7253f9400, ref=0x0, result=result@entry=0x7fe7253f9410, scope=<optimized out>, i=0, version=0x0, flags=2, skip=0x0, type_class=0, undef_map=0x7fe6f0001800) at dl-lookup.c:423
#18 0x00007fe7a72bab1f in _dl_lookup_symbol_x (undef_name=0x7fe6f00076b0 "Agent_OnAttach", undef_map=0x7fe6f0001800, ref=0x7fe7253f94e8, symbol_scope=0x7fe6f0001b88, version=0x0, type_class=0, flags=2, skip_map=0x0) at dl-lookup.c:829
#19 0x00007fe7a6e1a873 in do_sym (handle=0x7fe6f0001800, name=0x7fe6f00076b0 "Agent_OnAttach", who=<optimized out>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:168
#20 0x00007fe7a6e1ad1d in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:273
#21 0x00007fe7a68ba034 in dlsym_doit (a=a@entry=0x7fe7253f96f0) at dlsym.c:50
#22 0x00007fe7a72c0564 in _dl_catch_error (objname=0x7fe6f00017c0, errstring=0x7fe6f00017c8, mallocedp=0x7fe6f00017b8, operate=0x7fe7a68ba020 <dlsym_doit>, args=0x7fe7253f96f0) at dl-error.c:187
#23 0x00007fe7a68ba571 in _dlerror_run (operate=operate@entry=0x7fe7a68ba020 <dlsym_doit>, args=args@entry=0x7fe7253f96f0) at dlerror.c:163
#24 0x00007fe7a68ba088 in __dlsym (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach") at dlsym.c:70
#25 0x00007fe7a60138f4 in os::dll_lookup(void*, char const*) (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach")
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:2052
#26 0x00007fe7a600c34c in os::find_agent_function(AgentLibrary*, bool, char const**, unsigned long) (agent_lib=agent_lib@entry=0x7fe6f00079a0, check_lib=check_lib@entry=false,---Type <return> to continue, or q <return> to quit--- 
 syms=syms@entry=0x7fe7253f9860, syms_len=syms_len@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:490
#27 0x00007fe7a5e8f5f3 in JvmtiExport::load_agent_library(AttachOperation*, outputStream*) (op=0x7fe6f0000a30, st=0x7fe7253fad20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvmtiExport.cpp:2252
#28 0x00007fe7a5a43db1 in attach_listener_thread_entry(JavaThread*, Thread*) (thread=<optimized out>, __the_thread__=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/services/attachListener.cpp:453
#29 0x00007fe7a616aa07 in JavaThread::thread_main_inner() (this=this@entry=0x7fe708001000)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1699
#30 0x00007fe7a616aecc in JavaThread::run() (this=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1679
#31 0x00007fe7a6011d82 in java_start(Thread*) (thread=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:790
#32 0x00007fe7a66a36ba in start_thread (arg=0x7fe7253fb700) at pthread_create.c:333
#33 0x00007fe7a6dde3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[jrudolph]

Thanks for the report, @brendangregg. I agree, it might make sense to add such a file.

I wonder how that error can happen. It doesn't seem to be failing in the code of the agent itself but in dl_sym. From searching for check_match dlsym on google I found a few things that would suggest that maybe the agent.so file was changed between the two runs. Could that be the case here?

[jjhart]

More generally, what level of parity is required between the machine that compiled perf-map-agent and the machine running it?

For example, is it risky to compile perf-map-agent on linux kernel 3.4 and then run it on linux kernel 4.4? (with equivalent- or near-equivalent OpenJDK versions)? Or, conversely, holding the linux kernel constant but mutating the OpenJDK versions?

I ask b/c I've see one crash that may or may not have been perf-map-agent related (this was a few months ago), and as a result we've been a bit tentative about using it on prod. It's OpenJDK across the board, but there's some variation in kernel and JDK 1.8.* versions.

[jrudolph]

Hi @jjhart,

perf-map-agent is not related to the Linux kernel at all. It generates files which can then be read by user-level perf tools.

It's a JVM agent that is compiled against header files included with the JDK. If I remember correctly it does not even link against JDK libraries. Instead, all references are dynamically resolved. It might (accidentally?) rely on some particularities of a given JDK and it is also missing some error checks so that it might run into issues (though, we haven't seen only few reports about those).

[jrudolph]

More generally, what level of parity is required between the machine that compiled perf-map-agent and the machine running it?

To be clear about that: as far as I know, if you are on the right CPU architecture, the binary should work everywhere. If that's not the case, I would consider it a bug.

[jjhart]

@jrudolph - thanks for the clarification. I did indeed compile with the same JDK as used in prod, so whatever caused my crash is unrelated to the issue @brendangregg is reporting.