Symbol resolution fails for system libraries in DevTools stack traces in Performance tab

[pwasowski2]

The problem is described in: #53 (comment)

run an app in profile mode and open the Performance view in Flutter DevTools, you will see this screen:

As you can see in the stack traces at the bottom of the view (CPU Flame Chart), symbols are resolved to their offsets within the shared objects (/proc/self/fd/35/lib/libflutter_engine.so+0x.....) but not their actual names. I'm not sure whether this is normal (because the symbols were stripped out).

[bbrto21]

The inspiration from this conversation(#97 (comment)) has led to the following progress:

[bbrto21]

I checked the profile tool using three .so as follows
Among them, most symbols were resolved when using .so with the --unoptimized option.
@pwasowski2 Could you check it?

profile

profile.so.unstripped

profile_unopt

To use --unoptimized, you may need this

diff --git a/shell/platform/tizen/BUILD.gn b/shell/platform/tizen/BUILD.gn
index 809474371..550d19517 100644
--- a/shell/platform/tizen/BUILD.gn
+++ b/shell/platform/tizen/BUILD.gn
@@ -161,6 +161,7 @@ template("embedder_for_profile") {
     public_deps = [ ":flutter_engine" ]
 
     deps = [
+      "//flutter/runtime:libdart",
       "//flutter/shell/platform/common/cpp:common_cpp",
       "//flutter/shell/platform/common/cpp:common_cpp_input",
       "//flutter/shell/platform/common/cpp:common_cpp_library_headers",

[bbrto21]

As a result of test and trying based on several config options, I think it depends on "//build/config:debug"

[pwasowski2]

@bbrto21, I think you've nailed it 👌 Thanks a lot 😃

Native symbols indeed resolve after adding --unoptimized flag. Some of them (e.g. from libc.so) are still not resolved, but installing related *-debug.rpms solves the problem and proper names are shown for them, too.

The question now is, what should we do about it? IMO, turning --unoptimized on by default for release binaries is not an option. Adding this flag makes binaries like libflutter_tizen_mobile.so orders of magnitude larger (15 MB vs 0.35 MB when I checked it) and probably has other side effects, that we wouldn't like to see in release binaries, too.

What about just adding information, that adding --unoptimized flag switches on the symbol resolution, to the wiki (I guess this will be the right place: https://github.com/flutter-tizen/flutter-tizen/wiki/Miscellaneous)?

@bbrto21, @swift-kim what do you think about it?

[swift-kim]

turning --unoptimized on by default for release binaries is not an option.

Agreed.

and probably has other side effects,

Right. Turning on the unopt flag significantly lowers the performance as well. And that's why we can't use the "unoptimized" binaries for CPU profiling and performance analysis. So if we want to get correct results from the profiling tools, we need to find a way to fix the symbol resolution issue without disabling optimizations.

What about just adding information, that adding --unoptimized flag switches on the symbol resolution, to the wiki

I think the best place would be the "Building the engine" page.

[bbrto21]

@pwasowski2 @swift-kim I agree too

[pwasowski2]

Ok, I'll go further down the rabbit hole and see how we can get all the symbols for profile builds ;)

I'll update the wiki afterwards.

[pwasowski2]

I have investigated the problem a little bit more and have found out a way of resolving more symbols on profile builds.

First, look at this change draft of tools/gn : pwasowski2@b9b44b9
Flutter devs have commented out enable_profiling flag from gn wrapper script. It means, that enable_profiling is always false and thus this conditional is not executed: https://github.com/flutter/buildroot/blob/ff1975def53bfdaa1ff3f4b0cceabf54e1323392/build/config/compiler/BUILD.gn#L346.
(When enable_profiling is true, -g flag is added to cflags, which seems to be the flag for turning symbols on).

After applying pwasowski2@b9b44b9 and running a build with the following set of gn flags (no --unoptimized):

flutter/tools/gn \
--target-os linux \
--linux-cpu arm \
--no-goma \
--target-toolchain `pwd`/tizen_tools/toolchains \
--target-sysroot `pwd`/tizen_tools/sysroot/arm \
--target-triple armv7l-tizen-linux-gnueabi \
--runtime-mode profile \
--enable-fontconfig \
--embedder-for-target \
--disable-desktop-embeddings \
--build-tizen-shell

we get more symbols in the profiler view.

enable_profiling = false

enable_profiling = true

As you can see, many more symbols are resolved, but some of the libflutter_engine.so parts are still unresolved.
Unfortunately, I haven't found any way to resolve all symbols from this .so.

I have tried increasing symbol_level of profile builds to the same as debug builds: pwasowski2/buildroot@8f6a8a5, but it didn't help to resolve more symbols.

enable_profiling = true, symbol_level = 2

In case, you're wondering, if setting gcc's -g3 flag would enable resolution of all the symbols - the answer is no, as you can see in the last screenshot:

enable_profiling = true, symbol_level = 2, -g3 gcc flag

I think we can safely apply this pwasowski2@b9b44b9 to get almost all symbols, without setting --unoptimized or any other flag.

@swift-kim, @bbrto21 what do you think about it?

Edit: I've created a PR for enabling enable_profiling: #104

[swift-kim]

Thank you for the detailed analysis! The suggested change looks good to me, and I have a few quick questions:

• Are the remaining unresolved symbols related to text rendering? If so, could you check again by deleting the 271st line of the shell/platform/embedder/BUILD.gn file and rebuilding the engine? The version script file flutter_engine_exports.lst is supplied to libflutter_engine.so to prevent libfreetype symbols from being exported (Resolve freetype symbols locally #85).
• Did you try enabling the enable_full_stack_frames_for_profiling option which is defined along with enable_profiling in build/config/profiler.gni?

And some doubts:

• Would enabling enable_profiling cause any noticeable performance degradation? It seems they (Google) disabled it because it affected their benchmarks.
• Would enabling enable_profiling cause any issue on x86 emulator? It seems they (Google) had an issue related to unexpected text relocations on Android x86 by enabling the option, but this shouldn't be a problem on Tizen I think.

[bbrto21]

I have tried increasing symbol_level of profile builds to the same as debug builds: pwasowski2/buildroot@8f6a8a5, but it didn't help to resolve more symbols.

In case, you're wondering, if setting gcc's -g3 flag would enable resolution of all the symbols - the answer is no, as you can see in the last screenshot:

The last time I tried to use symbol_level, it only worked as expected with "//build/config:debug". of course, there is a slight performance penalty.
@pwasowski2 Could you check the below?

diff --git a/build/config/BUILDCONFIG.gn b/build/config/BUILDCONFIG.gn
index 44c276c..70169f8 100644
--- a/build/config/BUILDCONFIG.gn
+++ b/build/config/BUILDCONFIG.gn
@@ -5,6 +5,9 @@
 # define is_wasm
 import("//build/toolchain/wasm.gni")
 
+# define enable_profiling
+import("//build/config/profiler.gni")
+
 # =============================================================================
 # PLATFORM SELECTION
 # =============================================================================
@@ -369,7 +372,10 @@ if (is_clang) {
 }
 
 # Optimizations and debug checking.
-if (is_debug) {
+if (enable_profiling) {
+  _native_compiler_configs += [ "//build/config:debug" ]
+  _default_optimization_config = "//build/config/compiler:optimize"
+} else if (is_debug) {
   _native_compiler_configs += [ "//build/config:debug" ]
   _default_optimization_config = "//build/config/compiler:no_optimize"
 } else {
@@ -382,7 +388,7 @@ _native_compiler_configs += [ _default_optimization_config ]
 if (symbol_level == -1) {
   # Linux is slowed by having symbols as part of the target binary, whereas
   # Mac and Windows have them separate, so in Release Linux, default them off.
-  if (is_debug || !is_linux) {
+  if (is_debug || !is_linux || enable_profiling) {
     symbol_level = 2
   } else if (is_asan || is_lsan || is_tsan || is_msan) {
     # Sanitizers require symbols for filename suppressions to work.

• config:debug : https://github.com/flutter/buildroot/blob/ff1975def53bfdaa1ff3f4b0cceabf54e1323392/build/config/BUILD.gn#L66
• config:complier:optimize : https://github.com/flutter/buildroot/blob/ff1975def53bfdaa1ff3f4b0cceabf54e1323392/build/config/compiler/BUILD.gn#L888

[pwasowski2]

Thanks for feedback, I have tested proposed build configuration changes and here's what I've found:

Are the remaining unresolved symbols related to text rendering? If so, could you check again by deleting the 271st line of the shell/platform/embedder/BUILD.gn file and rebuilding the engine? The version script file flutter_engine_exports.lst is supplied to libflutter_engine.so to prevent libfreetype symbols from being exported (#85).

It seems, the unresolved symbols don't come from Freetype. After removeing the ldflags = [ "-Wl,--version-script=" + rebase_path("flutter_engine_exports.lst") ] from shell/platform/embedder/BUILD.gn, some symbols are still unresolved.

enable_profiling = true, 271st line of the shell/platform/embedder/BUILD.gn commented out

Did you try enabling the enable_full_stack_frames_for_profiling option which is defined along with enable_profiling in build/config/profiler.gni?

Yes, I have and unfortunately, it didn't help, too.

enable_profiling = true, enable_full_stack_frames_for_profiling enabled

The last time I tried to use symbol_level, it only worked as expected with "//build/config:debug". of course, there is a slight performance penalty.
@pwasowski2 Could you check the below?

I have applied the diff to my repo and some symbols were still unresolved. When I have tested symbols_level = 2 before, it was properly set by the compiler even without this change in build/config/BUILDCONFIG.gn b/build/config/BUILDCONFIG.gn:

 # Optimizations and debug checking.
-if (is_debug) {
+if (enable_profiling) {
+  _native_compiler_configs += [ "//build/config:debug" ]
+  _default_optimization_config = "//build/config/compiler:optimize"
+} else if (is_debug) {

Update
@swift-kim, regarding performance - I have run a little benchmark with enable_profiling enabled and disabled and there was no significant difference in UI rendering time. I have run infinite_list app from https://github.com/flutter/samples and scrolled the screen 10 times. As you can see in the screens below, framerate is almost the same.

enable_profiling enabled

enable_profiling disabled

Here's the result of the same benchmark for debug build with --unoptimized flag - the framerate is 4 times smaller than in profile builds

debug build with --unoptimized

[pwasowski2]

One more benchmark, that was missing in my previous comment:

profile build with --unoptimized

Adding --unoptimized flag to --profile build seems to decrease the performance when compared to --profile with enable_profiling enabled, but only slightly. With both these settings, the same amount of symbols is resolved.

@swift-kim, @bbrto21, I have made this PR ready for review #104. Could you please take a look at it? :)
If we merge this PR, I think, we no longer need to add add information about enabling resolution of more symbols via --unoptimized gn flag to wiki - the same amount of symbols will be resolved by default in all profile builds.

[bbrto21]

Looks good!, I think this issue is almost closed now.