Symbol Fundamentals 

Symbols play an important role in callstack generation for minidumps, as they are used to associate a dump's memory addresses to the appropriate function names. For 64 bit builds, executable files also store debug information and are required in addition to the symbol files for complete callstack resolution. Symbols and executables can be provided to Backtrace via a custom symbol server integration, automated uploads baked into the build pipeline, or manual uploads within the UI or via curl. 

Determine which symbols are missing 

The first step in resolving a missing symbols issue is to determine which symbols are missing.

 

Missing symbol files

Backtrace indicates that a crash report is missing symbols by setting an attribute, _missing_symbols, to 1, and in the debug view UI (in the callstack and the global annotations sections). 

When a crash report is missing standard symbol files, in the debug view you'll see a memory address in the callstack with an orange triangle "missing symbol" icon.

 

You can hover over callstack frames missing symbols to see more information (like the file path and debug ID),

 

 

and you can click the blue missing symbol header to see additional missing symbol information. 

Missing executable files

The debug view similarly indicates when a crash report is missing debug information from executable files, also known as CFI (call frame information). The callstack section will show a guessed function name or a memory address in gray font, and the frame will be enclosed in <>.

Upon hover, you'll see a "missing CFI" message with details about the missing file.

If either of the filename or the debug ID are missing in this hover message, then you can search the global annotations modules section for whichever information (the debug ID or filename) was provided in order to find the missing portion from the corresponding symbol file entry. 

These options should be sufficient in determining what symbols are missing from a crash report, but always feel free to reach out to our support team (by opening a support ticket, via the in-app widget, or emailing support@backtrace.io for additional assistance. 

Supply missing symbols 

In order for Backtrace to provide complete callstack resolution, the missing symbols must be supplied and the previously received crashes corresponding to those symbols must be reprocessed. How this is completed depends on users' symbol upload workflow. 

Manual upload

If you are uploading symbols manually, your next steps will be to retrieve and upload those symbols to Backtrace. After uploading missing symbols, corresponding crashes should be reprocessed in order for the symbols to apply. 

Custom symbol server integration

If you are using a custom symbol server, then your next step will be to verify that the symbol exists on the symbol server in the proper directory structure(ie <url>/<object_name>/<debug_id>/<file>). If it does not, then you know why it did not apply!

If it does exist, and reprocessing does not correct, then there are a few additional things to check. These items to check live in the symbol server section of your project settings (navigate from the main menu | project settings | symbols | symbol servers), unless otherwise noted. 

• White list
  • If this optional configuration is enabled, then a symbol's name must exist on the white list in order for Backtrace to download it from the symbol server.
• Skip list
  • Symbol name and debug ID combinations that fail to download from a custom symbol server are added to a skip list so that requests are not made for symbols which do not exist on the server. If a symbol that previously did not exist on the symbol server was uploaded, its name and debug ID may exist on this list. Removing the name and ID from the skiplist and reprocessing the crash object will allow the symbol to download. The symbols can removed from the skip list tab of the symbol server entry, or via our command line tool, morgue. 
• Logs
  • The symbol server entry's logs tab can be searched for additional information about the symbol download attempts and results / error messages. Searching the logs for the symbol's debug ID is generally most efficient. Note that we'll log a 200 upon successful download of a symbol, and common errors logged for failed downloads include 404, 401, etc. 
• Misc 
  • Backtrace applies downloaded symbols to all threads, but only downloads new symbols from the faulting thread. If the missing symbol is in a nonfaulting thread, it will not be downloaded from a custom symbol server. To workaround this, a symbol can be manually downloaded from the symbol server, uploaded to Backtrace, and then the crash can be reprocessed. Alternatively, the crash can be resubmitted with the thread in question marked faulting (using a modifier) and later deleted as needed. Once the symbol is downloaded from the symbol server, it will apply to all threads of new or reprocessed crashes. 

Process Missing Symbols on a Non-Faulting Thread

For a number of reasons (storage / performance), Backtrace does not by default process symbols which are only relevant to the non-faulting thread.  However, the debugger does have a feature that will let you pull that information if it is relevant to your debugging.

First, there are a few prerequisites to enabling this functionality:

1. You must be an admin user
2. Your instance's "_BACKTRACE_PROMPT_MISSING_SYMBOLS" window variable must be set to true.  This can be changed by SSH into your coronerd instance and editing the .../coronerd/coronerd.conf file.
3. The system must detect missing symbols on the non-faulting thread when you open an error report in the debugger (this is checked / happens automatically)

If all of the above are true, you will see an icon/button show up on the top right corner of the debugger as shown below.

Pressing this button will bring up a window confirming whether or not you want to download + process the symbols that are missing.  Confirming this will kick off the process for you (it might take a few seconds to complete).

Note that in order for the above to work, you must have actually provided / uploaded the symbols in question.  If you haven't done so, then the missing symbols errors wont be resolved.  If that is the case, you can simply follow the guide outlined about around how to upload missing symbols etc...

 

Resolve missing symbols 

Once missing symbols are uploaded or obtained from a symbol server, and the relevant crashes are reprocessed, the callstack will resolve completely. 

If there are any questions or remaining issues, always feel free to reach out to our support team by opening a support ticket, via the in-app widget, or emailing support@backtrace.io .