Integrating Apache Traffic Server
Overview
We will configure Apache Traffic Server (ATS) to use Backtrace's Invoker as a crash log helper. The workflow upon ATS crash will be:
- On startup, ATS will launch invoker, which will monitor the traffic_server process.
- On traffic_server crash, invoker will launch ptrace via a script which provides metadata.
- This script will, in turn, submit the crash file that ptrace generates to the coroner object store.
Prerequisites
- Apache Traffic Server (tested with version 6.2.0)
- Coroner Server running and accessible from the ATS machine. (See Coroner Server installation)
- Coroner Client package on the ATS machine. (See Coroner Client Installation)
- Backtrace Ptrace package installed on the ATS machine. (See Ptrace Installation)
- Backtrace Invoker package installed on the ATS machine:$ sudo apt-get install backtrace-invoker
- (Optional but recommended) - Install the morgue command-line queryingtool: (See: Morgue Installation)
Set Up Helper Scripts
invoker.sh
#!/bin/sh
exec /opt/backtrace/bin/invoker -t "/home/ats/backtrace.sh %p" $@
backtrace.sh
This will call ptrace
with any metadata you wish to gather, then submits it to coroner
. Here, we're extracting the version number via traffic_ctl
, but feel free to add any additional metadata that you need.
#!/bin/sh
## Change the following as needed
PROJECT=ats
TOKEN=ats
DM=/export/ats
CF=/etc/coronerd/coroner.cf
PATH=/opt/backtrace/bin:/usr/bin:/bin:$PATH
# Add commands to gather any metadata that you wish to pass to the ptrace invocation
VERSION=`/opt/ts/bin/traffic_ctl metric get proxy.node.version.manager.short|cut -d ' ' -f 2`
mkdir -p ${DM}/
ptrace --kv="version:$VERSION" $1 -O ${DM}/ats
if test "$?" == "0"; then
coroner -c $CF put -u ${PROJECT} ${TOKEN} ${DM}/*.btt
fi
Make sure that your traffic_server
processes can execute these scripts, and has permission to write to the destination folder in backtrace.sh
(referred to by the variable DM
)
$ chmod +x invoker.sh
$ chmod +x backtrace.sh
Configure ATS Crash Log Helper
Add or edit the following line in /etc/trafficserver/records.config
:
CONFIG proxy.config.crash_log_helper STRING /home/ats/invoker.sh
Verify invoker Launch
$ sudo /bin/trafficserver restart
$ ps aux | grep invoker
nobody 11616 0.0 0.1 4336 1396 ? T 16:43 0:00 /opt/backtrace/bin/invoker -t /home/ats/backtrace.sh %p --syslog --wait --host x86_64-unknown-linux-gnu
Test
Let's send a SIGABRT to traffic server...
$ sudo kill -SIGABRT
...and verify via the morgue tool that the number of crashes for that project has increased.
>morgue list myproj
...
...
Occurrences: 1 (100.00%)
Troubleshooting
Launch backtrace.sh manually
Try launching backtrace.sh
manually to make sure there are no errors:
$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
If there are any errors, continue with the troubleshooting steps below to verify the correct installation of Backtrace components and proper permissions.
Verify ptrace installation
Running /opt/backtrace/bin/ptrace
on a process you own should generate a .btt file:
$ ps
PID TTY TIME CMD
2291 pts/7 00:00:00 bash
11709 pts/7 00:00:00 ps
$ /opt/backtrace/bin/ptrace 2291
/home/test/bash.2291.1461078102.btt
$ ls *.btt
bash.2291.1461078102.btt
If not, verify your ptrace installation: Ptrace Installation
Verify coroner installation
Make sure that you are able to send snapshots to the server via coroner:
$ coroner -c put project1 project1_token bash.2291.1461078102.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
If not, verify your coroner client installation: Coroner Client Installation
Launch backtrace.sh manually
Try launching backtrace.sh
manually to make sure there are no errors:
$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
Make sure ATS has permission to run invoker.sh
$ chmod +x /path/to/invoker.sh
Make sure backtrace.sh is being invoked and generating the .btt file
If you comment out the last 3 lines of backtrace.sh
, you can verify that the invoker
is running the script and generating the .btt file in the desired location.