After a server migration or similar work, the following is minimum verification that should be completed.
Startup/Service
- Confirm coronerd version
-
$ /opt/backtrace/sbin/coronerd -v 1.42.1
-
- Ensure that coronerd, backtrace-proxy, and backtrace-nginx can be started, stopped and queried for status via systemctl
-
$ systemctl status coronerd ● coronerd.service - Backtrace I/O server Loaded: loaded (/etc/systemd/system/coronerd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago Main PID: 18126 (monitor) Tasks: 104 (limit: 4915) CGroup: /system.slice/coronerd.service ├─18126 /opt/backtrace/sbin/coronerd: monitoring child 18127 ├─18127 /opt/backtrace/sbin/coronerd -f -c /etc/coronerd/coronerd.conf (... etc …)
-
- Coronerd service check
- Coronerd starts and stays running
- Coronerd is listening on the necessary ports/firewall is open
-
$ systemctl status coronerd ● coronerd.service - Backtrace I/O server Loaded: loaded (/etc/systemd/system/coronerd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago Main PID: 18126 (monitor) Tasks: 104 (limit: 4915) CGroup: /system.slice/coronerd.service ├─18126 /opt/backtrace/sbin/coronerd: monitoring child 18127 ├─18127 /opt/backtrace/sbin/coronerd -f -c /etc/coronerd/coronerd.conf (... etc …)
- Check logs, look for abnormal messages from coronerd
- Confirm /api/is_configured endpoint returns a literal “1”
-
# Look carefully - there will be no line break after the “1” $ curl https://yolo.sp.backtrace.io/api/is_configured 1
-
- Confirm coronerd-crash-report is running
-
$ sudo systemctl status coronerd-crash-report ● coronerd-crash-report.service - Coronerd Crash Report server Loaded: loaded (/etc/systemd/system/coronerd-crash-report.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago Main PID: 18197 (monitor) Tasks: 2 (limit: 4915) CGroup: /system.slice/coronerd-crash-report.service ├─18197 /opt/backtrace/bin/coroner: monitoring child 18198 └─18198 /opt/backtrace/bin/coroner -c /etc/coronerd/coroner.cf daemon -f Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io systemd[1]: Starting Coronerd Crash Report server... Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io coronerd-crash-report[18189]: * Starting Backtrace I/O database error reporting coroner Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io coronerd-crash-report[18189]: ...done. Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io systemd[1]: Started Coronerd Crash Report server.
-
Server requirements
- ulimit -n set to at least 131072
-
# 18172 is the pid of the main coronerd process (see above) $ cat /proc/18127/limits | grep "open files" | tr -s ' ' | cut -d " " -f 4 524288
-
- Check coronerd’s memory and cpu usage with top
- Check disk space usage
- Check for correct storage path
Error Submission Flow
- End to end error flow
- Submit and reprocess a test error
- Submit error
- Check for sane response
- Retrieve objid/txid
- Query coronerd for this objid
- Compare returned results
- Submit symbol file
- Reprocess and ensure symbolification
Symbolification
- Uploaded symbols count/list matches old server
- Missing symbols count/list matches old server
- If using custom symbol servers:
- Symbol server configs present
- Configs not labelled as “Backtrace Server”
- Whitelist/Blacklist/Skiplist brought over
- Failure/success counts look good
- Symbold not logging anything irregular in syslog/journalctl
- Force symbold to test a download of a known good symbol
General UI
- Coronerd/console updated in Help | About
- Triage view for each project loads correctly
- Visit each config applet, make sure all open correctly
Login/SSO
- User accounts brought over and able to login
- If using SSO
- Configuration matches - Option to config SSO in UI is present and doesn’t return errors when navigated to
- “Login via SSO” button available
- Test an SSO login
- Test invitation request and confirm both receipt of email and functioning of the link
Network/Endpoints/Certs
- Ensure has the same listeners set up
- Ensure that SSL config matches up on each SSL-configured listener
- Make sure proper DNS name is set up on the necessary endpoints
- Misc. Services
- Link sharing functioning (if applicable)
- For each installed service:
- Check its conf file in /etc/backtrace/<servicename>/
- Check for it in morgue service status
- Verify it’s listening on its necessary port
Workflows/Reports
- Workflow configurations ported over
- Workflow fire properly (verify that the new server isn’t having trouble talking to the third-party service)
- Links within notifications
- If applicable - test manual ticket creation
- If using reporting, make sure reports ported over
- Send a test report
Data
- All projects exist with identical tokens as old system
- All projects have identical error count post-migration
- Attribute indices ported over
- Deduplication rules ported over
- Retention settings ported over
- Sampling settings ported over