Articles in this section

See more

Post migration verification steps

  • Updated

After a server migration or similar work, the following is minimum verification that should be completed. 

Startup/Service

  • Confirm coronerd version 
    • $ /opt/backtrace/sbin/coronerd -v 1.42.1
  • Ensure that coronerd, backtrace-proxy, and backtrace-nginx can be started, stopped and queried for status via systemctl 
    • $ systemctl status coronerd
● coronerd.service - Backtrace I/O server
   Loaded: loaded (/etc/systemd/system/coronerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago
 Main PID: 18126 (monitor)
    Tasks: 104 (limit: 4915)
   CGroup: /system.slice/coronerd.service
           ├─18126 /opt/backtrace/sbin/coronerd: monitoring child 18127
           ├─18127 /opt/backtrace/sbin/coronerd -f -c /etc/coronerd/coronerd.conf
           (... etc …)
  • Coronerd service check
    • Coronerd starts and stays running
    • Coronerd is listening on the necessary ports/firewall is open
    • $ systemctl status coronerd
● coronerd.service - Backtrace I/O server
   Loaded: loaded (/etc/systemd/system/coronerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago
 Main PID: 18126 (monitor)
    Tasks: 104 (limit: 4915)
   CGroup: /system.slice/coronerd.service
           ├─18126 /opt/backtrace/sbin/coronerd: monitoring child 18127
           ├─18127 /opt/backtrace/sbin/coronerd -f -c /etc/coronerd/coronerd.conf
           (... etc …)
  • Check logs, look for abnormal messages from coronerd
  • Confirm /api/is_configured endpoint returns a literal “1” 
    • # Look carefully - there will be no line break after the “1”
$ curl https://yolo.sp.backtrace.io/api/is_configured
1
  • Confirm coronerd-crash-report is running 
    • $ sudo systemctl status coronerd-crash-report
● coronerd-crash-report.service - Coronerd Crash Report server
   Loaded: loaded (/etc/systemd/system/coronerd-crash-report.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-11-12 01:41:25 UTC; 13h ago
 Main PID: 18197 (monitor)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/coronerd-crash-report.service
           ├─18197 /opt/backtrace/bin/coroner: monitoring child 18198
           └─18198 /opt/backtrace/bin/coroner -c /etc/coronerd/coroner.cf daemon -f

Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io systemd[1]: Starting Coronerd Crash Report server...
Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io coronerd-crash-report[18189]:  * Starting Backtrace I/O database error reporting coroner
Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io coronerd-crash-report[18189]:    ...done.
Nov 12 01:41:25 yolo01.linode-nj1.backtrace.io systemd[1]: Started Coronerd Crash Report server.

Server requirements 

  • ulimit -n set to at least 131072 
    • # 18172 is the pid of the main coronerd process (see above)
$ cat /proc/18127/limits | grep "open files" | tr -s ' ' | cut -d " " -f 4
524288
  • Check coronerd’s memory and cpu usage with top
  • Check disk space usage
  • Check for correct storage path

Error Submission Flow

  • End to end error flow
    • Submit and reprocess a test error
    • Submit error
    • Check for sane response
    • Retrieve objid/txid
    • Query coronerd for this objid
    • Compare returned results
    • Submit symbol file
    • Reprocess and ensure symbolification

Symbolification

  • Uploaded symbols count/list matches old server
  • Missing symbols count/list matches old server
  • If using custom symbol servers:
    • Symbol server configs present
    • Configs not labelled as “Backtrace Server”
    • Whitelist/Blacklist/Skiplist brought over
    • Failure/success counts look good
    • Symbold not logging anything irregular in syslog/journalctl
    • Force symbold to test a download of a known good symbol

General UI

  • Coronerd/console updated in Help | About
  • Triage view for each project loads correctly 
  • Visit each config applet, make sure all open correctly

Login/SSO

  • User accounts brought over and able to login
  • If using SSO
    • Configuration matches - Option to config SSO in UI is present and doesn’t return errors when navigated to
    • “Login via SSO” button available
    • Test an SSO login
  • Test invitation request and confirm both receipt of email and functioning of the link

Network/Endpoints/Certs

  • Ensure has the same listeners set up
  • Ensure that SSL config matches up on each SSL-configured listener
  • Make sure proper DNS name is set up on the necessary endpoints
  • Misc. Services
  • Link sharing functioning (if applicable)
  • For each installed service:
  • Check its conf file in /etc/backtrace/<servicename>/
  • Check for it in morgue service status
  • Verify it’s listening on its necessary port

Workflows/Reports

  • Workflow configurations ported over
  • Workflow fire properly (verify that the new server isn’t having trouble talking to the third-party service)
  • Links within notifications
  • If applicable - test manual ticket creation
  • If using reporting, make sure reports ported over
  • Send a test report

Data

  • All projects exist with identical tokens as old system
  • All projects have identical error count post-migration
  • Attribute indices ported over
  • Deduplication rules ported over
  • Retention settings ported over
  • Sampling settings ported over
 

Related articles