Morgue is a command line interface the to the Backtrace object store. It allows you to upload, download and issue queries on objects with-in the object store. It is also used as a mechanism to extract formatted responses to queries for integration with other environments.
Full details on how to download, install and use the morgue tool can be found in the readme file located at https://github.com/backtrace-labs/backtrace-morgue/blob/master/README.md
The information below is retained for reference and is up to date as of May 2018. Please note some new features have been introduced that are fully documented in the readme above.
Installation
It is recommended to install morgue using npm.
npm install backtrace-morgue -g
If you working from the repository, then instead use the following command.
npm install -g
This will install the morgue tool in your configured path. Refer to the morgue --help command to learn more.
Introduction
morgue
is a command-line interface to the Backtrace object store. It allows you to upload, download and issue queries on objects with-in the object store.
Usage
login
Usage: morgue login
The first step to using morgue is to log into a server.
$ morgue login http://localhost
User: sbahra
Password: **************
Logged in.
At this point, you are able to issue queries.
Describe
Usage: morgue describe
Requests a list and description of all metadata that can be queried against.
Example
$ morgue describe bidder uname
uname.machine: machine hardware name
uname.release: kernel release
uname.sysname: kernel name
uname.version: kernel version
Get
Usage: morgue get
Downloads the specified object from the Backtrace object store and prints to standard output. Optionally, output the file to disk.
The following options are available:
OptionDescription--resource=name Fetch the specified resource rather than the object.
Put
Usage: morgue put
Uploads object file to the Backtrace object store. User has the following options
- --compression=gzip|deflate uploaded file is compressed
- --kv=key1:value1,key2:value2,... upload key-values
- --form_data upload file by multipart/form-data post request
Modify
Usage: morgue modify
Modifies attributes of the given object in the manner specified. Both options below may be specified more than once.
- --set Set the given attribute=valuepair
- --clear Clear the given attribute
You are also able to modify multiple objects by specifying filters. The --filter, --age and --timearguments are accepted to modify.
Example
Set hostname to fqdn.example.com for object identifier 0.
$ morgue modify --set hostname=fqdn.example.com myproject 0
Set custom attribute reason to oom for all crashes containing memory_abort.
$ morgue modify --set reason=oom --filter=callstack,regular-expression,memory_abort
attachment
Usage: morgue attachment
Manage attachments associated with an object.
Filters
The filter option expects a comma-delimited list of the form .
The currently supported operations are equal, regular-expression, inverse-regular-expression, at-least, greater-than, at-most, less-than, contains, not-contains, is-set, and is-not-set.
Pagination
Pagination is handled with two flags
--limit=
controls the number of returned rows. --offset= controls the offset at which rows are returned, another way to put it is that it skips the first rows.
Aggregations
Aggregation is expressed through a myriad of command-line options that express different aggregation operations. Options are of form --=.
The * factor is used when aggregations are performed when no factor is specified or if an object does not have a valid value associated with the factor.
- --age Specify a relative timestamp to now. 1h ago, or 1d ago.
- --time Specify a range using Chrono.
- --unique provide a count of distinct values
- --histogram provide all distinct values
- --distribution provide a truncated histogram
- --mean calculate the mean of a column
- --sum sum all values
- --range provide the minimum and maximum values
- --count count all non-null values
- --bin provide a linear histogram of values
- --head provide the first value in a factor
- --tail provide the last value in a factor
- --object provide the maximum object identifier of a column
Sorting
Sorting of results is done with the stackable option --sort=. The term syntax is -.
- The optional - reverse the sort term order to descending, otherwise it defaults to ascending.
Multiple sort terms can be provided to break ties in case the previous referenced sort term has ties.
Example
Request all faults from application deployments owned by jdoe. Provide the timestamp, hostname, callstack and classifiers.
$ morgue list bidder --filter=tag_owner,equal,jdoe --select=timestamp --select=hostname --select=callstack --select=classifiers
*
#9d33 Thu Oct 13 2016 18:36:01 GMT-0400 (EDT) 5 months ago
hostname: 2235.bm-bidderc.prod.nym2
classifiers: abort stop
callstack:
assert ← int_set_union_all ← all_domain_lists ←
setup_phase_unlocked ← bid_handler_slave_inner ← bid_handler_slave ←
an_sched_process_task ← an_sched_slave ← event_base_loop ←
an_sched_enter ← bidder_slave ← an_sched_pthread_cb
#ef2f Thu Oct 13 2016 18:36:01 GMT-0400 (EDT) 5 months ago
hostname: 2066.bm-impbus.prod.nym2
classifiers: abort stop
callstack:
assert ← an_discovery_get_instances ← budget_init_discovery ←
main
#119bf Thu Oct 13 2016 18:36:01 GMT-0400 (EDT) 5 months ago
hostname: 2066.bm-impbus.prod.nym2
classifiers: abort stop
callstack:
assert ← an_discovery_get_instances ← budget_init_discovery ←
main
Request faults owned by jdoe, group them by fingerprint and aggregate the number of unique hosts, display a histogram of affected versions and provide a linear histogram of process age distribution.
$ morgue list bidder --age=1y --factor=fingerprint --filter=tag_owner,equal,jdoe --head=callstack --unique=hostname --histogram=tag --bin=process.age
823a55fb15bf697ba3041d736ade... ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁ 5 months ago
Date: Wed May 18 2016 18:44:35 GMT-0400 (EDT)
callstack:
assert ← int_set_union_all ← all_domain_lists ←
setup_phase_unlocked ← bid_handler_slave_inner ← bid_handler_slave ←
an_sched_process_task ← an_sched_slave ← event_base_loop ←
an_sched_enter ← bidder_slave ← an_sched_pthread_cb
histogram(tag):
8.20.4.adc783.0 ▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 1
unique(hostname): 1
bin(process.age):
7731 7732 ▆▆▆▆▆▆▆▆▆▆ 1
3b851ac1ab1421409159cc38edb2... ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁ 5 months ago
Date: Tue May 17 2016 17:28:26 GMT-0400 (EDT)
Tue May 17 2016 17:30:07 GMT-0400 (EDT)
callstack:
assert ← an_discovery_get_instances ← budget_init_discovery ←
main
histogram(tag):
4.44.0.adc783.1 ▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 2
unique(hostname): 1
bin(process.age):
23 24 ▆▆▆▆▆▆▆▆▆▆ 1
24 25 ▆▆▆▆▆▆▆▆▆▆ 1
Request faults for the last 2 years, group them by fingerprint, show the first object identifier in the group, sort the results by descending fingerprint, limit the results to 5 faults and skip the first 10 (according to sort order).
$ morgue list blackhole --age=2y --factor=fingerprint --object=fingerprint --limit=5 --offset=10 --sort="-;group"
fec4bfecf8e077cf44024f5668fa... ▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 2 years ago
First Occurrence: Tue Jan 12 2016 13:30:12 GMT-0500 (EST)
Occurrences: 360
object(fingerprint): 1c653d
fe7294a780a16e30b619e8d94a8a... █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 2 years ago
First Occurrence: Wed Oct 28 2015 11:30:47 GMT-0400 (EDT)
Last Occurrence: Wed Oct 28 2015 12:16:19 GMT-0400 (EDT)
Occurrences: 203
object(fingerprint): 1c23b3
fe5e0dda6cf0fb996a521dde4087... ▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1 year ago
First Occurrence: Tue Jun 14 2016 11:54:35 GMT-0400 (EDT)
Occurrences: 1
object(fingerprint): 2de5
fe46d9af7c65c084091fed51ef02... █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 2 years ago
First Occurrence: Tue Oct 27 2015 16:59:34 GMT-0400 (EDT)
Last Occurrence: Tue Oct 27 2015 20:05:30 GMT-0400 (EDT)
Occurrences: 3
object(fingerprint): 8f41
fdc0860ef6dfd3d0397b53043ab9... ▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1 year ago
First Occurrence: Tue Jun 07 2016 11:51:55 GMT-0400 (EDT)
Occurrences: 211
object(fingerprint): 1c1958
Request faults for the two years, group them by fingerprint, sum process.age, sort the results by descending sum of process.age per fingerprint, limit the results to 3 faults. Note here that 1 in -process.age;1 is the second operator (--sum) in this case.
$ morgue list blackhole --age=2y --factor=fingerprint --first=process.age --sum=process.age --limit=3 --sort="-process.age;1"
d9358a6fdb7eaa143254b6987d00... ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1 year ago
First Occurrence: Tue Sep 20 2016 21:59:46 GMT-0400 (EDT)
Last Occurrence: Tue Sep 20 2016 22:03:23 GMT-0400 (EDT)
Occurrences: 38586
sum(process.age): 56892098354615 sec
524b9f988c8ff9dfc1b3a0c71231... ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1 year ago
First Occurrence: Tue Sep 20 2016 22:01:52 GMT-0400 (EDT)
Last Occurrence: Tue Sep 20 2016 22:03:19 GMT-0400 (EDT)
Occurrences: 25737
sum(process.age): 37947233900547 sec
bffd05c6b745229fd1c648bbe2a7... ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1 year ago
First Occurrence: Tue Sep 20 2016 21:59:46 GMT-0400 (EDT)
Last Occurrence: Tue Sep 20 2016 22:03:01 GMT-0400 (EDT)
Occurrences: 20096
sum(process.age): 29630010305216 sec
Delete
Allows deleting objects.
Usage: morgue delete
Object IDs must be specified; they can be found in morgue list output. The object ID printed in the example above is 9d33.
The following options support partial deletes: --physical-only: Only delete the physical object; retain indexing. --crdb-only: Only delete the indexed data; requires physically deleted objects.
Flamegraph
Usage: morgue flamegraph [--filter=
Generate a flamegraph of callstacks of all objects matching the specified filter criteria. The --filter option behaves identically to the list sub-command. This functionality requires perl to be installed. To learn more about flamegraphs, please see http://www.brendangregg.com/flamegraphs.html.
Use --unique to only sample unique crashes. Use --reverse to begin sampling from leaf functions.
Report
Create and manage scheduled reports.
Usage: morgue report
Create
Usage: morgue report
Example:
$ morgue report MyProject create --rcpt=null@backtrace.io
--rcpt=list@backtrace.io --filter=environment,equal,prod
--title="Production Crashes weekly" --period=week
Repair
Usage: morgue repair
Repair a project's attribute database. For each corrupted pages of a project's attribute database, reprocess the affected objects (if possible). Once completed and successful, transition the database into normal mode.
Reprocess
Usage: morgue reprocess
Reprocess the project's objects. This command can be used to re-execute indexing, fingerprinting, and symbolification (where needed).
If a set of objects (or query) is specified, any values for --first and --last are replaced to match the object list. If no query, object list, or range is provided, all objects in the project are reprocessed.
Retention
Usage: morgue retention
Configure the retention policy for a given namespace, which can cover the coroner instance, or a specific universe or project.
Example
$ morgue retention clear a_project
success
$ morgue retention set blackhole --max-age=3600
$ morgue retention list
Project-level:
blackhole: max age: 1h
$
Sampling
Usage: morgue sampling
Retrieve the object sampling status, or reset it. Project is a required flag if fingerprint is specified.
Symbol
Usage: morgue symbol
Retrieve a list of uploaded symbols or symbol archives. By default, morgue symbol will return a summary of uploaded archives, available symbols and missing symbols. If archives is used, a list of uploaded, in-process and symbol processing errors are outputted. If list is used, then a list of uploaded symbols is returned. Ifmissing is used, then the set of missing symbols for the project are included.
Scrubber
Create, modify and delete data scrubbers.
Usage: morgue scrubber
Use --name to identify the scrubber. Use --regexp to specify the pattern to match and scrub. Use --builtin to specify a builtin scrubber, ssn, ccn, key and env are currently supported for social security number, credit card number, encryption key and environment variable. If --builtin=all in createsubcommand, all supported builtin scrubbers are created. --regexp and --builtin are mutually exclusive. Use --enable to activate the scrubber, 0 disables the scrubber while other integer values enable it.
Setup
Usage: morgue setup
If you are using an on-premise version of coronerd, use morgue setup to configure the initial organization and user. For example, if the server is backtrace.mycompany.com, then you would run morgue setup http://backtrace.mycompany.com. We recommend resetting your password after you enable SSL (done by configuring your certificates).
Nuke
Usage: morgue nuke --universe=
If you want to nuke an object and all of the dependencies of the object. Do not use this operation without making a back-up of your data.
Token
Usage: morgue token [create | list | delete] [--project=...] [--universe=...]
Create
Usage: morgue token create --project=
Capability can be any of: * symbol:post - Enable symbol uploads with the specified API token. * error:post - Enable error and dump submission with the specified API token. * query:post - Enable queries to be issued using the specified token.
Multiple capabilities can be specified by using --capability multiple times or using a comma-separated list.
List
Usage: morgue token list [--universe=...] [--project=...]
List API tokens in the specified universe, for all projects or a specified project.
Example
Allows you to perform queries on object metadata. You can perform either selection queries or aggregation queries, but not both at the same time.
Usage: morgue list
You may pass --verbose in order to get more detailed query performance data.
delete
Usage: morgue token delete
Delete the specified token by substring or exact match.
user
Usage: morgue user reset [--universe=...] [--user=...] [--password=...]
Modify users.
Currently, can only be used to reset user passwords. Prompts for user and password if either is not specified.
Tenant
Create isolated tenants for receiving error data and log in. Tenants provide namespace isolation. Users in one tenant are unable to interact with any objects outside of their tenant.
This is an enterprise feature and not enabled by default for self-serve customers. The tenant commands require superuser access.
Usage: morgue tenant
Examples
1.0 Create a Tenant
After logging into an object store as a superuser, we are able to simply create a tenant using the following command:
$ morgue tenant create testingxyz
Tenant successfully created at https://testingxyz.sp.backtrace.io
Wait a few minutes for propagation to complete.
Tenants are required to be contained with-in the same TLD. For example, a tenant of name X is expected to be contained in X.sp.backtrace.io.
After creating a tenant, you will probably need to invite an initial administrator user for the tenant. For that, please see invite sub-command listed below. You must use the --tenant option to invite an administrator to a particular tenant.
2.0 Delete a Tenant
After logging into an object store as a superuser, we are able to simply create a tenant using the following command:
$ morgue tenant delete testingxyz
Tenant successfully deleted.
Please note this is a destructive command from a configuration perspective. Unless you are maintaining backups, there is no way to restore your configuration data.
3.0 List Tenants
You can list existing tenants using the morgue tenant list command as below.
$ morgue tenant list
ID Tenant URL
1 test https://test.sp.backtrace.io
4 test1 https://test1.sp.backtrace.io
Invite
Invite new users into your system. Requires you to have logged in.
Usage: morgue invite
Examples
1.0 Invite a User
Below, we invite a new user into the tenant currently logged into (or the first tenant, if multiple exist). The default settings for the user are to use password authentication and have a member role.
$ morgue invite create user user@gmail.com
Invitation successfully created for user@backtrace.io
Sending e-mail...done
1.1 Invite a User as an Administrator
$ morgue invite create user user@gmail.com --role=admin
Invitation successfully created for user@backtrace.io
Sending e-mail...done
1.2 Invite a User into a Particular Tenant
$ morgue invite create user user@gmail.com --tenant=mystudio
Invitation successfully created for user@backtrace.io
Sending e-mail...done
2.0 List Pending Invitation
This will list invitations that have yet to be accepted or activated.
$ morgue invite list
Tenant Username Method Role Email Token
1 ashley2 password admin ashley2@backtrace.io f892200fa564...
1 jack1 password member jack@backtrace.io 39c1b80a7e00...
1 jack2 password member jack+2@backtrace.io c399bdf23873...
1 jack17131 password member jack+4512@backtrace.io 784d2a8ffe12...
1 jack25262 password member jack+24688@backtrace.io 97e306d3373a...
1 jack25629 password member jack+28155@backtrace.io ed02ceea2ba4...
1 jack28000 password member jack+3644@backtrace.io 3f87906bd5d9...
1 jack19468 password member jack+28771@backtrace.io 3c6b3a3aaf41...
1 jack15686 password member jack+4203@backtrace.io 78bd9cd127a8...
4 jack2268 password member jack+19325@backtrace.io 776c6d389f89...
4 jack20597 password member jack+24692@backtrace.io 48972737a85e...
4 jack4803 password member jack+30407@backtrace.io 4943913c86f3...
3.0 Delete an Invitation
Below, we demonstrate how to delete an invitation. We pass a token (or unique substring) for deletion.
$ morgue invite delete f8922
Invitation successfully deleted.