Object Store Installation
This guide goes over the installation and bring-up of coronerd, Backtrace's object-store for post-mortem assets.
If you are using Backtrace's hosted service, see Object store configuration.
Installation
- Follow the steps in Package Installation to install the Backtrace repo
Ubuntu/Debian
- If running Ubuntu 12.04, install libhwloc4:
$ apt-get install libhwloc4
- If running Ubuntu 14.04 or 16.04, install libhwloc5:
$ apt-get install libhwloc5
- To install coronerd using apt, run:
$ apt-get install backtrace-coronerd backtrace-gimli
RHEL
- To install coronerd using yum, run:
$ yum install backtrace-coronerd backtrace-gimli
System Configuration
The rest of this guide are preliminary system configuration needed before we start coronerd.
Increase file descriptor limits
- RHEL/CentOS and Ubuntu 14 (and other non-systemd-distros)
Edit /etc/security/limits.conf, and add the following:
* hard nofile 131072
* soft nofile 131072
root hard nofile 131072
root soft nofile 131072
Exit and re-login to the terminal for the changes to take effect.
- Ubuntu 16 and other systemd-based distros
Check /etc/systemd/system/coronerd.service and verify that the following line is present under the [Service] section:
LimitNOFILE=131072
If you needed to modify coronerd.service above, run:
systemctl daemon-reload
Increase max_map_count
echo "vm.max_map_count=16000000" /etc/sysctl.conf
sysctl -p
SSL
If you want to disable SSL on coronerd, see Disable SSL.
- If you want to use CA-trusted certificates, see CA-trusted certificates below.
- If you want to use self-signed certificates, see Self-signed certificates below.
CA-trusted certificates
First, copy your certificate chain file (or certificate file) and private key file to /etc/coronerd/ssl/.
For safety, make the private key have strict permissions:
$ chmod 600 /etc/coronerd/ssl/key.pem
$ ls -lptr /etc/coronerd/ssl/key.pem
-rw------- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/key.pem
Point coronerd to the certificate and key files in /etc/coronerd/coronerd.conf. Please note that all SSL objects must point to your certificate and key files.
Example:
{
//...
"console" : {
"path" : "/var/run/coronerd/coronerd.socket",
"bind" : {
"hostname" : "0.0.0.0",
"service" : "9040"
},
"backlog" : 16,
"ssl" : {
"certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
"key" : "/etc/coronerd/ssl/key.pem"
}
},
//...
"listener" : {
"write" : {
"http_bind" : [
{
"hostname" : "0.0.0.0",
"service" : "6097",
"concurrency" : 2000
}
],
"https_bind" : [
{
"hostname" : "0.0.0.0",
"service" : "6098",
"concurrency" : 2000
}
],
"threads" : 1,
"ssl" : {
"certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
"key" : "/etc/coronerd/ssl/key.pem"
}
},
"read" : {
"bind" : [
{
"hostname" : "0.0.0.0",
"service" : "4097"
}
],
"threads" : 1,
"ssl" : {
"certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
"key" : "/etc/coronerd/ssl/key.pem"
}
},
"http-console" : {
"bind" : [
{
"hostname" : "0.0.0.0",
"service" : "443",
"concurrency" : 1000
}
],
"columns" : {
//...
},
"threads" : 1,
"ssl" : {
"certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
"key" : "/etc/coronerd/ssl/key.pem"
}
}
}
}
If using a certificate file, add the certificate setting alongside the key setting. For more information, see Self-signed certificates below.
Self-signed certificates
A self-signed certificate does not exist in the web of trust which descends from a trusted root certificate authority (CA). Nonetheless, it can be used for the purposes of a secure channel if the self-signed certificate is manually set up to be considered a trusted certificate authority. Using self-signed certificates in this way will only affect the secure channels initiated by coroner and will not affect the secure channels and web-of-trust of the remainder of the system.
A self-signed certificate and key must be generated. Note that in the following command, the user-specified "Common Name" value must later match the host portion of the https://:port entry of the "write" configuration key of the desired universe section in coroner client's configuration. Generate the key (coronerd-key.pem) and certificate (coronerd-cert.pem) via the following command:
$ sudo mkdir /etc/coronerd/ssl
$ openssl req -nodes -new -x509 -keyout /etc/coronerd/ssl/coronerd-key.pem -out /etc/coronerd/ssl/coronerd-cert.pem
Generating a 2048 bit RSA private key
......................+++
......+++
writing new private key to '/etc/coronerd/ssl/coronerd-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:New York
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Backtrace I/O, LLC
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:127.0.0.1
Email Address []:support@backtrace.io$ ls -lptr /etc/coronerd/ssl/coronerd-key.pem /etc/coronerd/ssl/coronerd-cert.pem
-rw-rw-r-- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/coronerd-key.pem
-rw-rw-r-- 1 root root 1415 Nov 24 11:43 /etc/coronerd/ssl/coronerd-cert.pem
For safety, make the private key have strict permissions:
$ chmod 600 /etc/coronerd/ssl/coronerd-key.pem
$ ls -lptr /etc/coronerd/ssl/coronerd-key.pem
-rw------- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/coronerd-key.pem
The coroner-key.pem file should remain private and only accessible on the host(s) running coronerd. The coroner-cert.pem file is public and must be accessible on all hosts running the coroner client.
Step 2
coronerd must be configured to use the certificate and key files. Make sure that both files are accessible on the host which coronerd runs on. Then, in your coronerd configuration file, make sure the console, and write, read, and http-console sections under listener have SSL stanzas.
Example:
{
//...
"console" : {
"path" : "/var/run/coronerd/coronerd.socket",
"bind" : {
"hostname" : "0.0.0.0",
"service" : "9040"
},
"backlog" : 16,
"ssl" : {
"certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
"key" : "/etc/coronerd/ssl/coronerd-key.pem"
}
},
//...
"listener" : {
"write" : {
"http_bind" : [
{
"hostname" : "0.0.0.0",
"service" : "6097",
"concurrency" : 2000
}
],
"https_bind" : [
{
"hostname" : "0.0.0.0",
"service" : "6098",
"concurrency" : 2000
}
],
"threads" : 1,
"ssl" : {
"certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
"key" : "/etc/coronerd/ssl/coronerd-key.pem"
}
},
"read" : {
"bind" : [
{
"hostname" : "0.0.0.0",
"service" : "4097"
}
],
"threads" : 1,
"ssl" : {
"certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
"key" : "/etc/coronerd/ssl/coronerd-key.pem"
}
},
"http-console" : {
"bind" : [
{
"hostname" : "0.0.0.0",
"service" : "443",
"concurrency" : 1000
}
],
"columns" : {
//...
},
"threads" : 1,
"ssl" : {
"certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
"key" : "/etc/coronerd/ssl/coronerd-key.pem"
}
}
}
}
Step 3
Set the coroner client configuration file (coroner.cf) as noted in Configuration using self-signed certificates.
Troubleshooting
- Problem:
"error: cURL failure (SSL peer certificate or SSH remote key was not OK): SSL: certificate subject name 'XXX' does not match target host name 'YYY'"
This means that the certificate was generated with a "Common Name" field set to XXX, but that the client knows the server as YYY (i.e., uses a configuration "write" value of "https://YYY:port").
Solution 1:
Make sure that the client accesses the server via the name which is used as the "Common Name" in the certificate. This may require correcting DNS entries, IP addresses, and/or routes.
Solution 2:
Regenerate the certificate and key used by the coronerd server and have it use the new YYY hostname of the coronerd for the certificate's "Common Name".
Start coronerd
$ /etc/init.d/coronerd start
To verify that coronerd is running properly:
$ /etc/init.d/coronerd status
Create Organization and Admin User
Before accessing the coronerd object store via Web UI, we need to create the organization object and an initial admin user via the command-line morgue utility.
morgue
is installed by the NodeJS NPM utility. To install, simply run npm install backtrace-morgue -g. If you need NodeJS, see the Node.js website.
Launch `morgue setup, pointing to the URL of your coronerd instance. Follow the prompts to create the organization and initial admin user, then you'll be ready to continue setting up via the Web UI.
If using self-signed SSL certificates, pass the -k flag to morgue setup.
$ morgue setup https://coronerd.mydomain.com
Determining system state...unconfigured
Create an organization
We must first configure the organization that is using the object store.
Please provide a one word name for the organization using the object store.
For example, if your company name is "Appleseed Systems I/O", you could
use the name "appleseed". The name must be lowercase.
Organization name: testing-bt
Create an administrator
We must create an administrator user. This user will be used to configure
the server as well as perform system-wide administrative tasks.
Username: jdoe
E-mail address: jdoe@mydomain.com
Password: ************
Confirm password: ************