Monitoring
This page shows how to see the state of a running server.
Status
lasboltctl status asks the server through the control socket:
sudo -u lasbolt lasboltctl statuslasboltd 0.1.0-dev (protocol 1)
state: unsealed
uptime: 1m38s
users: 1 (1 enabled owners)
registration: invite
public url: https://vault.example.orgFor a script, add --json:
sudo -u lasbolt lasboltctl --json status{"ok":true,"result":{"version":"0.1.0-dev","protocol":1,"uptime_seconds":57,"users":1,"owners":1,"state":"unsealed","registration":"invite","public_url":"https://vault.example.org"}}Exit codes: 0 ok, 5 cannot connect. A code of 5 means that the server is down or that the socket is not there. See lasboltctl.
Health check
The server has no /health or /metrics route. To check from outside, request the root page. A good server answers 303 and Location: /login:
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/303Use this for a probe of your monitoring tool or of the proxy. A probe does not check the database or the backups. Check those with the log and with backup list.
Logs
lasboltd writes structured lines (key=value) to standard error. Under systemd, the journal holds them:
journalctl -u lasboltd -f
journalctl -u lasboltd --since "1 hour ago" -p warningThe server does not log each request in production. It does not log secrets, query strings or bodies. The audit log of each user (Settings › Audit log in the web app) holds security events such as logins and admin actions. The proxy log holds requests.
What the log lines mean
| Level | Message | Meaning and action |
|---|---|---|
| info | listening | The server runs. The line shows the address and the key provider. |
| info | backups on | The schedule is on. The line shows the folder, the interval, the number to keep and sealed=true or false. |
| info | backup made | A backup is done. |
| info | ctl socket on | The control socket is ready. |
| info | peer udp port on | The UDP port for peer sync is open. |
| warn | backups are off (--backup-interval 0) | No backup is made. Turn them on, or make sure that something else does the job. |
| warn | open registration: anyone who can reach this server can create an account | The registration mode is open. Set it to invite or closed. |
| warn | --allow-registration is set, but the registration setting is not open | The old flag does nothing now. Remove it. |
| warn | dev mode: plain HTTP, insecure-static key provider, not for production | --dev is on. Never on a real server. |
| warn | insecure-static key provider in use: data at rest is NOT protected | The same. |
| warn | peer udp port is off: cannot bind it | The UDP port is in use or not allowed. Set --peer-udp off to silence it, or free the port. |
| warn | ctl: peer refused | On Linux, a process of another user tried the control socket. Check who. |
| error | scheduled backup failed | A backup did not work. Check the disk space and the folder rights. Run lasboltd backup now by hand to see the error. |
| error | backup failed the check and was deleted | The copy did not pass the check. The previous backups are still there. Investigate before you trust the schedule. |
| error | backup retention failed | The server could not remove old backups. Check the folder rights. |
| error | user purge | The job that removes deleted accounts failed. |
| error | lasboltd stopped | The program ended. The err field says why, for example a wrong passphrase or a port in use. |
What to watch
Set alerts for these:
- The service is down.
systemctl is-active lasboltd, or the probe above. - The newest backup is too old. Compare the date of the newest file in
lasboltd backup listwith the interval. - Disk space in the data folder and in the folder of the off-site copies.
- Errors in the journal with the messages
scheduled backup failedandbackup failed the check. - The certificate of the proxy ends soon.
A simple check of the age of the newest backup:
find /var/lib/lasbolt/backups -name 'lasbolt-*' -mmin -1560 | grep -q . || echo "no backup in 26 hours"