Skip to content
ricochet

Health & readiness

This page covers the two health endpoints on the server’s ops listener, and how to wire them into a supervisor, a load balancer or a monitoring check.

The ops listener

Section titled “The ops listener”

The server runs a plain HTTP listener, separate from its libp2p protocols, for operators. It is on by default and bound to loopback:

/etc/ricochet/config.yaml
ops:
  enabled: true
  bind: 127.0.0.1
  port: 9090
  readiness_timeout: 2s

At startup the server logs operator HTTP surface listening with the address, and whether metrics and pprof are mounted. A GET / lists every mounted path.

It stays on loopback because readiness details, metrics and the /ops/* views disclose internal state and peer IDs. To reach it from another machine, use an SSH tunnel:

Terminal window
ssh -L 9090:127.0.0.1:9090 ops@ricochet.example.com
curl -s http://127.0.0.1:9090/readyz

Set bind to another address only on a network you already trust, or put an authenticating proxy in front of it. If the server cannot bind the listener, it fails to start rather than run without a health check. Setting enabled: false removes the listener, and the server logs a warning that there is no health check, readiness or metrics endpoint.

All ops endpoints answer GET and HEAD only. The full list is in Ops HTTP endpoints.

/healthz: liveness

Section titled “/healthz: liveness”
Terminal window
curl -s http://127.0.0.1:9090/healthz

Returns 200 with status ("ok") and uptimeSeconds. It touches no dependency. A database outage does not fail it, so a supervisor watching it does not restart the process for a fault a restart cannot fix. If /healthz does not answer, the process is gone or wedged.

/readyz: readiness

Section titled “/readyz: readiness”
Terminal window
curl -s http://127.0.0.1:9090/readyz

Returns 200 when the server can serve, 503 when it cannot. The body is JSON:

Field Meaning
ready The verdict
draining Present and true while the server is shutting down
checks.postgres.ok Whether the database ping succeeded; error says why not
details.uptimeSeconds Process uptime
details.admission enabled, maxInFlight, maxPerPeer, inFlight, admittedTotal, shedTotal, waitedTotal, acquireTimeout
details.mailboxCache size and capacity of the mailbox cache
details.postgresPool totalConns, idleConns, acquiredConns, maxConns, emptyAcquireCount, canceledAcquire

/readyz fails (503) in exactly two cases:

  • the PostgreSQL ping fails or takes longer than readiness_timeout;
  • the server is draining for shutdown.

Saturation does not make it fail. A busy server is working, and taking it out of rotation would push its load onto the busiest remaining servers. The details block is there to answer “why is it slow” without changing the verdict.

The server also runs these checks every intervals.health_check_min and publishes the verdict to other servers. See Service discovery.

Draining behind a load balancer

Section titled “Draining behind a load balancer”

On shutdown the server first flips /readyz to 503 and logs ops surface draining — /readyz now reports unready. If ops.drain_delay is set, it keeps serving for that long so a load balancer polling /readyz sees the 503 and removes the instance before it stops answering:

/etc/ricochet/config.yaml
ops:
  drain_delay: 20s      # a couple of probe intervals
  shutdown_timeout: 5s

drain_delay defaults to 0, which skips the wait; that is right when nothing is probing. See Upgrades & draining for the full shutdown sequence.

The health check script

Section titled “The health check script”

The .deb installs /opt/ricochet/health_check.sh. It asks both endpoints and exits with Nagios-style codes:

Exit Output Meaning
0 OK: Ricochet is healthy /healthz and /readyz both pass
1 WARNING: Ricochet is up but not ready, then the /readyz body Up, but the database is unreachable or the server is draining
2 CRITICAL: Ricochet not responding at <url> /healthz does not answer

It reads http://127.0.0.1:9090 by default; set OPS in its environment if you moved the listener. Run it from cron:

Terminal window
*/5 * * * * /opt/ricochet/health_check.sh >/dev/null || logger -t ricochet -p daemon.err "health check failed"

Troubleshooting

Section titled “Troubleshooting”
  • /readyz returns 503 with checks.postgres.ok: false: read checks.postgres.error. Check the database is up and reachable with the server’s credentials. See Database setup.
  • /readyz is slow, then 503: the database answers slower than readiness_timeout. Look at details.postgresPool: an acquiredConns equal to maxConns means the pool is exhausted.
  • Connection refused on port 9090: the listener is disabled, bound elsewhere, or the server is not running. GET / on the right address lists the mounted paths.
  • 405 method not allowed: the endpoints accept only GET and HEAD.