Watchdog
The SBN Media watchdog service - supervises, health-checks, and restarts the other services on a host.
Watchdog
The watchdog supervises all of the other SBN Media services on a host. It starts the services that host is configured to run, checks their health, restarts any that stop responding, and can stop services that use too much memory or CPU.
Overview
When SBN Media starts on a host, the watchdog starts first. It then starts and manages the other services on that host according to its configuration. It continuously health-checks each service and restarts a service that fails a check or exits unexpectedly. It can also control how many instances of a service run and how much memory or CPU each service may use.
- Runs: Always - one per host. It is the first service to start and manages the rest.
- Required: Yes. Without the watchdog the other services are not started or supervised.
- Depends on: NATS, which it uses to reach and monitor the services it manages.
Configuration
The watchdog is configured under the watchdog namespace. View the defaults with ./sbn-media config eject and set host-specific overrides in sbn-media.local.yaml.
Setting | Default | Description |
|---|---|---|
| empty | How many instances of each service to run on this host. A value of |
| empty | The minimum (and optional maximum) number of each service that should be running across the whole NATS cluster. The watchdog reports an error when the minimum is not met. |
| empty | A memory limit per service, in megabytes. A service that exceeds its limit is stopped and restarted. |
| empty | A CPU limit per service. A service that exceeds its limit is stopped and restarted. |
| 3 | How many readings in a row may exceed a memory or CPU limit before the watchdog acts, so a brief spike does not cause a restart. |
| 60000 | How often, in milliseconds, the watchdog checks service health. |
| 5000 | How long, in milliseconds, to wait for a service to answer a health check before treating it as unhealthy. |
| 10000 | How long, in milliseconds, to wait for NATS during a health check. |
Example:
After changing these settings in sbn-media.local.yaml, reload the configuration for them to take effect.
FAQ
Which services should I list under watchdog.host?
Only the services you want this host to run, and only services that the watchdog manages. Set a service to 0 to make sure it does not run on this host - the watchdog will stop it if it sees it running.
Why does a service keep restarting?
The most common cause is a memory or CPU limit set under watchdog.maxMemoryMb or watchdog.maxCPU. A service that repeatedly exceeds its limit is stopped and restarted. Check the limit for that service, and review its logs for the underlying reason.
Another reason may be that the service has a failure. Check the configuration or run the service with sbn-media service exec [service-name] -ld. That will tell you why the service crashes.
The dashboard reports that a service is missing across the cluster. What does that mean?
The watchdog.system section sets the minimum number of each service expected across the whole NATS cluster. If fewer than the minimum are running, the watchdog reports an error. Start the missing service on a host, or lower the minimum if it is set higher than you need.
Does the watchdog run on every host?
Yes. Each host runs its own watchdog, which manages the services on that host. The watchdog.system settings are how those watchdogs share a common expectation of what should be running across the cluster.
Related pages
- SBN Media Overview (
SBN-Media/overview) - Installing and Configuring SBN Media (
SBN-Media/installation)