# Monitoring hubs
A **hub** watches one project's fleet from your machine: the servers its `deploy.json` declares, the sites (targets)
on them, their databases and logs, and what each application says about itself. One hub per project --
`~/hubs/my.oviepro` for every client of my.oviepro, `~/hubs/mmoframework.fr` for the framework's site -- each with
its page, its command line and its history.
A hub declares nothing new about the fleet. Which machine hosts which client, how to reach it, where each site lives
and which database it uses are already in `deploy.json`: a server added there is watched at the next collection, and
it is reached exactly as a deployment reaches it (the project's `deploy.ssh.config`, your ssh agent, its
`deploy.credentials.json`).
**A hub only reads.** Every probe and every tool runs a fixed, read-only command; the page never sends one. Deploying,
rolling back, restoring stay `php scripts/admin.php` in the project, with their plan, confirmation and journal.
See it live: the library's demo, `http://localhost/MMO/master/libs/admin/` -- a made-up fleet of twelve clients on
three servers (`demo/src/DemoFleet.php` plays them) and this machine, for real.
## Making a hub
```bash
php libs/admin/scripts/new-hub.php ~/hubs/my.oviepro --project=~/dev/src/my.oviepro
cd ~/hubs/my.oviepro
php hub.php check # reads hub.json and the declarations, reaches every server once
php hub.php collect # every probe, now
php hub.php serve # the page, on http://127.0.0.1:/
php hub.php cron # the crontab line collecting at the hub's interval
```
`new-hub.php` options: `--project=` (its `deploy.json`, and the name of its `mmo.json`), `--declarations=a,b` (several
`deploy.json`), `--name=`, `--title=`, `--description=`, `--port=`, `--link=Label=url` (repeatable, shown in the page's
footer), `--dry-run`, `--force`. Run again, it adds what is missing and keeps the rest, `hub.json` included. Each hub
gets a port of its own (from its name), so several can be served at once.
The hub's folder:
| file | what it is |
|---|---|
| `hub.json` | what is watched and how (below) |
| `hub.credentials.json` | what the probes need that a deployment does not (mode 0600, git-ignored): copy `hub.credentials.example.json` |
| `data/` | what the hub saw: `latest.json`, `series/{subject}/{day}.jsonl`, `incidents.json` |
| `hub.php`, `www/index.php`, `bootstrap.php` | the command line, the pages, what both load |
| `www/bin/` | the theme mmo.client compiles the first time a page is asked for |
### Serving it
`php hub.php serve` runs the pages as **you**: the "Collect now" button and the tools reach the servers with your ssh
agent. Apache (the hub's `.htaccess`) runs it as Apache's user, which has neither your agent nor the right to read the
projects' credentials files: there, set `HubController`'s `canCollect` to `false` in `bootstrap.php` and let cron
collect -- and `www/bin` must be writable by Apache's user, which compiles the theme into it. Either way the api answers
this machine only (`localOnly`).
## A project's own hub
A project and all its variants have **one hub**, and it can be the project itself: its local site, in dev mode, shows
the hub's pages beside its Développeur menu (its boards, its checklists) -- watching its production servers and
releasing from the same place.
```bash
php libs/admin/scripts/new-hub.php ~/dev/src/my.oviepro --embed [--dry-run]
```
writes in the project `hub.json` (reading its `deploy.json`, its readings in `hub-data/`, linking its boards, its
checklists and the root hub's page of the project) and `hub.php` (the command line on the project's bootstrap:
`collect`, `status`, `cron`...), and prints what its `configs/dev.json` gets (`HubScaffold::EmbeddedConfig`):
`HubController` routed with `"canCollect": false` -- Apache has no ssh agent: `php hub.php cron` collects as you --
and `{MMO\admin}/monitoring/pages` after its theme's own pages. The pages are then `/hub`, `hub/servers`,
`hub/sites`... (the menu's « Supervision » group), behind the project's login; `hub-data/` and `hub.credentials.json`
stay out of git. The root hub links each project to its hub (`MMO\helpers\Project::$hub`).
## hub.json
```json
{
"name": "my.oviepro",
"title": "my.oviepro · fleet",
"declarations": ["~/dev/src/my.oviepro/deploy.json"],
"interval": 300,
"retention": 30,
"data": "{hub}/data",
"port": 8786,
"probes": {
"database": {"engine": "mongodb", "uri": "mongodb://127.0.0.1:27017"},
"logs": {"files": ["{folder}/shared/logs/*.log"]},
"application": {"path": "api/monitoring/metrics.json"}
},
"rules": [{"metric": "disk.*.usedPercent", "warning": 75, "critical": 90}],
"notify": [{"type": "desktop", "level": "critical"}],
"exclude": ["old-client"],
"links": {"Board": "http://localhost/my.oviepro/developer/boards"}
}
```
Paths may start with `~/`, `{hub}` or `{tmp}`; a relative path is the hub's. `//` comments are allowed. Also:
`includeDisabled` (watch the targets declared `"enabled": false`), `credentials` (another credentials file),
`transport` (the class asking the sites, the demo's `DemoTransport`).
## Probes
Each probe runs one script per subject (sections marked `@@name`, parsed here), so a server across the world costs one
round trip. A server is reached once per collection: when it does not answer, it is critical and its sites' shell
probes are `unknown` without trying again -- the http probe still runs, from here.
| probe | on | reads | metrics |
|---|---|---|---|
| `system` | server | `/proc`, `df`, `ps`, `systemctl` | `load.1/5/15`, `load.perCpu`, `cpu.usedPercent`, `cpu.iowaitPercent`, `memory.usedPercent`, `swap.usedPercent`, `disk.{mount}.usedPercent/freeGb/inodesPercent`, `disk.maxUsedPercent`, `net.rxKbps/txKbps`, `processes`, `uptime.days`, `reboot.required`, `services.failed` |
| `http` | site | its `url` + `healthPath`, expecting `healthExpect` (deploy.json), from here | `http.code`, `http.timeMs`, `http.ttfbMs`, `http.sizeKb`, `tls.daysLeft` |
| `release` | site | `current`, `releases/`, the journal, the lock, `du`, `backups/` | `release.count`, `release.locked`, `release.ageDays`, `size.{releases,shared,backups}Mb`, `backup.count`, `backup.ageHours` |
| `database` | site with a `database` | `mongosh` (or `mysql`, `vars.dbEngine` per target) on its server | `db.dataMb`, `db.storageMb`, `db.indexMb`, `db.objects`, `db.collections`, `db.connections.*`, `db.uptimeDays`, `db.slowOps`, `db.opsPerSec` / `db.queriesPerSec` |
| `logs` | site | its log files, from where the last reading stopped | `logs.newErrors`, `logs.newFatal`, `logs.sizeMb`, `logs.files` |
| `application` | site with a token | its `MetricsController` | the application's, prefixed `app.` |
What a probe cannot read (`ps` on a webspace) is left out -- no metric rather than a wrong 0. Cumulative numbers are
**counters**: two readings make a rate (`db.ops` → `db.opsPerSec`). Every public property of a probe is an option in
`probes` (`enabled`, `timeout`, `sample`, `services`, `files`, `errors`, `fatal`...). A probe of your own extends
`MMO\admin\monitoring\aProbe` and is named in `probes` with its `class`.
## Rules and incidents
Rules judge the numbers (the defaults: Settings on the page, `Rules::DEFAULTS`). A hub's rule for the same metric and
probe replaces the default, `"off": true` removes it, `"subjects"` keeps it to some servers or sites, `"below": true`
judges what is too low (days left on a certificate).
An **incident** opens the first time a metric crosses a rule (or a probe fails on its own: a 502, a server out of
reach), stays open while it does -- one incident, however many readings -- and resolves itself at the first reading
where it does not. It can be acknowledged (still open, no longer news). Opening, getting worse and resolving are told
through `notify`:
- `{"type": "desktop"}` -- `notify-send` on this machine;
- `{"type": "command", "command": "..."}` -- the event in its environment (`HUB_EVENT`, `HUB_SUBJECT`, `HUB_LEVEL`,
`HUB_MESSAGE`, `HUB_JSON`), never in its text;
- `{"type": "webhook", "url": "..."}` -- a JSON POST `{text, events}` (Slack, Mattermost, Discord, Google Chat);
- `{"type": "mail", "to": "..."}` -- php's `mail()`.
`"level": "critical"` keeps a channel for what is critical.
## The application's own metrics
A site says what no server metric can -- its users, its documents, its jobs waiting -- through mmo.admin's
`MetricsController`, declared in the site's routes (before its pages) and configured on each server:
```json
"MMO\\admin\\controllers\\MetricsController": {
"tokenHash": "",
"providers": ["MMO\\admin\\monitoring\\providers\\AnalyticsProvider", "Project\\metrics\\UsersMetrics"]
}
```
`php hub.php token` makes the token (for `hub.credentials.json`: `{"*": {"metricsToken": "..."}}`) and its hash (for
the site's `shared/configs/prod.json`): neither the repository nor the server holds what opens the route. The token
travels in `X-Hub-Token`. Without `tokenHash` the route answers 404.
It answers php and its opcache, the packages and their versions, the release, and every provider's numbers: a class
implementing `MMO\admin\monitoring\iMetricsProvider` (`collect(Metrics $metrics)`: `add`, `counter`, `fact`, `check`).
`AnalyticsProvider` brings today's mmo.metrics headline numbers (visitors, visits, page views, users, conversions,
errors), so the hub compares the clients' use of the application side by side.
## The pages and the api
A hub is an mmo.client application: its pages are mmo.admin's `{MMO\admin}/monitoring/pages/hub/*.phtml`, served by
mmo.client's `PageController` in the `` shell (the menu, the breadcrumb, the dark mode, the command palette),
drawn on the server by `MMO\admin\monitoring\HubPages` with the design system's classes (`.widgets`, `article.card`,
`table.data`, `.status-tag`) and `` for every curve. In French where the browser is (`src/i18n/fr.php`).
| page | what it shows |
|---|---|
| `hub/overview` | the fleet's tiles, the open incidents, a card per server (gauges, last day, its sites), a row per site |
| `hub/servers`, `hub/server?name=` | each server's numbers; one server's curves (cpu, load, memory, disks, network, processes), disks, processes, services, sites, tools, incidents |
| `hub/sites`, `hub/site?name=` | the sites (filtered by `?q=`); one site's checks, release, answer, database, curves (response time, errors, database, space, the application's), last error lines, largest collections, deployments, tools, incidents |
| `hub/databases` | every site's database side by side |
| `hub/incidents` | the open ones, or every one (`?all=1`) |
| `hub/deployments`, `hub/settings` | what was deployed; the probes, rules and channels as read |
`?range=` (21600, 86400, 604800, 2592000) sets the period of a page's curves. Three elements of mmo.admin
(`src/customElements/`, added to the theme by its bootstrap) do what a page does: `` (when it was
collected, « Collect now », the page reloaded every minute while it is in view), `` (acknowledge an
incident, its reason asked), `` (the read-only tools, what they printed). They call `HubController`'s
api: `GET api/hub.json`, `GET api/hub/servers/{name}.json` and `api/hub/targets/{name}.json` (`?range=&points=`),
`GET api/hub/series.json?subjects=&metrics=`, `GET api/hub/incidents.json`,
`POST api/hub/incidents/{id}/acknowledge.json`, `POST api/hub/collect.json`, `GET api/hub/deployments.json`,
`POST api/hub/tools/{tool}.json {subject}`, `GET api/hub/settings.json`.
An application of your own can show the same pages: `{MMO\admin}/monitoring/pages` in its theme's `pages`,
`HubController` (with its `hub`) in its routes.
## Tools
Read-only commands, written in `MMO\admin\monitoring\Tools`, never sent: on a server `processes`, `memory`, `disks`,
`connections`, `services`, `failed`, `errors` (the journal), `logins`, `updates`, `uptime`; on a site `releases`,
`space`, `logs`, `backups`, `journal`, `files`, `php`, `http`, `database`.
`php hub.php tool --subject=ovh1 --tool=processes`.
## The command line
`php hub.php` with: `collect [--subjects=] [--probes=] [-q]`, `status`, `incidents [--all]`, `ack --id= [--note=]`,
`watch [--interval=] [--times=]`, `tool --subject= [--tool=]`, `check`, `serve [--port=]`, `cron`, `token`, `prune`.
`-m` answers in JSON. `collect` and `status` end with 1 when something is critical.