path: root/docs/hub.md

# Hub

Zenserver's hub mode runs as a coordinator that manages multiple storage server instances on
a single host. Rather than serving storage requests directly, the hub listens for provision
and deprovision requests from an orchestrator and spawns dedicated zenserver child processes
on demand -- one per *module*. Each module is identified by an alphanumeric string (typically
associated with a content plugin or project). Each instance gets its own TCP port and data
directory under the hub's data directory.

Typical deployments:

- **Build farm**: build agents on the same host each receive an isolated zenserver instance
  for their build, provisioned by the build orchestration system.
- **Shared team cache**: an orchestrator provisions an instance per project or team and
  deprovisions it after a period of inactivity.

The hub handles the full lifecycle: hydrating new instances from a shared storage source,
monitoring running processes, handling crashes, and cleaning up deprovisioned instances
automatically.

## Instance Lifecycle

Each module slot progresses through a series of states managed by the hub.

```mermaid
stateDiagram-v2
    [*] --> Unprovisioned
    Unprovisioned --> Provisioning : Provision
    Provisioning --> Provisioned : ready
    Provisioning --> Unprovisioned : failed
    Provisioned --> Hibernating : Hibernate
    Provisioned --> Deprovisioning : Deprovision / timeout
    Provisioned --> Crashed : process exited
    Hibernating --> Hibernated : stopped
    Hibernating --> Provisioned : failed
    Hibernated --> Waking : Wake
    Hibernated --> Deprovisioning : Deprovision / timeout
    Waking --> Provisioned : ready
    Waking --> Hibernated : failed
    Deprovisioning --> Unprovisioned : done
    Crashed --> Recovering : watchdog
    Crashed --> Deprovisioning : Deprovision
    Recovering --> Provisioned : success
    Recovering --> Unprovisioned : failed
```

**Stable states:**

- **Unprovisioned** - no process running; the slot is ready for a new provision request.
- **Provisioned** - process running and serving requests. The watchdog monitors activity and
  deprovisions the instance after a configurable inactivity timeout (see
  [Watchdog Tuning](#watchdog-tuning)).
- **Hibernated** - process stopped, data directory preserved. The instance can be woken
  quickly without re-hydrating. The watchdog deprovisions (and deletes data) after a
  configurable inactivity timeout (see [Watchdog Tuning](#watchdog-tuning)).
- **Crashed** - process exited unexpectedly. The watchdog attempts an in-place restart
  automatically.

Transitioning states (`Provisioning`, `Hibernating`, `Waking`, `Deprovisioning`, `Recovering`)
are transient and held exclusively by one operation at a time. If hibernation fails (the
process cannot be stopped cleanly), the instance remains Provisioned.

**Hibernation vs deprovision:** hibernating stops the process but keeps the data directory
intact, allowing fast restart on the next Wake. Deprovisioning stops the process and deletes
all instance data. Explicit deprovision requests are always honoured; the watchdog timeout
path always deprovisions rather than hibernates.

## The Watchdog

The hub runs a background watchdog thread that manages instance lifecycle automatically.

**Cycle:** every `cycleintervalms` milliseconds (default 3 s) the watchdog:

1. Refreshes machine-level metrics (disk usage, system memory) used for provisioning limit
   checks.
2. Iterates over all active instance slots, spending at most `cycleprocessingbudgetms`
   (default 500 ms) per cycle with an `instancecheckthrottlems` (default 5 ms) pause between
   instance checks.

**Per-instance checks:**

For each **Provisioned** instance:
- Verifies the process is still running. If not, transitions to Crashed.
- Checks for inactivity. The check begins `inactivitycheckmarginseconds` (default 1 min)
  before the full timeout -- with a 10 min timeout, checking begins once 9 min of inactivity
  have elapsed. It makes a short HTTP request to the instance's
  `/stats/activity_counters` endpoint. Activity is any client storage request processed by
  the instance. If the activity counter increased since the previous check, the inactivity
  timer resets. If not, and the full `provisionedinactivitytimeoutseconds` has elapsed, the
  instance is automatically deprovisioned.

For each **Hibernated** instance:
- Checks elapsed time since last activity. No HTTP request is needed (process not running).
  After `hibernatedinactivitytimeoutseconds` (default 30 min), the instance is deprovisioned
  and its data deleted.

For each **Crashed** instance:
- Attempts an in-place restart (Recovering state) using the existing data directory, without
  re-hydrating. On success, the instance returns to Provisioned and an upstream notification
  is sent. On failure, the instance is deprovisioned.

**Summary with default settings:**

| Event | Time |
|---|---|
| Watchdog cycle | every 3 s |
| Activity check fires | 9 min after last activity |
| Provisioned auto-deprovision | 10 min after last activity |
| Hibernated auto-deprovision | 30 min after last activity |

## Running the Hub

```
zenserver hub [options]
```

For non-trivial deployments, pass a Lua configuration file rather than command-line flags:

```
zenserver hub --config /etc/zen/hub.lua
```

The hub's own config file is separate from the per-instance config (see
[Instance Management](#instance-management)).

## Configuration Reference

Options can be set via Lua config file or command-line flags. Lua values take precedence when
both are provided.

For general server options (listening port, data directory, logging, HTTP server selection)
that apply to zenserver in all modes, see the zenserver configuration documentation.

---

### Core Server Options

These generic options apply to zenserver in all modes, including hub.

| CLI flag        | Lua key                    | Description |
|-----------------|----------------------------|-------------|
| `--config`      | _(CLI only)_               | Path to the Lua configuration file for the hub. |
| `--data-dir`    | `server.datadir`           | Root directory for hub data, logs, and instance subdirectories. |
| `--port`        | `network.port`             | TCP port the hub listens on. |
| `--http`        | `network.httpserverclass`  | HTTP server implementation (`httpsys` or `asio`). |
| `--dedicated`   | `server.dedicated`         | Dedicated server mode: disables port probing and allocates more resources. |
| `--no-sentry`   | `server.sentry.disable`    | Disable Sentry crash reporting. |

---

### Instance Management

Controls the child storage server processes spawned by the hub.

Each instance stores its data in `<data-dir>/servers/<moduleid>/`. Instance directories
persist across hibernation and are removed only on deprovision.

| CLI flag                           | Lua key                        | Default                              | Description |
|------------------------------------|--------------------------------|--------------------------------------|-------------|
| `--hub-instance-base-port-number`  | `hub.instance.baseportnumber`  | `21000`                              | First port in the instance port pool. Instances are assigned ports sequentially from this base. Ports are not guaranteed to be stable across deprovision/re-provision cycles; a module may receive a different port after being deprovisioned and re-provisioned. |
| `--hub-instance-limit`             | `hub.instance.limits.count`   | `1000`                               | Maximum simultaneously provisioned instances. Provision requests are rejected once this limit is reached. |
| `--hub-instance-http`              | `hub.instance.http`            | `httpsys` (Windows), `asio` (Linux/macOS) | HTTP server implementation for child instances. On Windows, use `asio` if the hub runs without elevation and no URL reservation covers the instance port range. |
| `--hub-instance-http-threads`      | `hub.instance.httpthreads`     | `0`                                  | HTTP connection threads per child instance. `0` uses hardware concurrency. |
| `--hub-instance-corelimit`         | `hub.instance.corelimit`       | `0`                                  | Concurrency limit for child instances. `0` is automatic. |
| `--hub-instance-config`            | `hub.instance.config`          | _(none)_                             | Path to a Lua config file passed to every spawned child instance. Use this to configure storage paths, cache sizes, and other storage server settings. See the zenserver configuration documentation. |

---

### Resource Limits

The hub checks machine resources before accepting a provision request. If any enabled limit
is exceeded, the request is rejected. Limits are refreshed at the start of each watchdog
cycle.

Setting a limit to `0` disables it. Byte and percent limits are independent -- either can
trigger a rejection.

| CLI flag                                | Lua key                                 | Default | Description |
|-----------------------------------------|-----------------------------------------|---------|-------------|
| `--hub-provision-disk-limit-bytes`      | `hub.instance.limits.disklimitbytes`    | `0`     | Reject provision if used bytes on the filesystem volume containing the data directory exceeds this value. |
| `--hub-provision-disk-limit-percent`    | `hub.instance.limits.disklimitpercent`  | `0`     | Reject provision if used space on the filesystem volume containing the data directory exceeds this percentage of its total capacity (0-100). |
| `--hub-provision-memory-limit-bytes`    | `hub.instance.limits.memorylimitbytes`  | `0`     | Reject provision if used system-wide physical memory exceeds this many bytes. |
| `--hub-provision-memory-limit-percent`  | `hub.instance.limits.memorylimitpercent` | `0`    | Reject provision if used system-wide physical memory exceeds this percentage of total RAM (0-100). |

---

### Watchdog Tuning

| CLI flag                                                   | Lua key                                             | Default           | Description |
|------------------------------------------------------------|-----------------------------------------------------|-------------------|-------------|
| `--hub-watchdog-cycle-interval-ms`                         | `hub.watchdog.cycleintervalms`                      | `3 s (3000 ms)`   | Milliseconds between watchdog cycles. |
| `--hub-watchdog-cycle-processing-budget-ms`                | `hub.watchdog.cycleprocessingbudgetms`              | `500 ms`          | Maximum milliseconds spent processing instances per cycle. |
| `--hub-watchdog-instance-check-throttle-ms`                | `hub.watchdog.instancecheckthrottlems`              | `5 ms`            | Milliseconds to wait between successive instance checks within a cycle. |
| `--hub-watchdog-provisioned-inactivity-timeout-seconds`    | `hub.watchdog.provisionedinactivitytimeoutseconds`  | `10 min (600 s)`  | Seconds of inactivity before a running instance is automatically deprovisioned. `0` disables automatic deprovisioning for running instances. |
| `--hub-watchdog-hibernated-inactivity-timeout-seconds`     | `hub.watchdog.hibernatedinactivitytimeoutseconds`   | `30 min (1800 s)` | Seconds of inactivity before a hibernated instance is deprovisioned and its data deleted. `0` disables automatic deprovisioning for hibernated instances. |
| `--hub-watchdog-inactivity-check-margin-seconds`           | `hub.watchdog.inactivitycheckmarginseconds`         | `1 min (60 s)`    | Activity check window opens this many seconds before the provisioned inactivity timeout. With defaults, checking begins at 9 min and the hard timeout is at 10 min. |
| `--hub-watchdog-activity-check-connect-timeout-ms`         | `hub.watchdog.activitycheckconnecttimeoutms`        | `100 ms`          | Connect timeout in milliseconds for activity check requests. |
| `--hub-watchdog-activity-check-request-timeout-ms`         | `hub.watchdog.activitycheckrequesttimeoutms`        | `200 ms`          | Request timeout in milliseconds for activity check requests. |

---

### Hydration

Hydration pre-populates a new instance's data directory from a shared source before the
instance starts serving requests. This allows instances to start warm with a populated cache
rather than building up content from scratch.

If neither hydration option is set, the hub automatically creates a `hydration_storage`
directory under its own data directory and uses that as the file hydration source. This is
suitable for single-host deployments where instances share locally cached data.

`targetspec` and `targetconfig` are mutually exclusive.

| CLI flag                          | Lua key                      | Default                   | Description |
|-----------------------------------|------------------------------|---------------------------|-------------|
| `--hub-hydration-target-spec`     | `hub.hydration.targetspec`   | _(local path, see above)_ | Shorthand URI for the hydration source. Must use the `file://` prefix for file targets: `file:///absolute/path`. |
| `--hub-hydration-target-config`   | `hub.hydration.targetconfig` | _(none)_                  | Path to a JSON file specifying the hydration source. Supports `file` and `s3` backends. |

**File backend** (`hub.hydration.targetconfig` JSON):

```json
{
  "type": "file",
  "settings": {
    "path": "/data/hydration_storage"
  }
}
```

**S3 backend** (`hub.hydration.targetconfig` JSON):

```json
{
  "type": "s3",
  "settings": {
    "uri": "s3://bucket-name/optional/prefix",
    "region": "us-east-1",
    "endpoint": "https://custom-endpoint",
    "path-style": false
  }
}
```

S3 settings:

| Field | Required | Description |
|---|---|---|
| `uri` | Yes | S3 URI. Include a prefix path to isolate hub data within a shared bucket. |
| `region` | No | AWS region. Defaults to `us-east-1`; also reads `AWS_DEFAULT_REGION` / `AWS_REGION`. |
| `endpoint` | No | Custom endpoint URL for S3-compatible services (MinIO, Ceph, etc.). |
| `path-style` | No | Use path-style S3 URLs instead of virtual-hosted. Required by some S3-compatible services. Default `false`. |

S3 credentials are read from environment variables (`AWS_ACCESS_KEY_ID`,
`AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`). If `AWS_ACCESS_KEY_ID` is not set, the hub
falls back to IMDS instance credentials.

---

### Upstream Notifications

The hub can notify an external system when instance state changes. Notifications are disabled
when `endpoint` is empty.

| CLI flag                               | Lua key                               | Default  | Description |
|----------------------------------------|---------------------------------------|----------|-------------|
| `--upstream-notification-endpoint`     | `hub.upstreamnotification.endpoint`   | _(none)_ | URL that receives state-change notifications. |
| `--upstream-notification-instance-id`  | `hub.upstreamnotification.instanceid` | _(none)_ | Identifier sent in notification payloads to distinguish this hub from others. |

---

### Consul Integration

The hub can register with a Consul agent for service discovery and health reporting. Disabled
when `endpoint` is empty.

| CLI flag                             | Lua key                              | Default              | Description |
|--------------------------------------|--------------------------------------|----------------------|-------------|
| `--consul-endpoint`                  | `hub.consul.endpoint`                | _(none)_             | Consul agent URL. Example: `http://localhost:8500`. |
| `--consul-token-env`                 | `hub.consul.tokenenv`                | `CONSUL_HTTP_TOKEN`  | Name of the environment variable from which the Consul access token is read. |
| `--consul-health-interval-seconds`   | `hub.consul.healthintervalseconds`   | `10 s`               | Interval in seconds between Consul health checks. |
| `--consul-deregister-after-seconds`  | `hub.consul.deregisterafterseconds`  | `30 s`               | Seconds after which Consul deregisters the service if health checks stop passing. |

---

### Windows: Job Object

| CLI flag                 | Lua key              | Default | Description |
|--------------------------|----------------------|---------|-------------|
| `--hub-use-job-object`   | `hub.usejobobject`   | `true`  | Assigns child processes to a Windows Job Object configured to kill all children when the job handle closes. This ensures child processes are terminated if the hub exits unexpectedly. Disable only if the hub runs inside an existing Job Object that does not permit nested jobs. |

---

## Example Configurations

### Basic

Single-host deployment with S3 hydration, an instance count cap, and ASIO HTTP (avoids the
http.sys elevation requirement on Windows).

```lua
hub = {
  instance = {
    baseportnumber = 21000,
    limits = {
      count = 20,
      disklimitpercent = 90,
    },
    http = "asio",
    config = "/etc/zen/instance.lua",
  },

  hydration = {
    targetconfig = "/etc/zen/hydration.json",
  },
}
```

`/etc/zen/hydration.json`:

```json
{
  "type": "s3",
  "settings": {
    "uri": "s3://my-zen-cache/hub",
    "region": "us-east-1"
  }
}
```

### Full Production

Build-farm hub with S3 hydration, Consul registration, upstream notifications, resource
limits, and tuned watchdog.

```lua
hub = {
  -- Upstream notification: called on instance state changes
  upstreamnotification = {
    endpoint = "https://orchestrator.internal/zen/notifications",
    instanceid = "build-farm-hub-01",
  },

  -- Consul: service registration and health reporting
  consul = {
    endpoint = "http://localhost:8500",
    tokenenv = "CONSUL_HTTP_TOKEN",
    healthintervalseconds = 10,
    deregisterafterseconds = 30,
  },

  instance = {
    -- Port range starts at 21000 (hub assigns sequentially)
    baseportnumber = 21000,
    limits = {
      count = 100,

      -- Reject provisions when disk usage exceeds 90%
      disklimitpercent = 90,

      -- Reject provisions when system RAM usage exceeds 85%
      memorylimitpercent = 85,
    },

    -- Use asio to avoid http.sys elevation requirement for child instances
    http = "asio",
    httpthreads = 4,

    -- Config file applied to every child instance
    config = "/etc/zen/instance.lua",
  },

  -- Hydrate new instances from S3
  hydration = {
    targetconfig = "/etc/zen/hydration.json",
  },

  watchdog = {
    cycleintervalms = 3000,

    -- Deprovision running instances after 10 minutes of inactivity
    provisionedinactivitytimeoutseconds = 600,
    inactivitycheckmarginseconds = 60,

    -- Deprovision hibernated instances after 1 hour
    hibernatedinactivitytimeoutseconds = 3600,
  },
}
```

`/etc/zen/hydration.json`:

```json
{
  "type": "s3",
  "settings": {
    "uri": "s3://my-zen-cache/build-farm",
    "region": "us-east-1"
  }
}
```

---

## Deprecated Flags

These CLI flags still work but should not be used in new configurations.

| Deprecated flag | Current flag | Lua key |
|---|---|---|
| `--instance-id` | `--upstream-notification-instance-id` | `hub.upstreamnotification.instanceid` |
| `--hub-base-port-number` | `--hub-instance-base-port-number` | `hub.instance.baseportnumber` |