incremental dehydrate (#921)

- Feature: Incremental CAS-based hydration/dehydration replacing the previous full-copy approach
- Feature: S3 hydration backend with multipart upload/download support
- Feature: Configurable thread pools for hub instance provisioning and hydration
  `--hub-instance-provision-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation.
  `--hub-hydration-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation.
- Improvement: Hub triggers GC on instance before deprovisioning to compact storage before dehydration
- Improvement: GC status now reports pending triggers as running
- Improvement: S3 client debug logging gated behind verbose mode to reduce log noise at default verbosity
- Improvement: Hub dashboard Resources tile now shows total memory
- Improvement: `filesystemutils` moved from `zenremotestore` to `zenutil` for broader reuse
- Improvement: Hub uses separate provision and hydration worker pools to avoid deadlocks
- Improvement: Hibernate/wake/deprovision on non-existent or already-in-target-state modules are idempotent
- Improvement: `ScopedTemporaryDirectory` with empty path now creates a temporary directory instead of asserting

1 files changed, 26 insertions, 6 deletions

diff --git a/docs/hub.md b/docs/hub.md
index a6b68891b..52a6fa3be 100644
--- a/docs/hub.md
+++ b/docs/hub.md
@@ -61,9 +61,14 @@ are transient and held exclusively by one operation at a time. If hibernation fa
 process cannot be stopped cleanly), the instance remains Provisioned.
 
 **Hibernation vs deprovision:** hibernating stops the process but keeps the data directory
-intact, allowing fast restart on the next Wake. Deprovisioning stops the process and deletes
-all instance data. Explicit deprovision requests are always honoured; the watchdog timeout
-path always deprovisions rather than hibernates.
+intact, allowing fast restart on the next Wake. Deprovisioning triggers a GC cycle, then
+dehydrates the instance's state back to the configured backend, then deletes all local
+instance data. Explicit deprovision requests are always honoured; the watchdog timeout path
+always deprovisions rather than hibernates.
+
+**Idempotent operations:** hibernating an already-hibernated instance, waking an
+already-provisioned instance, or deprovisioning a non-existent module all return success
+without side effects.
 
 ## The Watchdog
 
@@ -162,6 +167,7 @@ persist across hibernation and are removed only on deprovision.
 | `--hub-instance-http`              | `hub.instance.http`            | `httpsys` (Windows), `asio` (Linux/macOS) | HTTP server implementation for child instances. On Windows, use `asio` if the hub runs without elevation and no URL reservation covers the instance port range. |
 | `--hub-instance-http-threads`      | `hub.instance.httpthreads`     | `0`                                  | HTTP connection threads per child instance. `0` uses hardware concurrency. |
 | `--hub-instance-corelimit`         | `hub.instance.corelimit`       | `0`                                  | Concurrency limit for child instances. `0` is automatic. |
+| `--hub-instance-provision-threads` | `hub.instance.provisionthreads` | `max(cpu/4, 2)`                     | Thread count for the instance provisioning worker pool. Controls parallel I/O during provision and deprovision operations. Set to `0` for synchronous operation. |
 | `--hub-instance-config`            | `hub.instance.config`          | _(none)_                             | Path to a Lua config file passed to every spawned child instance. Use this to configure storage paths, cache sizes, and other storage server settings. See the zenserver configuration documentation. |
 
 ---
@@ -201,9 +207,18 @@ trigger a rejection.
 
 ### Hydration
 
-Hydration pre-populates a new instance's data directory from a shared source before the
-instance starts serving requests. This allows instances to start warm with a populated cache
-rather than building up content from scratch.
+Hydration pre-populates a new instance's data directory from a shared storage backend before
+the instance starts serving requests. On deprovision, the hub dehydrates the instance's
+state back to the same backend so the next provision of that module starts warm.
+
+The hydration system is incremental and content-addressed: files are hashed and stored in a
+CAS (content-addressable store) on the backend. Only files that changed since the last
+dehydration are uploaded or downloaded, and unchanged content is shared across modules. A
+cached state object tracks the file manifest between hydrate/dehydrate cycles to avoid
+redundant hashing and transfers.
+
+Before dehydrating, the hub triggers a GC cycle on the instance to compact storage, reducing
+the amount of data transferred to the backend.
 
 If neither hydration option is set, the hub automatically creates a `hydration_storage`
 directory under its own data directory and uses that as the file hydration source. This is
@@ -215,6 +230,7 @@ suitable for single-host deployments where instances share locally cached data.
 |-----------------------------------|------------------------------|---------------------------|-------------|
 | `--hub-hydration-target-spec`     | `hub.hydration.targetspec`   | _(local path, see above)_ | Shorthand URI for the hydration source. Must use the `file://` prefix for file targets: `file:///absolute/path`. |
 | `--hub-hydration-target-config`   | `hub.hydration.targetconfig` | _(none)_                  | Path to a JSON file specifying the hydration source. Supports `file` and `s3` backends. |
+| `--hub-hydration-threads`         | `hub.hydration.threads`      | `max(cpu/4, 2)`           | Thread count for the hydration/dehydration worker pool. Controls parallel file hashing and backend I/O during hydrate/dehydrate. Set to `0` for synchronous operation. |
 
 **File backend** (`hub.hydration.targetconfig` JSON):
 
@@ -365,6 +381,9 @@ hub = {
     http = "asio",
     httpthreads = 4,
 
+    -- Threads for provision/deprovision I/O (0 = synchronous)
+    provisionthreads = 4,
+
     -- Config file applied to every child instance
     config = "/etc/zen/instance.lua",
   },
@@ -372,6 +391,7 @@ hub = {
   -- Hydrate new instances from S3
   hydration = {
     targetconfig = "/etc/zen/hydration.json",
+    threads = 4,
   },
 
   watchdog = {