hub async s3 client (#1024)

- Feature: `AsyncHttpClient` adds cancellable request tokens, streaming GET to a file (`AsyncDownload`), zero-copy chunk-callback GET (`AsyncStream`), pull-mode body source for streaming `AsyncPut`, retry layer mirroring the synchronous client, and a submit-side in-flight cap (`HttpClientSettings::MaxConcurrentRequests`) so hub-scale fanout against a single host cannot stall queued handles into curl's connect-timeout window
- Feature: Hub hydration can route S3 transfers through a non-blocking `AsyncHttpClient` (curl_multi + asio) backed by a single io thread; hydrate and dehydrate now pipeline requests instead of blocking worker threads
  - `--hub-hydration-async-enabled` (Lua: `hub.hydration.async.enabled`, default true)
  - `--hub-hydration-async-max-concurrent-requests` (Lua: `hub.hydration.async.maxconcurrentrequests`, default `clamp(cpu*4, 128, 512)`)
- Feature: Hub provision/deprovision/obliterate now run as two phases on separate worker pools so per-module hydration cannot starve child-process spawn/despawn (and vice versa)
  - New `--hub-instance-spawn-threads` (Lua: `hub.instance.spawnthreads`, default `clamp(cpu/8, 4, 16)`) drives child-process spawn/despawn
  - `--hub-instance-provision-threads` (Lua: `hub.instance.provisionthreads`) now drives per-module hydrate/dehydrate scheduling only; default changed from `max(cpu/4, 2)` to `clamp(cpu/8, 4, 12)`
  - `--hub-hydration-threads` (Lua: `hub.hydration.threads`) now controls per-file workers inside a single hydrate/dehydrate; default changed from `max(cpu/4, 2)` to `clamp(cpu/8, 4, 12)`
- Feature: `AsyncHttpClient` owns its `asio::io_context` and one io thread by default; the `(BaseUri, io_context&)` constructor is preserved for callers that want to share an externally-driven `io_context` across clients (caller MUST keep the loop running until the client destructs)
- Feature: `Hub::Configuration` C++ struct fields renamed (`OptionalProvisionWorkerPool`/`OptionalHydrationWorkerPool` -> `OptionalProvisionPool`/`OptionalSpawnPool`/`OptionalHydrationPool`). Embedders constructing `Hub` directly must update field names; provision and spawn pools must both be set or both null (asserted at construction).
- Bugfix: `S3Client` signing-key cache no longer returns stale signatures after IMDS-rotated credentials change `AccessKeyId`; cache is now keyed on `(DateStamp, AccessKeyId)`

1 files changed, 3 insertions, 8 deletions

diff --git a/scripts/test_scripts/hub/perf_configs/hub.lua b/scripts/test_scripts/hub/perf_configs/hub.lua
index f3cf3e697..ff9ab582e 100644
--- a/scripts/test_scripts/hub/perf_configs/hub.lua
+++ b/scripts/test_scripts/hub/perf_configs/hub.lua
@@ -12,19 +12,14 @@ hub = {
       disklimitpercent = 90, -- default: 0 (disabled)
     },
     corelimit = 4, -- default: 0 (auto)
-    provisionthreads = 8, -- default: auto
+    -- provisionthreads / spawnthreads / hub.hydration.threads left unset: defaults
+    -- (clamp(cpu/8,4,12) / clamp(cpu/8,4,16) / clamp(cpu/8,4,12)) are tuned from
+    -- 1000-module sweep at 128 vCPU + 30ms latency. Override here only to A/B test.
     -- NOTE: hub.instance.config (path to instance lua) is overridden via
     -- --hub-instance-config on the CLI. If left here, it would be resolved
     -- relative to the hub's CWD at spawn time (NOT this file's dir).
   },
 
-  hydration = {
-    -- Match production's per-module download pool size. Without this, the
-    -- default auto-picks hardware_concurrency/4 which on --corelimit=128
-    -- would be 32. Prod logs consistently show "16 threads" in Download phase.
-    threads = 16,
-  },
-
   watchdog = {
     cycleintervalms = 5000, -- default: 3000. slower cycle, 1000 instances to scan
     cycleprocessingbudgetms = 1000, -- default: 500. more budget per cycle for larger instance count