hydration with pack (#1016)

- Feature: Hub hydration packs small files into raw CAS pack blobs to reduce request count for modules dominated by tiny metadata files
  - `--hub-hydration-enable-pack` (Lua: `hub.hydration.enablepack`, default true)
  - `--hub-hydration-pack-threshold-bytes` (Lua: `hub.hydration.packthresholdbytes`, default 256 KiB)
  - `--hub-hydration-max-pack-bytes` (Lua: `hub.hydration.maxpackbytes`, default 4 MiB)
- Feature: Hub hydration and dehydration can be disabled per direction
  - `--hub-enable-hydration` (Lua: `hub.enablehydration`, default true)
  - `--hub-enable-dehydration` (Lua: `hub.enabledehydration`, default true)
- Feature: Hub hydration accepts a configurable file exclude list via `HydrationOptions` `excludes` (array of wildcards). Built-in defaults skip transient runtime files (`.lock`, `.sentry-native/*`, `state_marker`, `*.bak`, `gc/reserve.gc`, `auth/*`) so they no longer participate in dehydrate scans. Override semantics: a present field replaces the default outright; explicit `[]` opts out of all defaults.
- Improvement: Hub hydration completion logs now report per-request average and max latency, peak in-flight workers, queue wait, and hash-cache hit percentage; loose and pack-blob transfers are reported separately
- Improvement: Hub hydration pre-creates unique parent directories before scheduling parallel writes
- Improvement: S3 hydration retries transient HTTP failures (timeouts, 429 throttling, 5xx server errors, connection errors) up to 3 times via the HTTP client retry layer
- Improvement: S3 hydration multipart chunk size is persisted in `state.cbo` per module so hydrate replays the partitioning used at dehydrate; default raised to 64 MiB (was 32 MiB)
- Improvement: Hub hydration `Obliterate` retries backend delete once before falling back to local cleanup

1 files changed, 32 insertions, 0 deletions

diff --git a/src/zenutil/cloud/s3client.cpp b/src/zenutil/cloud/s3client.cpp
index f8bed92da..ab80cfcc7 100644
--- a/src/zenutil/cloud/s3client.cpp
+++ b/src/zenutil/cloud/s3client.cpp
@@ -148,11 +148,31 @@ namespace {
 		return true;
 	}
 
+	/// True if the response indicates S3 throttling (503 SlowDown / ServiceUnavailable / 429).
+	/// Code is checked on both the HTTP status and the XML error code so we catch proxies that
+	/// return 200 with a SlowDown body.
+	bool IsS3Throttled(const HttpClient::Response& Response, std::string_view ErrorCode)
+	{
+		const int Status = static_cast<int>(Response.StatusCode);
+		if (Status == 503 || Status == 429)
+		{
+			return true;
+		}
+		if (ErrorCode == "SlowDown" || ErrorCode == "ServiceUnavailable" || ErrorCode == "ThrottlingException" ||
+			ErrorCode == "RequestLimitExceeded" || ErrorCode == "TooManyRequests")
+		{
+			return true;
+		}
+		return false;
+	}
+
 	/// Build a human-readable error message for a failed S3 response. When the response body
 	/// contains an S3 `<Error>` element, the Code and Message fields are included in the string
 	/// so transient 4xx/5xx failures (SignatureDoesNotMatch, AuthorizationHeaderMalformed, etc.)
 	/// show up in logs instead of being swallowed. Falls back to the generic HTTP/transport
 	/// message when no XML body is available (HEAD responses, transport errors).
+	/// Also emits a distinct `S3 THROTTLED` warning when the response indicates throttling so
+	/// callers can grep for it without parsing combined error text.
 	std::string S3ErrorMessage(std::string_view Prefix, const HttpClient::Response& Response)
 	{
 		if (!Response.Error.has_value() && Response.ResponsePayload)
@@ -164,9 +184,21 @@ namespace {
 			{
 				ExtendableStringBuilder<256> Decoded;
 				DecodeXmlEntities(Message, Decoded);
+				if (IsS3Throttled(Response, Code))
+				{
+					ZEN_WARN("S3 THROTTLED [{}] status={} code='{}' message='{}'",
+							 Prefix,
+							 static_cast<int>(Response.StatusCode),
+							 Code,
+							 Decoded.ToView());
+				}
 				return fmt::format("{}: HTTP status ({}) {} - {}", Prefix, static_cast<int>(Response.StatusCode), Code, Decoded.ToView());
 			}
 		}
+		if (IsS3Throttled(Response, {}))
+		{
+			ZEN_WARN("S3 THROTTLED [{}] status={} (no XML body)", Prefix, static_cast<int>(Response.StatusCode));
+		}
 		return Response.ErrorMessage(Prefix);
 	}