Request validation and resilience improvements (#864)

### Security: Input validation & path safety
- **Reject local file references by default** in package parsing — only allow when explicitly opted in by the service (`ParseFlags::kAllowLocalReferences`) and validated by an `ILocalRefPolicy` (fail-closed: no policy = rejected)
- **`DataRootLocalRefPolicy`** restricts local ref paths to the server's data root via canonical path prefix matching
- **Validate attachment hashes** in compute HTTP handlers — decompresses and re-hashes each attachment at ingestion time to reject tampered payloads
- **Path traversal validation** for worker descriptions (`pathvalidation.h`) — rejects absolute paths, `..` components, Windows reserved device names, and invalid filename characters
- **Harden CbPackage parsing** against corrupt inputs — overflow-safe attachment count, bounds checks on local ref offset/size, graceful failure instead of `ZEN_ASSERT` for untrusted data
- **Harden legacy package parser** — reject zero-size binary fields, missing mappers, and optionally validate resolved attachment hashes
- **Bounds check in `CbPackageReader::MarshalLocalChunkReference`** — detect when `MakeFromFile` silently clamps offset+size to file size

### Reliability: Lock consolidation & bug fixes
- **Consolidate three action map locks into one** (`m_ActionMapLock`) — eliminates deadlock risk from multi-lock ordering, simplifies state transitions, and fixes a race where newly enqueued actions were briefly invisible to `GetActionResult`/`FindActionResult`
- **Fix infinite loop in `BaseRunnerGroup::SubmitActions`** when actions exceed total runner capacity — cap round-robin at `TotalCapacity` and default unassigned results to "No capacity"
- **Fix `MakeSafeAbsolutePathInPlace` for UNC paths** — `\server\share` now correctly becomes `\?\UNC\server\share` instead of `\?\server\share`
- **Fix `max_retries=0`** — previously fell through to the default of 3; now correctly means "no retries"

### New: ManagedProcessRunner
- Cross-platform process runner backed by `SubprocessManager` — uses async exit callbacks instead of polling, delegates CPU/memory metrics to the manager's built-in sampler
- `ProcessGroup` (JobObject on Windows, process group on POSIX) for bulk cancellation on shutdown
- `--managed` flag on `zen exec inproc` to select this runner
- Refactored monitor thread lifecycle — `StartMonitorThread()` now called from derived constructors to avoid calling virtual functions from base constructor

### Process management
- **Suppress crash dialogs** via `JOB_OBJECT_UILIMIT_ERRORMODE` + `SEM_NOGPFAULTERRORBOX` in both `WindowsProcessRunner` and `JobObject::Initialize` — prevents WER/Dr. Watson modal dialogs from blocking the monitor thread
- **CREATE_SUSPENDED → AssignProcessToJobObject → ResumeThread** pattern in `WindowsProcessRunner` — ensures job object assignment before process execution
- **Move stdout/stderr callbacks to `Spawn()` parameters** in `SubprocessManager` — prevents race where early output could be missed before callback installation
- Consistent PID logging across all runner types

### Test infrastructure
- **`zentest-appstub`**: Added `Fail` (configurable exit code) and `Crash` (abort / nullptr deref) test functions
- **Compute integration tests**: exit code handling, auto-retry exhaustion, manual reschedule after failure, mixed success/failure queues, crash handling (abort + nullptr), crash auto-retry, immediate query visibility after enqueue
- **Package format tests**: truncated header, bad magic, attachment count overflow, truncated data, local ref rejection/acceptance, policy enforcement (inside/outside root, traversal, no-policy fail-closed)
- **Legacy package parser tests**: empty input, zero-size binary, hash resolution with/without mapper, hash mismatch detection
- **UNC path tests** for `MakeSafeAbsolutePath`

### Misc
- ANSI color helper macros (`ZEN_RED`, `ZEN_BRIGHT_WHITE`, etc.) and `ZEN_BOLD`/`ZEN_DIM`/etc.
- Generic `fmt::formatter` for types with free `ToString` functions
- Compute dashboard: truncated hash display with monospace font and hover for full value
- Renamed `usonpackage_forcelink` → `cbpackage_forcelink`
- Compute enabled by default in xmake config (releases still explicitly disable)

10 files changed, 151 insertions, 29 deletions

diff --git a/src/zenserver/frontend/html/pages/compute.js b/src/zenserver/frontend/html/pages/compute.js
index d1a880954..2eb4d4e9b 100644
--- a/src/zenserver/frontend/html/pages/compute.js
+++ b/src/zenserver/frontend/html/pages/compute.js
@@ -24,6 +24,12 @@ function formatTime(date)
 	return date.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit", second: "2-digit" });
 }
 
+function truncateHash(hash)
+{
+	if (!hash || hash.length <= 15) return hash;
+	return hash.slice(0, 6) + "\u2026" + hash.slice(-6);
+}
+
 function formatDuration(startDate, endDate)
 {
 	if (!startDate || !endDate) return "-";
@@ -305,11 +311,7 @@ export class Page extends ZenPage
 	{
 		const workerIds = data.workers || [];
 
-		if (this._workers_table)
-		{
-			this._workers_table.clear();
-		}
-		else
+		if (!this._workers_table)
 		{
 			this._workers_table = this._workers_host.add_widget(
 				Table,
@@ -320,6 +322,7 @@ export class Page extends ZenPage
 
 		if (workerIds.length === 0)
 		{
+			this._workers_table.clear();
 			return;
 		}
 
@@ -349,6 +352,9 @@ export class Page extends ZenPage
 					id,
 				);
 
+				// Worker ID column: monospace for hex readability
+				row.get_cell(5).style("fontFamily", "'SF Mono', 'Cascadia Mono', Consolas, 'DejaVu Sans Mono', monospace");
+
 				// Make name clickable to expand detail
 				const cell = row.get_cell(0);
 				cell.tag().text(name).on_click(() => this._toggle_worker_detail(id, desc));
@@ -546,6 +552,11 @@ export class Page extends ZenPage
 				["LSN", "queue", "status", "function", "started", "finished", "duration", "worker ID", "action ID"],
 				Table.Flag_FitLeft|Table.Flag_PackRight|Table.Flag_Sortable|Table.Flag_AlignNumeric, -1
 			);
+
+			// Right-align hash column headers to match data cells
+			const hdr = this._history_table.inner().firstElementChild;
+			hdr.children[7].style.textAlign = "right";
+			hdr.children[8].style.textAlign = "right";
 		}
 
 		// Entries arrive oldest-first; reverse to show newest at top
@@ -560,7 +571,10 @@ export class Page extends ZenPage
 			const startDate = filetimeToDate(entry.time_Running);
 			const endDate = filetimeToDate(entry.time_Completed ?? entry.time_Failed);
 
-			this._history_table.add_row(
+			const workerId = entry.workerId || "-";
+			const actionId = entry.actionId || "-";
+
+			const row = this._history_table.add_row(
 				lsn,
 				queueId,
 				status,
@@ -568,9 +582,15 @@ export class Page extends ZenPage
 				formatTime(startDate),
 				formatTime(endDate),
 				formatDuration(startDate, endDate),
-				entry.workerId || "-",
-				entry.actionId || "-",
+				truncateHash(workerId),
+				truncateHash(actionId),
 			);
+
+			// Hash columns: force right-align (AlignNumeric misses hex strings starting with a-f),
+			// use monospace for readability, and show full value on hover
+			const mono = "'SF Mono', 'Cascadia Mono', Consolas, 'DejaVu Sans Mono', monospace";
+			row.get_cell(7).style("textAlign", "right").style("fontFamily", mono).attr("title", workerId);
+			row.get_cell(8).style("textAlign", "right").style("fontFamily", mono).attr("title", actionId);
 		}
 	}
 
diff --git a/src/zenserver/storage/cache/httpstructuredcache.cpp b/src/zenserver/storage/cache/httpstructuredcache.cpp
index c1727270c..8ad48225b 100644
--- a/src/zenserver/storage/cache/httpstructuredcache.cpp
+++ b/src/zenserver/storage/cache/httpstructuredcache.cpp
@@ -80,7 +80,8 @@ HttpStructuredCacheService::HttpStructuredCacheService(ZenCacheStore&		   InCach
 													   HttpStatusService&	   StatusService,
 													   UpstreamCache&		   UpstreamCache,
 													   const DiskWriteBlocker* InDiskWriteBlocker,
-													   OpenProcessCache&	   InOpenProcessCache)
+													   OpenProcessCache&	   InOpenProcessCache,
+													   const ILocalRefPolicy*  InLocalRefPolicy)
 : m_Log(logging::Get("cache"))
 , m_CacheStore(InCacheStore)
 , m_StatsService(StatsService)
@@ -90,6 +91,7 @@ HttpStructuredCacheService::HttpStructuredCacheService(ZenCacheStore&		   InCach
 , m_DiskWriteBlocker(InDiskWriteBlocker)
 , m_OpenProcessCache(InOpenProcessCache)
 , m_RpcHandler(m_Log, m_CacheStats, UpstreamCache, InCacheStore, InCidStore, InDiskWriteBlocker)
+, m_LocalRefPolicy(InLocalRefPolicy)
 {
 	m_StatsService.RegisterHandler("z$", *this);
 	m_StatusService.RegisterHandler("z$", *this);
@@ -114,6 +116,18 @@ HttpStructuredCacheService::BaseUri() const
 	return "/z$/";
 }
 
+bool
+HttpStructuredCacheService::AcceptsLocalFileReferences() const
+{
+	return true;
+}
+
+const ILocalRefPolicy*
+HttpStructuredCacheService::GetLocalRefPolicy() const
+{
+	return m_LocalRefPolicy;
+}
+
 void
 HttpStructuredCacheService::Flush()
 {
diff --git a/src/zenserver/storage/cache/httpstructuredcache.h b/src/zenserver/storage/cache/httpstructuredcache.h
index fc80b449e..f606126d6 100644
--- a/src/zenserver/storage/cache/httpstructuredcache.h
+++ b/src/zenserver/storage/cache/httpstructuredcache.h
@@ -76,11 +76,14 @@ public:
 							   HttpStatusService&	   StatusService,
 							   UpstreamCache&		   UpstreamCache,
 							   const DiskWriteBlocker* InDiskWriteBlocker,
-							   OpenProcessCache&	   InOpenProcessCache);
+							   OpenProcessCache&	   InOpenProcessCache,
+							   const ILocalRefPolicy*  InLocalRefPolicy = nullptr);
 	~HttpStructuredCacheService();
 
-	virtual const char* BaseUri() const override;
-	virtual void		HandleRequest(HttpServerRequest& Request) override;
+	virtual const char*			   BaseUri() const override;
+	virtual void				   HandleRequest(HttpServerRequest& Request) override;
+	virtual bool				   AcceptsLocalFileReferences() const override;
+	virtual const ILocalRefPolicy* GetLocalRefPolicy() const override;
 
 	void Flush();
 
@@ -125,6 +128,7 @@ private:
 	const DiskWriteBlocker*	 m_DiskWriteBlocker = nullptr;
 	OpenProcessCache&		 m_OpenProcessCache;
 	CacheRpcHandler			 m_RpcHandler;
+	const ILocalRefPolicy*	 m_LocalRefPolicy = nullptr;
 
 	void ReplayRequestRecorder(const CacheRequestContext& Context, cache::IRpcRequestReplayer& Replayer, uint32_t ThreadCount);
 
diff --git a/src/zenserver/storage/localrefpolicy.cpp b/src/zenserver/storage/localrefpolicy.cpp
new file mode 100644
index 000000000..47ef13b28
--- /dev/null
+++ b/src/zenserver/storage/localrefpolicy.cpp
@@ -0,0 +1,29 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#include "localrefpolicy.h"
+
+#include <zencore/except_fmt.h>
+#include <zencore/fmtutils.h>
+
+#include <filesystem>
+
+namespace zen {
+
+DataRootLocalRefPolicy::DataRootLocalRefPolicy(const std::filesystem::path& DataRoot)
+: m_CanonicalRoot(std::filesystem::weakly_canonical(DataRoot).string())
+{
+}
+
+void
+DataRootLocalRefPolicy::ValidatePath(const std::filesystem::path& Path) const
+{
+	std::filesystem::path CanonicalFile = std::filesystem::weakly_canonical(Path);
+	std::string			  FileStr		= CanonicalFile.string();
+
+	if (FileStr.size() < m_CanonicalRoot.size() || FileStr.compare(0, m_CanonicalRoot.size(), m_CanonicalRoot) != 0)
+	{
+		throw zen::invalid_argument("local file reference '{}' is outside allowed data root", CanonicalFile);
+	}
+}
+
+}  // namespace zen
diff --git a/src/zenserver/storage/localrefpolicy.h b/src/zenserver/storage/localrefpolicy.h
new file mode 100644
index 000000000..3686d1880
--- /dev/null
+++ b/src/zenserver/storage/localrefpolicy.h
@@ -0,0 +1,25 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zenhttp/localrefpolicy.h>
+
+#include <filesystem>
+#include <string>
+
+namespace zen {
+
+/// Local ref policy that restricts file paths to a canonical data root directory.
+/// Uses weakly_canonical + string prefix comparison to detect path traversal.
+class DataRootLocalRefPolicy : public ILocalRefPolicy
+{
+public:
+	explicit DataRootLocalRefPolicy(const std::filesystem::path& DataRoot);
+
+	void ValidatePath(const std::filesystem::path& Path) const override;
+
+private:
+	std::string m_CanonicalRoot;
+};
+
+}  // namespace zen
diff --git a/src/zenserver/storage/projectstore/httpprojectstore.cpp b/src/zenserver/storage/projectstore/httpprojectstore.cpp
index a7c8c66b6..afd0d8f82 100644
--- a/src/zenserver/storage/projectstore/httpprojectstore.cpp
+++ b/src/zenserver/storage/projectstore/httpprojectstore.cpp
@@ -656,7 +656,8 @@ HttpProjectService::HttpProjectService(CidStore&					Store,
 									   JobQueue&					InJobQueue,
 									   bool							InRestrictContentTypes,
 									   const std::filesystem::path& InOidcTokenExePath,
-									   bool							InAllowExternalOidcTokenExe)
+									   bool							InAllowExternalOidcTokenExe,
+									   const ILocalRefPolicy*		InLocalRefPolicy)
 : m_Log(logging::Get("project"))
 , m_CidStore(Store)
 , m_ProjectStore(Projects)
@@ -668,6 +669,7 @@ HttpProjectService::HttpProjectService(CidStore&					Store,
 , m_RestrictContentTypes(InRestrictContentTypes)
 , m_OidcTokenExePath(InOidcTokenExePath)
 , m_AllowExternalOidcTokenExe(InAllowExternalOidcTokenExe)
+, m_LocalRefPolicy(InLocalRefPolicy)
 {
 	ZEN_MEMSCOPE(GetProjectHttpTag());
 
@@ -820,6 +822,18 @@ HttpProjectService::BaseUri() const
 	return "/prj/";
 }
 
+bool
+HttpProjectService::AcceptsLocalFileReferences() const
+{
+	return true;
+}
+
+const ILocalRefPolicy*
+HttpProjectService::GetLocalRefPolicy() const
+{
+	return m_LocalRefPolicy;
+}
+
 void
 HttpProjectService::HandleRequest(HttpServerRequest& Request)
 {
@@ -1668,7 +1682,8 @@ HttpProjectService::HandleOplogOpNewRequest(HttpRouterRequest& Req)
 
 	CbPackage Package;
 
-	if (!legacy::TryLoadCbPackage(Package, Payload, &UniqueBuffer::Alloc, &Resolver))
+	const bool ValidateHashes = false;
+	if (!legacy::TryLoadCbPackage(Package, Payload, &UniqueBuffer::Alloc, &Resolver, ValidateHashes))
 	{
 		CbValidateError ValidateResult;
 		if (CbObject Core = ValidateAndReadCompactBinaryObject(IoBuffer(Payload), ValidateResult);
@@ -2763,7 +2778,11 @@ HttpProjectService::HandleRpcRequest(HttpRouterRequest& Req)
 		case HttpContentType::kCbPackage:
 			try
 			{
-				Package = ParsePackageMessage(Payload);
+				ParseFlags PkgFlags = (HttpReq.IsLocalMachineRequest() && AcceptsLocalFileReferences()) ? ParseFlags::kAllowLocalReferences
+																										: ParseFlags::kDefault;
+				const ILocalRefPolicy* PkgPolicy =
+					EnumHasAllFlags(PkgFlags, ParseFlags::kAllowLocalReferences) ? GetLocalRefPolicy() : nullptr;
+				Package = ParsePackageMessage(Payload, {}, PkgFlags, PkgPolicy);
 				Cb		= Package.GetObject();
 			}
 			catch (const std::invalid_argument& ex)
diff --git a/src/zenserver/storage/projectstore/httpprojectstore.h b/src/zenserver/storage/projectstore/httpprojectstore.h
index e3ed02f26..8aa345fa7 100644
--- a/src/zenserver/storage/projectstore/httpprojectstore.h
+++ b/src/zenserver/storage/projectstore/httpprojectstore.h
@@ -47,11 +47,14 @@ public:
 					   JobQueue&					InJobQueue,
 					   bool							InRestrictContentTypes,
 					   const std::filesystem::path& InOidcTokenExePath,
-					   bool							AllowExternalOidcTokenExe);
+					   bool							AllowExternalOidcTokenExe,
+					   const ILocalRefPolicy*		InLocalRefPolicy = nullptr);
 	~HttpProjectService();
 
-	virtual const char* BaseUri() const override;
-	virtual void		HandleRequest(HttpServerRequest& Request) override;
+	virtual const char*			   BaseUri() const override;
+	virtual void				   HandleRequest(HttpServerRequest& Request) override;
+	virtual bool				   AcceptsLocalFileReferences() const override;
+	virtual const ILocalRefPolicy* GetLocalRefPolicy() const override;
 
 	virtual void	 HandleStatusRequest(HttpServerRequest& Request) override;
 	virtual void	 HandleStatsRequest(HttpServerRequest& Request) override;
@@ -117,6 +120,7 @@ private:
 	bool					   m_RestrictContentTypes;
 	std::filesystem::path	   m_OidcTokenExePath;
 	bool					   m_AllowExternalOidcTokenExe;
+	const ILocalRefPolicy*	   m_LocalRefPolicy;
 
 	Ref<TransferThreadWorkers> GetThreadWorkers(bool BoostWorkers, bool SingleThreaded);
 };
diff --git a/src/zenserver/storage/zenstorageserver.cpp b/src/zenserver/storage/zenstorageserver.cpp
index bc0a8f4ac..6b1da5f12 100644
--- a/src/zenserver/storage/zenstorageserver.cpp
+++ b/src/zenserver/storage/zenstorageserver.cpp
@@ -223,6 +223,7 @@ ZenStorageServer::InitializeServices(const ZenStorageServerConfig& ServerOptions
 
 	ZEN_INFO("instantiating project service");
 
+	m_LocalRefPolicy   = std::make_unique<DataRootLocalRefPolicy>(m_DataRoot);
 	m_JobQueue		   = MakeJobQueue(8, "bgjobs");
 	m_OpenProcessCache = std::make_unique<OpenProcessCache>();
 
@@ -236,7 +237,8 @@ ZenStorageServer::InitializeServices(const ZenStorageServerConfig& ServerOptions
 													  *m_JobQueue,
 													  ServerOptions.RestrictContentTypes,
 													  ServerOptions.OidcTokenExecutable,
-													  ServerOptions.AllowExternalOidcTokenExe});
+													  ServerOptions.AllowExternalOidcTokenExe,
+													  m_LocalRefPolicy.get()});
 
 	if (ServerOptions.WorksSpacesConfig.Enabled)
 	{
@@ -713,7 +715,8 @@ ZenStorageServer::InitializeStructuredCache(const ZenStorageServerConfig& Server
 																			m_StatusService,
 																			*m_UpstreamCache,
 																			m_GcManager.GetDiskWriteBlocker(),
-																			*m_OpenProcessCache);
+																			*m_OpenProcessCache,
+																			m_LocalRefPolicy.get());
 
 	m_StatsReporter.AddProvider(m_CacheStore.Get());
 	m_StatsReporter.AddProvider(m_CidStore.get());
diff --git a/src/zenserver/storage/zenstorageserver.h b/src/zenserver/storage/zenstorageserver.h
index fad22ad54..e3c6248e6 100644
--- a/src/zenserver/storage/zenstorageserver.h
+++ b/src/zenserver/storage/zenstorageserver.h
@@ -11,6 +11,7 @@
 #include <zenstore/cache/structuredcachestore.h>
 #include <zenstore/gc.h>
 #include <zenstore/projectstore.h>
+#include "localrefpolicy.h"
 
 #include "admin/admin.h"
 #include "buildstore/httpbuildstore.h"
@@ -65,15 +66,16 @@ private:
 	void InitializeServices(const ZenStorageServerConfig& ServerOptions);
 	void RegisterServices();
 
-	std::unique_ptr<JobQueue>		  m_JobQueue;
-	GcManager						  m_GcManager;
-	GcScheduler						  m_GcScheduler{m_GcManager};
-	std::unique_ptr<CidStore>		  m_CidStore;
-	Ref<ZenCacheStore>				  m_CacheStore;
-	std::unique_ptr<OpenProcessCache> m_OpenProcessCache;
-	HttpTestService					  m_TestService;
-	std::unique_ptr<CidStore>		  m_BuildCidStore;
-	std::unique_ptr<BuildStore>		  m_BuildStore;
+	std::unique_ptr<DataRootLocalRefPolicy> m_LocalRefPolicy;
+	std::unique_ptr<JobQueue>				m_JobQueue;
+	GcManager								m_GcManager;
+	GcScheduler								m_GcScheduler{m_GcManager};
+	std::unique_ptr<CidStore>				m_CidStore;
+	Ref<ZenCacheStore>						m_CacheStore;
+	std::unique_ptr<OpenProcessCache>		m_OpenProcessCache;
+	HttpTestService							m_TestService;
+	std::unique_ptr<CidStore>				m_BuildCidStore;
+	std::unique_ptr<BuildStore>				m_BuildStore;
 
 #if ZEN_WITH_TESTS
 	HttpTestingService m_TestingService;
diff --git a/src/zenserver/zenserver.cpp b/src/zenserver/zenserver.cpp
index 6aa02eb87..087b40d6a 100644
--- a/src/zenserver/zenserver.cpp
+++ b/src/zenserver/zenserver.cpp
@@ -272,6 +272,8 @@ ZenServerBase::GetBuildOptions(StringBuilderBase& OutOptions, char Separator) co
 	OutOptions << Separator;
 	OutOptions << "ZEN_WITH_MEMTRACK=" << (ZEN_WITH_MEMTRACK ? "1" : "0");
 	OutOptions << Separator;
+	OutOptions << "ZEN_WITH_COMPUTE_SERVICES=" << (ZEN_WITH_COMPUTE_SERVICES ? "1" : "0");
+	OutOptions << Separator;
 	OutOptions << "ZEN_WITH_TRACE=" << (ZEN_WITH_TRACE ? "1" : "0");
 }