Compute OIDC auth, async Horde agents, and orchestrator improvements (#913)

Rework of the Horde agent subsystem from synchronous per-thread I/O to an async ASIO-driven architecture, plus provisioner scale-down with graceful draining, OIDC authentication, scheduler improvements, and dashboard UI for provisioner control.

### Async Horde Agent Rewrite
- Replace synchronous `HordeAgent` (one thread per agent, blocking I/O) with `AsyncHordeAgent` — an ASIO state machine running on a shared `io_context` thread pool
- Replace `TcpComputeTransport`/`AesComputeTransport` with `AsyncTcpComputeTransport`/`AsyncAesComputeTransport`
- Replace `AgentMessageChannel` with `AsyncAgentMessageChannel` using frame queuing and ASIO timers
- Delete `ComputeBuffer` and `ComputeChannel` ring-buffer classes (no longer needed)

### Provisioner Drain / Scale-Down
- `HordeProvisioner` can now drain agents when target core count is lowered: queries each agent's `/compute/session/status` for workload, selects candidates by largest-fit/lowest-workload, and sends `/compute/session/drain`
- Configurable `--horde-drain-grace-period` (default 300s) before force-kill
- Implement `IProvisionerStateProvider` interface to expose provisioner state to the orchestrator HTTP layer
- Forward `--coordinator-session`, `--provision-clean`, and `--provision-tracehost` through both Horde and Nomad provisioners to spawned workers

### OIDC Authentication
- `HordeClient` accepts an `AccessTokenProvider` (refreshable token function) as alternative to static `--horde-token`
- Wire up `OidcToken.exe` auto-discovery via `httpclientauth::CreateFromOidcTokenExecutable` with `--HordeUrl` mode
- New `--horde-oidctoken-exe-path` CLI option for explicit path override

### Orchestrator & Scheduler
- Orchestrator generates a session ID at startup; workers include `coordinator_session` in announcements so the orchestrator can reject stale-session workers
- New `Rejected` action state — when a remote runner declines at capacity, the action is rescheduled without retry count increment
- Reduce scheduler lock contention: snapshot pending actions under shared lock, sort/trim outside the lock
- Parallelize remote action submission across runners via `WorkerThreadPool` with slow-submit warnings
- New action field `FailureReason` populated by all runner types (exit codes, sandbox failures, exceptions)
- New endpoints: `session/drain`, `session/status`, `session/sunset`, `provisioner/status`, `provisioner/target`

### Remote Execution
- Eager-attach mode for `RemoteHttpRunner` — bundles all attachments upfront in a `CbPackage` for single-roundtrip submits
- Track in-flight submissions to prevent over-queuing
- Show remote runner hostname in `GetDisplayName()`
- `--announce-url` to override the endpoint announced to the coordinator (e.g. relay-visible address)

### Frontend Dashboard
- Delete standalone `compute.html` (925 lines) and `orchestrator.html` (669 lines), consolidated into JS page modules
- Add provisioner panel to orchestrator dashboard: target/active/estimated core counts, draining agent count
- Editable target-cores input with debounced POST to `/orch/provisioner/target`
- Per-agent provisioning status badges (active / draining / deallocated) in the agents table
- Active vs total CPU counts in agents summary row

### CLI
- New `zen compute record-start` / `record-stop` subcommands
- `zen exec` progress bar with submit and completion phases, atomic work counters, `--progress` mode (Pretty/Plain/Quiet)

### Other
- `DataDir` supports environment variable expansion
- Worker manifest validation checks for `worker.zcb` marker to detect incomplete cached directories
- Linux/Mac runners `nice(5)` child processes to avoid starving the main server
- `ComputeService::SetShutdownCallback` wired to `RequestExit` via `session/sunset`
- Curl HTTP client logs effective URL on failure
- `MachineInfo` carries `Pool` and `Mode` from Horde response
- Horde bundle creation includes `.pdb` on Windows

1 files changed, 27 insertions, 0 deletions

diff --git a/src/zencompute/runners/functionrunner.h b/src/zencompute/runners/functionrunner.h
index 56c3f3af0..449f0e228 100644
--- a/src/zencompute/runners/functionrunner.h
+++ b/src/zencompute/runners/functionrunner.h
@@ -10,6 +10,10 @@
 #	include <filesystem>
 #	include <vector>
 
+namespace zen {
+class WorkerThreadPool;
+}
+
 namespace zen::compute {
 
 struct SubmitResult
@@ -37,6 +41,22 @@ public:
 	[[nodiscard]] virtual bool						IsHealthy()							   = 0;
 	[[nodiscard]] virtual size_t					QueryCapacity();
 	[[nodiscard]] virtual std::vector<SubmitResult> SubmitActions(const std::vector<Ref<RunnerAction>>& Actions);
+	[[nodiscard]] virtual std::string_view			GetDisplayName() const { return "local"; }
+
+	// Accumulated stats from the most recent SubmitActions call.
+	// Reset before each call, populated by the runner implementation.
+	struct SubmitStats
+	{
+		std::atomic<size_t>	  TotalAttachments{0};
+		std::atomic<uint64_t> TotalAttachmentBytes{0};
+
+		void Reset()
+		{
+			TotalAttachments.store(0, std::memory_order_relaxed);
+			TotalAttachmentBytes.store(0, std::memory_order_relaxed);
+		}
+	};
+	SubmitStats m_LastSubmitStats;
 
 	// Best-effort cancellation of a specific in-flight action. Returns true if the
 	// cancellation signal was successfully sent. The action will transition to Cancelled
@@ -68,6 +88,8 @@ public:
 	bool					  CancelAction(int ActionLsn);
 	void					  CancelRemoteQueue(int QueueId);
 
+	void SetWorkerPool(WorkerThreadPool* Pool) { m_WorkerPool = Pool; }
+
 	size_t GetRunnerCount()
 	{
 		return m_RunnersLock.WithSharedLock([this] { return m_Runners.size(); });
@@ -79,6 +101,7 @@ protected:
 	RwLock							 m_RunnersLock;
 	std::vector<Ref<FunctionRunner>> m_Runners;
 	std::atomic<int>				 m_NextSubmitIndex{0};
+	WorkerThreadPool*				 m_WorkerPool = nullptr;
 };
 
 /** Typed RunnerGroup that adds type-safe runner addition and predicate-based removal.
@@ -151,6 +174,7 @@ struct RunnerAction : public RefCounted
 	CbObject	ActionObj;
 	int			Priority = 0;
 	std::string ExecutionLocation;	// "local" or remote hostname
+	std::string FailureReason;		// human-readable reason when action fails (empty on success)
 
 	// CPU usage and total CPU time of the running process, sampled periodically by the local runner.
 	// CpuUsagePercent: -1.0 means not yet sampled; >=0.0 is the most recent reading as a percentage.
@@ -168,6 +192,7 @@ struct RunnerAction : public RefCounted
 		Completed,	 // Finished successfully with results available
 		Failed,		 // Execution failed (transient error, eligible for retry)
 		Abandoned,	 // Infrastructure termination (e.g. spot eviction, session abandon)
+		Rejected,	 // Runner declined (e.g. at capacity) — rescheduled without retry cost
 		Cancelled,	 // Intentional user cancellation (never retried)
 		Retracted,	 // Pulled back for rescheduling on a different runner (no retry cost)
 		_Count
@@ -194,6 +219,8 @@ struct RunnerAction : public RefCounted
 				return "Failed";
 			case State::Abandoned:
 				return "Abandoned";
+			case State::Rejected:
+				return "Rejected";
 			case State::Cancelled:
 				return "Cancelled";
 			case State::Retracted: