aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
diff options
context:
space:
mode:
authorStefan Boberg <[email protected]>2026-03-30 15:07:08 +0200
committerGitHub Enterprise <[email protected]>2026-03-30 15:07:08 +0200
commit3540d676733efaddecf504b30e9a596465bd43f8 (patch)
tree7a8d8b3d2da993e30c34e3ff36f659b90a2b228e /src/zenhttp
parentinclude rawHash in structure output for builds ls command (#903) (diff)
downloadzen-3540d676733efaddecf504b30e9a596465bd43f8.tar.xz
zen-3540d676733efaddecf504b30e9a596465bd43f8.zip
Request validation and resilience improvements (#864)
### Security: Input validation & path safety - **Reject local file references by default** in package parsing — only allow when explicitly opted in by the service (`ParseFlags::kAllowLocalReferences`) and validated by an `ILocalRefPolicy` (fail-closed: no policy = rejected) - **`DataRootLocalRefPolicy`** restricts local ref paths to the server's data root via canonical path prefix matching - **Validate attachment hashes** in compute HTTP handlers — decompresses and re-hashes each attachment at ingestion time to reject tampered payloads - **Path traversal validation** for worker descriptions (`pathvalidation.h`) — rejects absolute paths, `..` components, Windows reserved device names, and invalid filename characters - **Harden CbPackage parsing** against corrupt inputs — overflow-safe attachment count, bounds checks on local ref offset/size, graceful failure instead of `ZEN_ASSERT` for untrusted data - **Harden legacy package parser** — reject zero-size binary fields, missing mappers, and optionally validate resolved attachment hashes - **Bounds check in `CbPackageReader::MarshalLocalChunkReference`** — detect when `MakeFromFile` silently clamps offset+size to file size ### Reliability: Lock consolidation & bug fixes - **Consolidate three action map locks into one** (`m_ActionMapLock`) — eliminates deadlock risk from multi-lock ordering, simplifies state transitions, and fixes a race where newly enqueued actions were briefly invisible to `GetActionResult`/`FindActionResult` - **Fix infinite loop in `BaseRunnerGroup::SubmitActions`** when actions exceed total runner capacity — cap round-robin at `TotalCapacity` and default unassigned results to "No capacity" - **Fix `MakeSafeAbsolutePathInPlace` for UNC paths** — `\server\share` now correctly becomes `\?\UNC\server\share` instead of `\?\server\share` - **Fix `max_retries=0`** — previously fell through to the default of 3; now correctly means "no retries" ### New: ManagedProcessRunner - Cross-platform process runner backed by `SubprocessManager` — uses async exit callbacks instead of polling, delegates CPU/memory metrics to the manager's built-in sampler - `ProcessGroup` (JobObject on Windows, process group on POSIX) for bulk cancellation on shutdown - `--managed` flag on `zen exec inproc` to select this runner - Refactored monitor thread lifecycle — `StartMonitorThread()` now called from derived constructors to avoid calling virtual functions from base constructor ### Process management - **Suppress crash dialogs** via `JOB_OBJECT_UILIMIT_ERRORMODE` + `SEM_NOGPFAULTERRORBOX` in both `WindowsProcessRunner` and `JobObject::Initialize` — prevents WER/Dr. Watson modal dialogs from blocking the monitor thread - **CREATE_SUSPENDED → AssignProcessToJobObject → ResumeThread** pattern in `WindowsProcessRunner` — ensures job object assignment before process execution - **Move stdout/stderr callbacks to `Spawn()` parameters** in `SubprocessManager` — prevents race where early output could be missed before callback installation - Consistent PID logging across all runner types ### Test infrastructure - **`zentest-appstub`**: Added `Fail` (configurable exit code) and `Crash` (abort / nullptr deref) test functions - **Compute integration tests**: exit code handling, auto-retry exhaustion, manual reschedule after failure, mixed success/failure queues, crash handling (abort + nullptr), crash auto-retry, immediate query visibility after enqueue - **Package format tests**: truncated header, bad magic, attachment count overflow, truncated data, local ref rejection/acceptance, policy enforcement (inside/outside root, traversal, no-policy fail-closed) - **Legacy package parser tests**: empty input, zero-size binary, hash resolution with/without mapper, hash mismatch detection - **UNC path tests** for `MakeSafeAbsolutePath` ### Misc - ANSI color helper macros (`ZEN_RED`, `ZEN_BRIGHT_WHITE`, etc.) and `ZEN_BOLD`/`ZEN_DIM`/etc. - Generic `fmt::formatter` for types with free `ToString` functions - Compute dashboard: truncated hash display with monospace font and hover for full value - Renamed `usonpackage_forcelink` → `cbpackage_forcelink` - Compute enabled by default in xmake config (releases still explicitly disable)
Diffstat (limited to 'src/zenhttp')
-rw-r--r--src/zenhttp/httpserver.cpp24
-rw-r--r--src/zenhttp/include/zenhttp/httpserver.h14
-rw-r--r--src/zenhttp/include/zenhttp/localrefpolicy.h21
-rw-r--r--src/zenhttp/include/zenhttp/packageformat.h25
-rw-r--r--src/zenhttp/packageformat.cpp548
5 files changed, 588 insertions, 44 deletions
diff --git a/src/zenhttp/httpserver.cpp b/src/zenhttp/httpserver.cpp
index e05c9815f..38021be16 100644
--- a/src/zenhttp/httpserver.cpp
+++ b/src/zenhttp/httpserver.cpp
@@ -479,6 +479,18 @@ HttpService::HandlePackageRequest(HttpServerRequest& HttpServiceRequest)
return Ref<IHttpPackageHandler>();
}
+bool
+HttpService::AcceptsLocalFileReferences() const
+{
+ return false;
+}
+
+const ILocalRefPolicy*
+HttpService::GetLocalRefPolicy() const
+{
+ return nullptr;
+}
+
//////////////////////////////////////////////////////////////////////////
HttpServerRequest::HttpServerRequest(HttpService& Service) : m_Service(Service)
@@ -705,7 +717,10 @@ HttpServerRequest::ReadPayloadPackage()
{
if (IoBuffer Payload = ReadPayload())
{
- return ParsePackageMessage(std::move(Payload));
+ ParseFlags Flags =
+ (IsLocalMachineRequest() && m_Service.AcceptsLocalFileReferences()) ? ParseFlags::kAllowLocalReferences : ParseFlags::kDefault;
+ const ILocalRefPolicy* Policy = EnumHasAllFlags(Flags, ParseFlags::kAllowLocalReferences) ? m_Service.GetLocalRefPolicy() : nullptr;
+ return ParsePackageMessage(std::move(Payload), {}, Flags, Policy);
}
return {};
@@ -1273,7 +1288,12 @@ HandlePackageOffers(HttpService& Service, HttpServerRequest& Request, Ref<IHttpP
return PackageHandlerRef->CreateTarget(Cid, Size);
};
- CbPackage Package = ParsePackageMessage(Request.ReadPayload(), CreateBuffer);
+ ParseFlags PkgFlags = (Request.IsLocalMachineRequest() && Service.AcceptsLocalFileReferences())
+ ? ParseFlags::kAllowLocalReferences
+ : ParseFlags::kDefault;
+ const ILocalRefPolicy* PkgPolicy =
+ EnumHasAllFlags(PkgFlags, ParseFlags::kAllowLocalReferences) ? Service.GetLocalRefPolicy() : nullptr;
+ CbPackage Package = ParsePackageMessage(Request.ReadPayload(), CreateBuffer, PkgFlags, PkgPolicy);
PackageHandlerRef->OnRequestComplete();
}
diff --git a/src/zenhttp/include/zenhttp/httpserver.h b/src/zenhttp/include/zenhttp/httpserver.h
index 5eaed6004..76f219f04 100644
--- a/src/zenhttp/include/zenhttp/httpserver.h
+++ b/src/zenhttp/include/zenhttp/httpserver.h
@@ -12,6 +12,7 @@
#include <zencore/string.h>
#include <zencore/uid.h>
#include <zenhttp/httpcommon.h>
+#include <zenhttp/localrefpolicy.h>
#include <zentelemetry/hyperloglog.h>
#include <zentelemetry/stats.h>
@@ -193,9 +194,16 @@ public:
HttpService() = default;
virtual ~HttpService() = default;
- virtual const char* BaseUri() const = 0;
- virtual void HandleRequest(HttpServerRequest& HttpServiceRequest) = 0;
- virtual Ref<IHttpPackageHandler> HandlePackageRequest(HttpServerRequest& HttpServiceRequest);
+ [[nodiscard]] virtual const char* BaseUri() const = 0;
+ virtual void HandleRequest(HttpServerRequest& HttpServiceRequest) = 0;
+ virtual Ref<IHttpPackageHandler> HandlePackageRequest(HttpServerRequest& HttpServiceRequest);
+
+ /// Whether this service accepts local file references in inbound packages from local clients.
+ [[nodiscard]] virtual bool AcceptsLocalFileReferences() const;
+
+ /// Returns the local ref policy for validating file paths in inbound local references.
+ /// Returns nullptr by default, which causes file-path local refs to be rejected (fail-closed).
+ [[nodiscard]] virtual const ILocalRefPolicy* GetLocalRefPolicy() const;
// Internals
diff --git a/src/zenhttp/include/zenhttp/localrefpolicy.h b/src/zenhttp/include/zenhttp/localrefpolicy.h
new file mode 100644
index 000000000..0b37f9dc7
--- /dev/null
+++ b/src/zenhttp/include/zenhttp/localrefpolicy.h
@@ -0,0 +1,21 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <filesystem>
+
+namespace zen {
+
+/// Policy interface for validating local file reference paths in inbound CbPackage messages.
+/// Implementations should throw std::invalid_argument if the path is not allowed.
+class ILocalRefPolicy
+{
+public:
+ virtual ~ILocalRefPolicy() = default;
+
+ /// Validate that a local file reference path is allowed.
+ /// Throws std::invalid_argument if the path escapes the allowed root.
+ virtual void ValidatePath(const std::filesystem::path& Path) const = 0;
+};
+
+} // namespace zen
diff --git a/src/zenhttp/include/zenhttp/packageformat.h b/src/zenhttp/include/zenhttp/packageformat.h
index 1a5068580..66e3f6e55 100644
--- a/src/zenhttp/include/zenhttp/packageformat.h
+++ b/src/zenhttp/include/zenhttp/packageformat.h
@@ -5,6 +5,7 @@
#include <zencore/compactbinarypackage.h>
#include <zencore/iobuffer.h>
#include <zencore/iohash.h>
+#include <zenhttp/localrefpolicy.h>
#include <functional>
#include <gsl/gsl-lite.hpp>
@@ -97,11 +98,22 @@ gsl_DEFINE_ENUM_BITMASK_OPERATORS(RpcAcceptOptions);
std::vector<IoBuffer> FormatPackageMessage(const CbPackage& Data, FormatFlags Flags, void* TargetProcessHandle = nullptr);
CompositeBuffer FormatPackageMessageBuffer(const CbPackage& Data, FormatFlags Flags, void* TargetProcessHandle = nullptr);
-CbPackage ParsePackageMessage(
- IoBuffer Payload,
- std::function<IoBuffer(const IoHash& Cid, uint64_t Size)> CreateBuffer = [](const IoHash&, uint64_t Size) -> IoBuffer {
+
+enum class ParseFlags
+{
+ kDefault = 0,
+ kAllowLocalReferences = (1u << 0), // Allow packages containing local file references (local clients only)
+};
+
+gsl_DEFINE_ENUM_BITMASK_OPERATORS(ParseFlags);
+
+CbPackage ParsePackageMessage(
+ IoBuffer Payload,
+ std::function<IoBuffer(const IoHash& Cid, uint64_t Size)> CreateBuffer = [](const IoHash&, uint64_t Size) -> IoBuffer {
return IoBuffer{Size};
- });
+ },
+ ParseFlags Flags = ParseFlags::kDefault,
+ const ILocalRefPolicy* Policy = nullptr);
bool IsPackageMessage(IoBuffer Payload);
bool ParsePackageMessageWithLegacyFallback(const IoBuffer& Response, CbPackage& OutPackage);
@@ -122,10 +134,11 @@ CompositeBuffer FormatPackageMessageBuffer(const CbPackage& Data, void* Targe
class CbPackageReader
{
public:
- CbPackageReader();
+ CbPackageReader(ParseFlags Flags = ParseFlags::kDefault);
~CbPackageReader();
void SetPayloadBufferCreator(std::function<IoBuffer(const IoHash& Cid, uint64_t Size)> CreateBuffer);
+ void SetLocalRefPolicy(const ILocalRefPolicy* Policy);
/** Process compact binary package data stream
@@ -149,6 +162,8 @@ private:
kReadingBuffers
} m_CurrentState = State::kInitialState;
+ ParseFlags m_Flags;
+ const ILocalRefPolicy* m_LocalRefPolicy = nullptr;
std::function<IoBuffer(const IoHash& Cid, uint64_t Size)> m_CreateBuffer;
std::vector<IoBuffer> m_PayloadBuffers;
std::vector<CbAttachmentEntry> m_AttachmentEntries;
diff --git a/src/zenhttp/packageformat.cpp b/src/zenhttp/packageformat.cpp
index 9c62c1f2d..267ce386c 100644
--- a/src/zenhttp/packageformat.cpp
+++ b/src/zenhttp/packageformat.cpp
@@ -36,6 +36,71 @@ const std::string_view HandlePrefix(":?#:");
typedef eastl::fixed_vector<IoBuffer, 16> IoBufferVec_t;
+/// Enforce local-ref path policy. Handle-based refs bypass the policy since they use OS handle security.
+/// If no policy is set, file-path local refs are rejected (fail-closed).
+static void
+ApplyLocalRefPolicy(const ILocalRefPolicy* Policy, const std::filesystem::path& Path)
+{
+ if (Policy)
+ {
+ Policy->ValidatePath(Path);
+ }
+ else
+ {
+ throw std::invalid_argument("local file reference rejected: no validation policy");
+ }
+}
+
+// Validates the CbPackageHeader magic and attachment count. Returns the total
+// chunk count (AttachmentCount + 1, including the implicit root object).
+static uint32_t
+ValidatePackageHeader(const CbPackageHeader& Hdr)
+{
+ if (Hdr.HeaderMagic != kCbPkgMagic)
+ {
+ throw std::invalid_argument(
+ fmt::format("invalid CbPackage header magic, expected {0:x}, got {1:x}", static_cast<uint32_t>(kCbPkgMagic), Hdr.HeaderMagic));
+ }
+ // ChunkCount is AttachmentCount + 1 (the root object is implicit). Guard against
+ // UINT32_MAX wrapping to 0, which would bypass subsequent size checks.
+ if (Hdr.AttachmentCount == UINT32_MAX)
+ {
+ throw std::invalid_argument("invalid CbPackage, attachment count overflow");
+ }
+ return Hdr.AttachmentCount + 1;
+}
+
+struct ValidatedLocalRef
+{
+ bool Valid = false;
+ const CbAttachmentReferenceHeader* Header = nullptr;
+ std::string_view Path;
+ std::string Error;
+};
+
+// Validates that the attachment buffer contains a well-formed local reference
+// header and path. On failure, Valid is false and Error contains the reason.
+static ValidatedLocalRef
+ValidateLocalRef(const IoBuffer& AttachmentBuffer)
+{
+ if (AttachmentBuffer.Size() < sizeof(CbAttachmentReferenceHeader))
+ {
+ return {.Error = fmt::format("local ref attachment too small for header (size {})", AttachmentBuffer.Size())};
+ }
+
+ const CbAttachmentReferenceHeader* AttachRefHdr = AttachmentBuffer.Data<CbAttachmentReferenceHeader>();
+
+ if (AttachmentBuffer.Size() < sizeof(CbAttachmentReferenceHeader) + AttachRefHdr->AbsolutePathLength)
+ {
+ return {.Error = fmt::format("local ref attachment too small for path (need {}, have {})",
+ sizeof(CbAttachmentReferenceHeader) + AttachRefHdr->AbsolutePathLength,
+ AttachmentBuffer.Size())};
+ }
+
+ const char* PathPointer = reinterpret_cast<const char*>(AttachRefHdr + 1);
+ return {.Valid = true, .Header = AttachRefHdr, .Path = std::string_view(PathPointer, AttachRefHdr->AbsolutePathLength)};
+}
+
IoBufferVec_t FormatPackageMessageInternal(const CbPackage& Data, FormatFlags Flags, void* TargetProcessHandle);
std::vector<IoBuffer>
@@ -361,7 +426,10 @@ IsPackageMessage(IoBuffer Payload)
}
CbPackage
-ParsePackageMessage(IoBuffer Payload, std::function<IoBuffer(const IoHash&, uint64_t)> CreateBuffer)
+ParsePackageMessage(IoBuffer Payload,
+ std::function<IoBuffer(const IoHash&, uint64_t)> CreateBuffer,
+ ParseFlags Flags,
+ const ILocalRefPolicy* Policy)
{
ZEN_TRACE_CPU("ParsePackageMessage");
@@ -372,17 +440,13 @@ ParsePackageMessage(IoBuffer Payload, std::function<IoBuffer(const IoHash&, uint
BinaryReader Reader(Payload);
- const CbPackageHeader* Hdr = reinterpret_cast<const CbPackageHeader*>(Reader.GetView(sizeof(CbPackageHeader)).GetData());
- if (Hdr->HeaderMagic != kCbPkgMagic)
- {
- throw std::invalid_argument(
- fmt::format("invalid CbPackage header magic, expected {0:x}, got {1:x}", static_cast<uint32_t>(kCbPkgMagic), Hdr->HeaderMagic));
- }
+ const CbPackageHeader* Hdr = reinterpret_cast<const CbPackageHeader*>(Reader.GetView(sizeof(CbPackageHeader)).GetData());
+ const uint32_t ChunkCount = ValidatePackageHeader(*Hdr);
Reader.Skip(sizeof(CbPackageHeader));
- const uint32_t ChunkCount = Hdr->AttachmentCount + 1;
-
- if (Reader.Remaining() < sizeof(CbAttachmentEntry) * ChunkCount)
+ // Widen to uint64_t so the multiplication cannot wrap on 32-bit.
+ const uint64_t AttachmentTableSize = uint64_t(sizeof(CbAttachmentEntry)) * ChunkCount;
+ if (Reader.Remaining() < AttachmentTableSize)
{
throw std::invalid_argument(fmt::format("invalid CbPackage, missing attachment entry data (need {} bytes, have {} bytes)",
sizeof(CbAttachmentEntry) * ChunkCount,
@@ -417,15 +481,22 @@ ParsePackageMessage(IoBuffer Payload, std::function<IoBuffer(const IoHash&, uint
if (Entry.Flags & CbAttachmentEntry::kIsLocalRef)
{
- // Marshal local reference - a "pointer" to the chunk backing file
-
- ZEN_ASSERT(AttachmentBuffer.Size() >= sizeof(CbAttachmentReferenceHeader));
+ if (!EnumHasAllFlags(Flags, ParseFlags::kAllowLocalReferences))
+ {
+ throw std::invalid_argument(
+ fmt::format("package contains local reference (attachment #{}) but local references are not allowed", i));
+ }
- const CbAttachmentReferenceHeader* AttachRefHdr = AttachmentBuffer.Data<CbAttachmentReferenceHeader>();
- const char* PathPointer = reinterpret_cast<const char*>(AttachRefHdr + 1);
+ // Marshal local reference - a "pointer" to the chunk backing file
- ZEN_ASSERT(AttachmentBuffer.Size() >= (sizeof(CbAttachmentReferenceHeader) + AttachRefHdr->AbsolutePathLength));
- std::string_view PathView(PathPointer, AttachRefHdr->AbsolutePathLength);
+ ValidatedLocalRef LocalRef = ValidateLocalRef(AttachmentBuffer);
+ if (!LocalRef.Valid)
+ {
+ MalformedAttachments.push_back(std::make_pair(i, fmt::format("{} for {}", LocalRef.Error, Entry.AttachmentHash)));
+ continue;
+ }
+ const CbAttachmentReferenceHeader* AttachRefHdr = LocalRef.Header;
+ std::string_view PathView = LocalRef.Path;
IoBuffer FullFileBuffer;
@@ -461,13 +532,29 @@ ParsePackageMessage(IoBuffer Payload, std::function<IoBuffer(const IoHash&, uint
}
else
{
+ ApplyLocalRefPolicy(Policy, Path);
FullFileBuffer = PartialFileBuffers.insert_or_assign(Path.string(), IoBufferBuilder::MakeFromFile(Path)).first->second;
}
}
if (FullFileBuffer)
{
- IoBuffer ChunkReference = AttachRefHdr->PayloadByteOffset == 0 && AttachRefHdr->PayloadByteSize == FullFileBuffer.GetSize()
+ // Guard against offset+size overflow or exceeding the file bounds.
+ const uint64_t FileSize = FullFileBuffer.GetSize();
+ if (AttachRefHdr->PayloadByteOffset > FileSize ||
+ AttachRefHdr->PayloadByteSize > FileSize - AttachRefHdr->PayloadByteOffset)
+ {
+ MalformedAttachments.push_back(
+ std::make_pair(i,
+ fmt::format("Local ref offset/size out of bounds (offset {}, size {}, file size {}) for {}",
+ AttachRefHdr->PayloadByteOffset,
+ AttachRefHdr->PayloadByteSize,
+ FileSize,
+ Entry.AttachmentHash)));
+ continue;
+ }
+
+ IoBuffer ChunkReference = AttachRefHdr->PayloadByteOffset == 0 && AttachRefHdr->PayloadByteSize == FileSize
? FullFileBuffer
: IoBuffer(FullFileBuffer, AttachRefHdr->PayloadByteOffset, AttachRefHdr->PayloadByteSize);
@@ -630,7 +717,9 @@ ParsePackageMessageWithLegacyFallback(const IoBuffer& Response, CbPackage& OutPa
return OutPackage.TryLoad(Response);
}
-CbPackageReader::CbPackageReader() : m_CreateBuffer([](const IoHash&, uint64_t Size) -> IoBuffer { return IoBuffer{Size}; })
+CbPackageReader::CbPackageReader(ParseFlags Flags)
+: m_Flags(Flags)
+, m_CreateBuffer([](const IoHash&, uint64_t Size) -> IoBuffer { return IoBuffer{Size}; })
{
}
@@ -644,6 +733,12 @@ CbPackageReader::SetPayloadBufferCreator(std::function<IoBuffer(const IoHash& Ci
m_CreateBuffer = CreateBuffer;
}
+void
+CbPackageReader::SetLocalRefPolicy(const ILocalRefPolicy* Policy)
+{
+ m_LocalRefPolicy = Policy;
+}
+
uint64_t
CbPackageReader::ProcessPackageHeaderData(const void* Data, uint64_t DataBytes)
{
@@ -657,12 +752,14 @@ CbPackageReader::ProcessPackageHeaderData(const void* Data, uint64_t DataBytes)
return sizeof m_PackageHeader;
case State::kReadingHeader:
- ZEN_ASSERT(DataBytes == sizeof m_PackageHeader);
- memcpy(&m_PackageHeader, Data, sizeof m_PackageHeader);
- ZEN_ASSERT(m_PackageHeader.HeaderMagic == kCbPkgMagic);
- m_CurrentState = State::kReadingAttachmentEntries;
- m_AttachmentEntries.resize(m_PackageHeader.AttachmentCount + 1);
- return (m_PackageHeader.AttachmentCount + 1) * sizeof(CbAttachmentEntry);
+ {
+ ZEN_ASSERT(DataBytes == sizeof m_PackageHeader);
+ memcpy(&m_PackageHeader, Data, sizeof m_PackageHeader);
+ const uint32_t ChunkCount = ValidatePackageHeader(m_PackageHeader);
+ m_CurrentState = State::kReadingAttachmentEntries;
+ m_AttachmentEntries.resize(ChunkCount);
+ return uint64_t(ChunkCount) * sizeof(CbAttachmentEntry);
+ }
case State::kReadingAttachmentEntries:
ZEN_ASSERT(DataBytes == ((m_PackageHeader.AttachmentCount + 1) * sizeof(CbAttachmentEntry)));
@@ -691,16 +788,19 @@ CbPackageReader::MarshalLocalChunkReference(IoBuffer AttachmentBuffer)
{
// Marshal local reference - a "pointer" to the chunk backing file
- ZEN_ASSERT(AttachmentBuffer.Size() >= sizeof(CbAttachmentReferenceHeader));
-
- const CbAttachmentReferenceHeader* AttachRefHdr = AttachmentBuffer.Data<CbAttachmentReferenceHeader>();
- const char8_t* PathPointer = reinterpret_cast<const char8_t*>(AttachRefHdr + 1);
-
- ZEN_ASSERT(AttachmentBuffer.Size() >= (sizeof(CbAttachmentReferenceHeader) + AttachRefHdr->AbsolutePathLength));
+ ValidatedLocalRef LocalRef = ValidateLocalRef(AttachmentBuffer);
+ if (!LocalRef.Valid)
+ {
+ throw std::invalid_argument(LocalRef.Error);
+ }
- std::u8string_view PathView{PathPointer, AttachRefHdr->AbsolutePathLength};
+ const CbAttachmentReferenceHeader* AttachRefHdr = LocalRef.Header;
+ std::filesystem::path Path(Utf8ToWide(LocalRef.Path));
- std::filesystem::path Path{PathView};
+ if (!LocalRef.Path.starts_with(HandlePrefix))
+ {
+ ApplyLocalRefPolicy(m_LocalRefPolicy, Path);
+ }
IoBuffer ChunkReference = IoBufferBuilder::MakeFromFile(Path, AttachRefHdr->PayloadByteOffset, AttachRefHdr->PayloadByteSize);
@@ -714,6 +814,17 @@ CbPackageReader::MarshalLocalChunkReference(IoBuffer AttachmentBuffer)
AttachRefHdr->PayloadByteSize));
}
+ // MakeFromFile silently clamps offset+size to the file size. Detect this
+ // to avoid returning a short buffer that could cause subtle downstream issues.
+ if (ChunkReference.GetSize() != AttachRefHdr->PayloadByteSize)
+ {
+ throw std::invalid_argument(fmt::format("local ref offset/size out of bounds for '{}' (requested offset {}, size {}, got size {})",
+ PathToUtf8(Path),
+ AttachRefHdr->PayloadByteOffset,
+ AttachRefHdr->PayloadByteSize,
+ ChunkReference.GetSize()));
+ }
+
return ChunkReference;
};
@@ -732,6 +843,13 @@ CbPackageReader::Finalize()
{
IoBuffer AttachmentBuffer = m_PayloadBuffers[CurrentAttachmentIndex];
+ if ((Entry.Flags & CbAttachmentEntry::kIsLocalRef) && !EnumHasAllFlags(m_Flags, ParseFlags::kAllowLocalReferences))
+ {
+ throw std::invalid_argument(
+ fmt::format("package contains local reference (attachment #{}) but local references are not allowed",
+ CurrentAttachmentIndex));
+ }
+
if (CurrentAttachmentIndex == 0)
{
// Root object
@@ -815,6 +933,13 @@ CbPackageReader::Finalize()
TEST_SUITE_BEGIN("http.packageformat");
+/// Permissive policy that allows any path, for use in tests that exercise local ref
+/// functionality but are not testing path validation.
+struct PermissiveLocalRefPolicy : public ILocalRefPolicy
+{
+ void ValidatePath(const std::filesystem::path&) const override {}
+};
+
TEST_CASE("CbPackage.Serialization")
{
// Make a test package
@@ -922,6 +1047,169 @@ TEST_CASE("CbPackage.LocalRef")
RemainingBytes -= ByteCount;
};
+ PermissiveLocalRefPolicy AllowAllPolicy;
+ CbPackageReader Reader(ParseFlags::kAllowLocalReferences);
+ Reader.SetLocalRefPolicy(&AllowAllPolicy);
+ uint64_t InitialRead = Reader.ProcessPackageHeaderData(nullptr, 0);
+ uint64_t NextBytes = Reader.ProcessPackageHeaderData(ConsumeBytes(InitialRead), InitialRead);
+ NextBytes = Reader.ProcessPackageHeaderData(ConsumeBytes(NextBytes), NextBytes);
+ auto Buffers = Reader.GetPayloadBuffers();
+
+ for (auto& PayloadBuffer : Buffers)
+ {
+ CopyBytes(PayloadBuffer.MutableData(), PayloadBuffer.GetSize());
+ }
+
+ Reader.Finalize();
+}
+
+TEST_CASE("CbPackage.Validation.TruncatedHeader")
+{
+ // Payload too small for a CbPackageHeader
+ uint8_t Bytes[] = {0xcc, 0xaa, 0x77, 0xaa};
+ IoBuffer Payload(IoBuffer::Wrap, Bytes, sizeof(Bytes));
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.BadMagic")
+{
+ CbPackageHeader Hdr{};
+ Hdr.HeaderMagic = 0xDEADBEEF;
+ Hdr.AttachmentCount = 0;
+ IoBuffer Payload(IoBuffer::Wrap, &Hdr, sizeof(Hdr));
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.AttachmentCountOverflow")
+{
+ CbPackageHeader Hdr{};
+ Hdr.HeaderMagic = kCbPkgMagic;
+ Hdr.AttachmentCount = UINT32_MAX;
+ IoBuffer Payload(IoBuffer::Wrap, &Hdr, sizeof(Hdr));
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.TruncatedAttachmentTable")
+{
+ // Valid header but not enough data for the attachment entries
+ CbPackageHeader Hdr{};
+ Hdr.HeaderMagic = kCbPkgMagic;
+ Hdr.AttachmentCount = 10;
+ IoBuffer Payload(IoBuffer::Wrap, &Hdr, sizeof(Hdr));
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.TruncatedAttachmentData")
+{
+ // Valid header + one attachment entry claiming more data than available
+ std::vector<uint8_t> Data(sizeof(CbPackageHeader) + sizeof(CbAttachmentEntry));
+
+ CbPackageHeader* Hdr = reinterpret_cast<CbPackageHeader*>(Data.data());
+ Hdr->HeaderMagic = kCbPkgMagic;
+ Hdr->AttachmentCount = 0; // ChunkCount = 1 (root object)
+
+ CbAttachmentEntry* Entry = reinterpret_cast<CbAttachmentEntry*>(Data.data() + sizeof(CbPackageHeader));
+ Entry->PayloadSize = 9999; // way more than available
+ Entry->Flags = CbAttachmentEntry::kIsObject;
+ Entry->AttachmentHash = IoHash();
+
+ IoBuffer Payload(IoBuffer::Wrap, Data.data(), Data.size());
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.LocalRefRejectedByDefault")
+{
+ // Build a valid package with local refs backed by compressed-format files,
+ // then verify it's rejected with default ParseFlags and accepted when allowed.
+ ScopedTemporaryDirectory TempDir;
+ auto Path1 = TempDir.Path() / "abcd";
+ auto Path2 = TempDir.Path() / "efgh";
+
+ // Compress data and write to disk, then create file-backed compressed attachments.
+ // The files must contain compressed-format data because ParsePackageMessage expects it
+ // when resolving local refs.
+ CompressedBuffer Comp1 =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("abcd")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+ CompressedBuffer Comp2 =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("efgh")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+
+ IoHash Hash1 = Comp1.DecodeRawHash();
+ IoHash Hash2 = Comp2.DecodeRawHash();
+
+ {
+ IoBuffer Buf1 = Comp1.GetCompressed().Flatten().AsIoBuffer();
+ IoBuffer Buf2 = Comp2.GetCompressed().Flatten().AsIoBuffer();
+ WriteFile(Path1, Buf1);
+ WriteFile(Path2, Buf2);
+ }
+
+ // Create attachments from file-backed buffers so FormatPackageMessage uses local refs
+ CbAttachment Attach1{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(Path1)), Hash1};
+ CbAttachment Attach2{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(Path2)), Hash2};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("abcd", Attach1);
+ Cbo.AddAttachment("efgh", Attach2);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach1);
+ Pkg.AddAttachment(Attach2);
+ Pkg.SetObject(Cbo.Save());
+
+ IoBuffer Payload = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten().AsIoBuffer();
+
+ // Default flags should reject local refs
+ CHECK_THROWS_AS(ParsePackageMessage(Payload), std::invalid_argument);
+
+ // With kAllowLocalReferences + a permissive policy, the local-ref gate is passed (the full round-trip
+ // for local refs through ParsePackageMessage is covered by CbPackage.LocalRef via CbPackageReader)
+ PermissiveLocalRefPolicy AllowAllPolicy;
+ CbPackage Result = ParsePackageMessage(Payload, {}, ParseFlags::kAllowLocalReferences, &AllowAllPolicy);
+ CHECK(Result.GetObject());
+ CHECK(Result.GetAttachments().size() == 2);
+}
+
+TEST_CASE("CbPackage.Validation.LocalRefRejectedByReader")
+{
+ // Same test but via CbPackageReader
+ ScopedTemporaryDirectory TempDir;
+ auto FilePath = TempDir.Path() / "testdata";
+
+ {
+ IoBuffer Buf = IoBufferBuilder::MakeCloneFromMemory(MakeMemoryView("testdata"));
+ WriteFile(FilePath, Buf);
+ }
+
+ IoBuffer FileBuffer = IoBufferBuilder::MakeFromFile(FilePath);
+ CbAttachment Attach{SharedBuffer(FileBuffer)};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("data", Attach);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach);
+ Pkg.SetObject(Cbo.Save());
+
+ SharedBuffer Buffer = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten();
+ const uint8_t* CursorPtr = reinterpret_cast<const uint8_t*>(Buffer.GetData());
+ uint64_t RemainingBytes = Buffer.GetSize();
+
+ auto ConsumeBytes = [&](uint64_t ByteCount) {
+ ZEN_ASSERT(ByteCount <= RemainingBytes);
+ void* ReturnPtr = (void*)CursorPtr;
+ CursorPtr += ByteCount;
+ RemainingBytes -= ByteCount;
+ return ReturnPtr;
+ };
+
+ auto CopyBytes = [&](void* TargetBuffer, uint64_t ByteCount) {
+ ZEN_ASSERT(ByteCount <= RemainingBytes);
+ memcpy(TargetBuffer, CursorPtr, ByteCount);
+ CursorPtr += ByteCount;
+ RemainingBytes -= ByteCount;
+ };
+
+ // Default flags should reject
CbPackageReader Reader;
uint64_t InitialRead = Reader.ProcessPackageHeaderData(nullptr, 0);
uint64_t NextBytes = Reader.ProcessPackageHeaderData(ConsumeBytes(InitialRead), InitialRead);
@@ -933,7 +1221,199 @@ TEST_CASE("CbPackage.LocalRef")
CopyBytes(PayloadBuffer.MutableData(), PayloadBuffer.GetSize());
}
- Reader.Finalize();
+ CHECK_THROWS_AS(Reader.Finalize(), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.BadMagicViaReader")
+{
+ CbPackageHeader Hdr{};
+ Hdr.HeaderMagic = 0xBADCAFE;
+ Hdr.AttachmentCount = 0;
+
+ CbPackageReader Reader;
+ uint64_t InitialRead = Reader.ProcessPackageHeaderData(nullptr, 0);
+ CHECK_THROWS_AS(Reader.ProcessPackageHeaderData(&Hdr, InitialRead), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.Validation.AttachmentCountOverflowViaReader")
+{
+ CbPackageHeader Hdr{};
+ Hdr.HeaderMagic = kCbPkgMagic;
+ Hdr.AttachmentCount = UINT32_MAX;
+
+ CbPackageReader Reader;
+ uint64_t InitialRead = Reader.ProcessPackageHeaderData(nullptr, 0);
+ CHECK_THROWS_AS(Reader.ProcessPackageHeaderData(&Hdr, InitialRead), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.LocalRefPolicy.PathOutsideRoot")
+{
+ // A file outside the allowed root should be rejected by the policy
+ ScopedTemporaryDirectory AllowedRoot;
+ ScopedTemporaryDirectory OutsideDir;
+
+ auto OutsidePath = OutsideDir.Path() / "secret.dat";
+ {
+ IoBuffer Buf = IoBufferBuilder::MakeCloneFromMemory(MakeMemoryView("secret"));
+ WriteFile(OutsidePath, Buf);
+ }
+
+ // Create file-backed compressed attachment from outside root
+ CompressedBuffer Comp =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("secret")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+ IoHash Hash = Comp.DecodeRawHash();
+ {
+ IoBuffer Buf = Comp.GetCompressed().Flatten().AsIoBuffer();
+ WriteFile(OutsidePath, Buf);
+ }
+
+ CbAttachment Attach{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(OutsidePath)), Hash};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("data", Attach);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach);
+ Pkg.SetObject(Cbo.Save());
+
+ IoBuffer Payload = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten().AsIoBuffer();
+
+ // Policy rooted at AllowedRoot should reject the file in OutsideDir
+ struct TestPolicy : public ILocalRefPolicy
+ {
+ std::string Root;
+ void ValidatePath(const std::filesystem::path& Path) const override
+ {
+ std::string CanonicalFile = std::filesystem::weakly_canonical(Path).string();
+ if (CanonicalFile.size() < Root.size() || CanonicalFile.compare(0, Root.size(), Root) != 0)
+ {
+ throw std::invalid_argument("path outside root");
+ }
+ }
+ } Policy;
+ Policy.Root = std::filesystem::weakly_canonical(AllowedRoot.Path()).string();
+
+ CHECK_THROWS_AS(ParsePackageMessage(Payload, {}, ParseFlags::kAllowLocalReferences, &Policy), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.LocalRefPolicy.PathInsideRoot")
+{
+ // A file inside the allowed root should be accepted by the policy
+ ScopedTemporaryDirectory TempRoot;
+
+ auto FilePath = TempRoot.Path() / "data.dat";
+
+ CompressedBuffer Comp =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("hello")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+ IoHash Hash = Comp.DecodeRawHash();
+ {
+ IoBuffer Buf = Comp.GetCompressed().Flatten().AsIoBuffer();
+ WriteFile(FilePath, Buf);
+ }
+
+ CbAttachment Attach{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(FilePath)), Hash};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("data", Attach);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach);
+ Pkg.SetObject(Cbo.Save());
+
+ IoBuffer Payload = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten().AsIoBuffer();
+
+ struct TestPolicy : public ILocalRefPolicy
+ {
+ std::string Root;
+ void ValidatePath(const std::filesystem::path& Path) const override
+ {
+ std::string CanonicalFile = std::filesystem::weakly_canonical(Path).string();
+ if (CanonicalFile.size() < Root.size() || CanonicalFile.compare(0, Root.size(), Root) != 0)
+ {
+ throw std::invalid_argument("path outside root");
+ }
+ }
+ } Policy;
+ Policy.Root = std::filesystem::weakly_canonical(TempRoot.Path()).string();
+
+ CbPackage Result = ParsePackageMessage(Payload, {}, ParseFlags::kAllowLocalReferences, &Policy);
+ CHECK(Result.GetObject());
+ CHECK(Result.GetAttachments().size() == 1);
+}
+
+TEST_CASE("CbPackage.LocalRefPolicy.PathTraversal")
+{
+ // A file path containing ".." that resolves outside root should be rejected
+ ScopedTemporaryDirectory TempRoot;
+ ScopedTemporaryDirectory OutsideDir;
+
+ auto OutsidePath = OutsideDir.Path() / "evil.dat";
+
+ CompressedBuffer Comp =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("evil")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+ IoHash Hash = Comp.DecodeRawHash();
+ {
+ IoBuffer Buf = Comp.GetCompressed().Flatten().AsIoBuffer();
+ WriteFile(OutsidePath, Buf);
+ }
+
+ CbAttachment Attach{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(OutsidePath)), Hash};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("data", Attach);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach);
+ Pkg.SetObject(Cbo.Save());
+
+ IoBuffer Payload = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten().AsIoBuffer();
+
+ struct TestPolicy : public ILocalRefPolicy
+ {
+ std::string Root;
+ void ValidatePath(const std::filesystem::path& Path) const override
+ {
+ std::string CanonicalFile = std::filesystem::weakly_canonical(Path).string();
+ if (CanonicalFile.size() < Root.size() || CanonicalFile.compare(0, Root.size(), Root) != 0)
+ {
+ throw std::invalid_argument("path outside root");
+ }
+ }
+ } Policy;
+ // Root is TempRoot, but the file lives in OutsideDir
+ Policy.Root = std::filesystem::weakly_canonical(TempRoot.Path()).string();
+
+ CHECK_THROWS_AS(ParsePackageMessage(Payload, {}, ParseFlags::kAllowLocalReferences, &Policy), std::invalid_argument);
+}
+
+TEST_CASE("CbPackage.LocalRefPolicy.NoPolicyFailClosed")
+{
+ // When local refs are allowed but no policy is provided, file-path refs should be rejected
+ ScopedTemporaryDirectory TempDir;
+
+ auto FilePath = TempDir.Path() / "data.dat";
+
+ CompressedBuffer Comp =
+ CompressedBuffer::Compress(SharedBuffer::MakeView(MakeMemoryView("data")), OodleCompressor::NotSet, OodleCompressionLevel::None);
+ IoHash Hash = Comp.DecodeRawHash();
+ {
+ IoBuffer Buf = Comp.GetCompressed().Flatten().AsIoBuffer();
+ WriteFile(FilePath, Buf);
+ }
+
+ CbAttachment Attach{CompressedBuffer::FromCompressedNoValidate(IoBufferBuilder::MakeFromFile(FilePath)), Hash};
+
+ CbObjectWriter Cbo;
+ Cbo.AddAttachment("data", Attach);
+
+ CbPackage Pkg;
+ Pkg.AddAttachment(Attach);
+ Pkg.SetObject(Cbo.Save());
+
+ IoBuffer Payload = FormatPackageMessageBuffer(Pkg, FormatFlags::kAllowLocalReferences).Flatten().AsIoBuffer();
+
+ // kAllowLocalReferences but nullptr policy => fail-closed
+ CHECK_THROWS_AS(ParsePackageMessage(Payload, {}, ParseFlags::kAllowLocalReferences, nullptr), std::invalid_argument);
}
TEST_SUITE_END();