diff options
| author | Stefan Boberg <[email protected]> | 2026-04-20 21:50:41 +0200 |
|---|---|---|
| committer | GitHub Enterprise <[email protected]> | 2026-04-20 21:50:41 +0200 |
| commit | 2dfb5da16b97a6c12e01977af5b5188522178a4e (patch) | |
| tree | 428aa0aa8e6079c64438931e0fd4f828c613c94d /thirdparty/raw_pdb/src/PDB_DBIStream.cpp | |
| parent | Add CompactString utility type (#990) (diff) | |
| download | archived-zen-2dfb5da16b97a6c12e01977af5b5188522178a4e.tar.xz archived-zen-2dfb5da16b97a6c12e01977af5b5188522178a4e.zip | |
zen trace analysis support (#945)
Integrates the **tourist** trace analysis library and builds a full `zen trace` command suite for working with Unreal Engine `.utrace` files.
### Trace analysis library (`thirdparty/tourist/`)
- Adds the tourist library as a third-party dependency with three modules: **foundation** (platform primitives, memory, scheduling), **trace** (UE Trace protocol decoding), and **analysis** (event dispatching and analyzer framework).
- Cross-platform support for Windows, Linux, and macOS.
### `zen trace` CLI commands (`src/zen/cmds/`, `src/zen/trace/`)
- **`zen trace analyze`** — Summarize a `.utrace` file: session metadata, thread inventory, command line + build configuration, CPU profiling scopes, timing, event rates, log messages, and (with symbols) memory allocation metrics including live-allocs dumps, callstack-keyed aggregation, and allocation churn. Optional HTML output for memory reports.
- **`zen trace inspect`** — Dump the event schema (declared types, fields, sizes) from a trace file.
- **`zen trace trim`** — Extract a time-window from a trace into a new `.utrace` file.
- **`zen trace serve`** — Launch a local HTTP server hosting an interactive trace viewer; opens in the default browser.
### Symbolication (`src/zen/trace/symbol_resolver.*`, `thirdparty/raw_pdb/`)
- Pluggable resolver with multiple backends: `pdb` (in-tree raw_pdb), `dbghelp` (Windows), `llvm-symbolizer` (all platforms), `atos` (macOS). An `auto` backend picks the best available tool per platform.
- Microsoft Symbol Server support: downloads PDBs on demand using a redirect-aware HTTP client.
- Local PDB cache keyed by image GUID preserves symbols across binary recompilation.
- Callstack trimming heuristic strips UE internal noise from reports.
- Binary analysis cache (`.ucache_z`) avoids re-resolving the same trace.
### Interactive trace viewer (`src/zen/frontend/html/`, `src/zen/trace/trace_viewer_service.*`)
- Timeline: scope-level detail, horizontal zoom/pan, vertical scrolling, viewport-driven loading with pre-computed LOD for responsive navigation of large traces.
- Thread grouping (collapsible sidebar sections) synthesized from name suffixes, natural sort order, visual distinction between lane threads and OS threads.
- Bookmark and region annotations; region categories with per-category toggles; bookmark marker toggle in the toolbar.
- Filterable Logs tab showing captured `UE_LOG` output.
- Stats tab with per-scope aggregate statistics.
- Memory tab with interactive allocation analysis and an allocation size histogram.
- CsvProfiler event parsing and chart UI.
### Other in-branch supporting changes
- **Cross-platform browser launcher** (`browser_launcher.{h,cpp}`) used by `trace serve`.
- **`ReciprocalU64`** fast 64-bit integer division (zencore/intmath) for trace analyzers.
- **`parallelsort`** cross-platform parallel sort helper (zenutil).
- Frontend zip build rule so the viewer's HTML assets are bundled into `zen.exe`.
- `/Zo` flag for better optimized debug info on Windows release builds.
- `trace-tests.cpp` in the `zen-test` harness (harness itself landed on main via #985).
Diffstat (limited to 'thirdparty/raw_pdb/src/PDB_DBIStream.cpp')
| -rw-r--r-- | thirdparty/raw_pdb/src/PDB_DBIStream.cpp | 335 |
1 files changed, 335 insertions, 0 deletions
diff --git a/thirdparty/raw_pdb/src/PDB_DBIStream.cpp b/thirdparty/raw_pdb/src/PDB_DBIStream.cpp new file mode 100644 index 000000000..5c9bf1512 --- /dev/null +++ b/thirdparty/raw_pdb/src/PDB_DBIStream.cpp @@ -0,0 +1,335 @@ +// Copyright 2011-2022, Molecular Matters GmbH <[email protected]> +// See LICENSE.txt for licensing details (2-clause BSD License: https://opensource.org/licenses/BSD-2-Clause) + +#include "PDB_PCH.h" +#include "PDB_DBIStream.h" +#include "PDB_RawFile.h" + + +namespace +{ + // the DBI stream always resides at index 3 + static constexpr const uint32_t DBIStreamIndex = 3u; + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetModuleInfoSubstreamOffset(const PDB::DBI::StreamHeader& /* dbiHeader */) PDB_NO_EXCEPT + { + return sizeof(PDB::DBI::StreamHeader); + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetSectionContributionSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetModuleInfoSubstreamOffset(dbiHeader) + dbiHeader.moduleInfoSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetSectionMapSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetSectionContributionSubstreamOffset(dbiHeader) + dbiHeader.sectionContributionSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetSourceInfoSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetSectionMapSubstreamOffset(dbiHeader) + dbiHeader.sectionMapSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetTypeServerMapSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetSourceInfoSubstreamOffset(dbiHeader) + dbiHeader.sourceInfoSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetECSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetTypeServerMapSubstreamOffset(dbiHeader) + dbiHeader.typeServerMapSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline uint32_t GetDebugHeaderSubstreamOffset(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return GetECSubstreamOffset(dbiHeader) + dbiHeader.ecSize; + } + + + // ------------------------------------------------------------------------------------------------ + // ------------------------------------------------------------------------------------------------ + PDB_NO_DISCARD static inline bool HasDebugHeaderSubstream(const PDB::DBI::StreamHeader& dbiHeader) PDB_NO_EXCEPT + { + return dbiHeader.optionalDebugHeaderSize != 0u; + } +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB::DBIStream::DBIStream(void) PDB_NO_EXCEPT + : m_header() + , m_stream() +{ +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB::DBIStream::DBIStream(const RawFile& file, const DBI::StreamHeader& header) PDB_NO_EXCEPT + : m_header(header) + , m_stream(file.CreateMSFStream<DirectMSFStream>(DBIStreamIndex)) +{ +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::HasValidDBIStream(const RawFile& file) PDB_NO_EXCEPT +{ + DirectMSFStream stream = file.CreateMSFStream<DirectMSFStream>(DBIStreamIndex); + if (stream.GetSize() < sizeof(DBI::StreamHeader)) + { + return ErrorCode::InvalidStream; + } + + const DBI::StreamHeader header = stream.ReadAtOffset<DBI::StreamHeader>(0u); + if (header.signature != DBI::StreamHeader::Signature) + { + return ErrorCode::InvalidSignature; + } + else if (header.version != DBI::StreamHeader::Version::V70) + { + return ErrorCode::UnknownVersion; + } + + return ErrorCode::Success; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::DBIStream PDB::CreateDBIStream(const RawFile& file) PDB_NO_EXCEPT +{ + DirectMSFStream stream = file.CreateMSFStream<DirectMSFStream>(DBIStreamIndex); + const DBI::StreamHeader header = stream.ReadAtOffset<DBI::StreamHeader>(0u); + + return DBIStream { file, header }; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::DBIStream::HasValidSymbolRecordStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + return (m_header.symbolRecordStreamIndex != PDB::NilStreamIndex) ? ErrorCode::Success : ErrorCode::InvalidStreamIndex; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::DBIStream::HasValidImageSectionStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + // the debug header stream is optional. if it's not there, we can't get the image section stream either. + if (!HasDebugHeaderSubstream(m_header)) + { + return ErrorCode::InvalidStreamIndex; + } + + // find the debug header sub-stream + const uint32_t debugHeaderOffset = GetDebugHeaderSubstreamOffset(m_header); + + // validate that we have enough data to read the debug header + // (the header field optionalDebugHeaderSize might claim there's a debug header, + // but the stream might not have enough data - this happens with some .ni.pdb files) + if (debugHeaderOffset + sizeof(DBI::DebugHeader) > m_stream.GetSize()) + { + return ErrorCode::InvalidStream; + } + + const DBI::DebugHeader& debugHeader = m_stream.ReadAtOffset<DBI::DebugHeader>(debugHeaderOffset); + + if (debugHeader.sectionHeaderStreamIndex == DBI::DebugHeader::InvalidStreamIndex) + { + return ErrorCode::InvalidStreamIndex; + } + + return ErrorCode::Success; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::DBIStream::HasValidPublicSymbolStream(const RawFile& file) const PDB_NO_EXCEPT +{ + if (m_header.publicStreamIndex == PDB::NilStreamIndex) + { + return ErrorCode::InvalidStreamIndex; + } + + DirectMSFStream publicStream = file.CreateMSFStream<DirectMSFStream>(m_header.publicStreamIndex); + + // the public symbol stream always begins with a header, we are not interested in that. + // following the public symbol stream header is a hash table header. + const HashTableHeader hashHeader = publicStream.ReadAtOffset<HashTableHeader>(sizeof(PublicStreamHeader)); + if (hashHeader.signature != HashTableHeader::Signature) + { + return ErrorCode::InvalidSignature; + } + else if (hashHeader.version != HashTableHeader::Version) + { + return ErrorCode::UnknownVersion; + } + + return ErrorCode::Success; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::DBIStream::HasValidGlobalSymbolStream(const RawFile& file) const PDB_NO_EXCEPT +{ + if (m_header.globalStreamIndex == PDB::NilStreamIndex) + { + return ErrorCode::InvalidStreamIndex; + } + + DirectMSFStream globalStream = file.CreateMSFStream<DirectMSFStream>(m_header.globalStreamIndex); + + // the global symbol stream starts with a hash table header + const HashTableHeader hashHeader = globalStream.ReadAtOffset<HashTableHeader>(0u); + if (hashHeader.signature != HashTableHeader::Signature) + { + return ErrorCode::InvalidSignature; + } + else if (hashHeader.version != HashTableHeader::Version) + { + return ErrorCode::UnknownVersion; + } + + return ErrorCode::Success; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ErrorCode PDB::DBIStream::HasValidSectionContributionStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + if (m_header.sectionContributionSize < sizeof(DBI::SectionContribution::Version)) + { + return ErrorCode::InvalidStream; + } + + // find the section contribution sub-stream + // https://llvm.org/docs/PDB/DbiStream.html#section-contribution-substream + const uint32_t streamOffset = GetSectionContributionSubstreamOffset(m_header); + + const DBI::SectionContribution::Version version = m_stream.ReadAtOffset<DBI::SectionContribution::Version>(streamOffset); + if (version != DBI::SectionContribution::Version::Ver60) + { + return ErrorCode::UnknownVersion; + } + + return ErrorCode::Success; +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::CoalescedMSFStream PDB::DBIStream::CreateSymbolRecordStream(const RawFile& file) const PDB_NO_EXCEPT +{ + // the symbol record stream holds the actual CodeView data of the symbols + return file.CreateMSFStream<CoalescedMSFStream>(m_header.symbolRecordStreamIndex); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ImageSectionStream PDB::DBIStream::CreateImageSectionStream(const RawFile& file) const PDB_NO_EXCEPT +{ + // find the debug header sub-stream + const uint32_t debugHeaderOffset = GetDebugHeaderSubstreamOffset(m_header); + const DBI::DebugHeader& debugHeader = m_stream.ReadAtOffset<DBI::DebugHeader>(debugHeaderOffset); + + // from there, grab the section header stream + return ImageSectionStream(file, debugHeader.sectionHeaderStreamIndex); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::PublicSymbolStream PDB::DBIStream::CreatePublicSymbolStream(const RawFile& file) const PDB_NO_EXCEPT +{ + DirectMSFStream publicStream = file.CreateMSFStream<DirectMSFStream>(m_header.publicStreamIndex); + + // the public symbol stream always begins with a header, we are not interested in that. + // following the public symbol stream header is a hash table header. + // we use this to work out how many symbol records are referenced by the public symbol stream. + const HashTableHeader hashHeader = publicStream.ReadAtOffset<HashTableHeader>(sizeof(PublicStreamHeader)); + const uint32_t recordCount = hashHeader.size / sizeof(HashRecord); + + return PublicSymbolStream(file, m_header.publicStreamIndex, recordCount); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::GlobalSymbolStream PDB::DBIStream::CreateGlobalSymbolStream(const RawFile& file) const PDB_NO_EXCEPT +{ + DirectMSFStream globalStream = file.CreateMSFStream<DirectMSFStream>(m_header.globalStreamIndex); + + // the global symbol stream starts with a hash table header. + // we use this to work out how many symbol records are referenced by the global symbol stream. + const HashTableHeader hashHeader = globalStream.ReadAtOffset<HashTableHeader>(0u); + const uint32_t recordCount = hashHeader.size / sizeof(HashRecord); + + return GlobalSymbolStream(file, m_header.globalStreamIndex, recordCount); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::SourceFileStream PDB::DBIStream::CreateSourceFileStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + // find the source info sub-stream + // https://llvm.org/docs/PDB/DbiStream.html#file-info-substream + const uint32_t streamOffset = GetSourceInfoSubstreamOffset(m_header); + + return SourceFileStream(m_stream, m_header.sourceInfoSize, streamOffset); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::SectionContributionStream PDB::DBIStream::CreateSectionContributionStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + // find the section contribution sub-stream + // https://llvm.org/docs/PDB/DbiStream.html#section-contribution-substream + const uint32_t streamOffset = GetSectionContributionSubstreamOffset(m_header); + + return SectionContributionStream(m_stream, m_header.sectionContributionSize - sizeof(DBI::SectionContribution::Version), streamOffset + sizeof(DBI::SectionContribution::Version)); +} + + +// ------------------------------------------------------------------------------------------------ +// ------------------------------------------------------------------------------------------------ +PDB_NO_DISCARD PDB::ModuleInfoStream PDB::DBIStream::CreateModuleInfoStream(const RawFile& /* file */) const PDB_NO_EXCEPT +{ + // find the module info sub-stream + // https://llvm.org/docs/PDB/DbiStream.html#module-info-substream + const uint32_t streamOffset = GetModuleInfoSubstreamOffset(m_header); + + return ModuleInfoStream(m_stream, m_header.moduleInfoSize, streamOffset); +} |