aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
* | /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | | | | * don't provide size if compressed buffer validation fails
* | caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-2512-28/+50
| | | | | | | | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* | stronger validation of payload existance (#229)Dan Engelbrecht2024-11-256-101/+197
| | | | | | | | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* | add missing projectstore expire time in gc log (#227)Dan Engelbrecht2024-11-251-0/+1
| |
* | split zencore/memory.h -> memoryview.h, memcmp.h (#228)Stefan Boberg2024-11-2522-55/+69
| | | | | | | | | | | | | | minor clean-up `zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
* | Insights-compatible memory tracking (#214)Stefan Boberg2024-11-2558-169/+5988
| | | | | | | | | | | | | | | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* | fix inconsistencies in filecas due to failing to remove payload file during ↵Dan Engelbrecht2024-11-224-429/+438
| | | | | | | | | | | | | | | | GC (#224) make sure we rewrite filecas entries if chunk size changes (due to compression changes) hardening of move/write files in filecas if we encounter a filecas entry with mismatching size (due to pre-existing bug) we validate the file and update the index if we find a bad filecas file on disk we now attempt to remove it
* | fixed off-by-one in GetPidStatus (Linux) which might cause spurious errors ↵Stefan Boberg2024-11-221-2/+2
| | | | | | | | (#226)
* | fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | | | | | * bail attachment import on corrupt attachment block * changelog
* | memory/string support cleanup and additions (#220)Stefan Boberg2024-11-199-181/+188
| | | | | | | | | | | | | | | | | | * removed unused memory classes * added align.h alignment helpers used in upcoming changes * added char16_t StringLength * avoid memory alloc in SetCurrentThreadName * added command line parsing helpers to zencore/commandline.h * removed IoBuffer direct VirtualAlloc path
* | some fixes to timestamp caching in logging (#219)Stefan Boberg2024-11-182-7/+13
| | | | | | * some fixes to timestamp caching in logging
* | fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | | | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* | Dashboard: oplog tree view (#217)Martin Ridgers2024-11-1816-1385/+1682
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Turned tables and progress bars and friends into "widgets!" * A step to abstracting away a page's the internal DOM structure * Folded sector creation into Page and pivoted it to a widget host * Try and keep start/count as numbers regardless of input * No need for the entry table to be defined up front now * Add op count and log sixe to oplog list page * Cache left side toolbar object * Bounds count page start when building list of oplog entrie * Start/end navigation tools * Build rest of entry page while waiting for indexer to load * Consistent naming with other pages * Spacially consolidate fetching code * Hide fetch latency to speed up index generation workers * Extract dashboard structure from zen.js monolith * Fix breadcrumbs after restructuring * Add view link to actions cell of oplogs list * Generator to enumerate names of entries in indexer * Methods for simple traversal of component relations * is() to check if a component is of a certain type * Extend attr() to get and unset a component's attributes * Unsetting all styles of anchor tags was underisrable * Restore page name as id of container element * A tree view of an oplog * Move helper class out to private module scope * Small tweak to use left var that already exists * Changelog update * Updated frontend .zip archive
* | oplog prep gc fix (#216)Dan Engelbrecht2024-11-1518-453/+1494
| | | | | | | | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* | fixed some issues with ZenServerInstance::SpawnServer (#218)Stefan Boberg2024-11-142-17/+27
| | | | | | | | * previously it would assign a child identifier twice in some cases, which would lead to confusing log output * added pid as context in debug logging when launching a process
* | Self-hosted dashboard: Searchable oplog and links between oplog entry ↵v5.5.12-pre0Martin Ridgers2024-11-1110-372/+1000
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependencies (#213) * Consistent use of semicolons * Added fallback if oplog entry assumptions do not hold * 'marker' and 'expired' cells were incorrectly friendly * Two spaces when there should only be one * Robustness against .text(undefined) calls * A single step into JavaScript modules * Turned Fetcher into a module * Friendly into a module * Specialise Cbo field name comparison as TextDecoder() is very slow * Prefer is_named() over get_name() * Incorrect logic checking if a server reply was okay * Try and make sure it's always numbers that flow through Friendly * Added a progress bar component * Swap key and package hash columns * CbObject cloning * Dark and light themes depending on browser settings * Adjust styling of input boxes * Add theme swatches to test page * Turns out one can nest CSS selectors * Separate swatch for links/actions * Generate theme by lerping intermediate colours * Clearer progress bar * Chromium was complaining about label-less input elements * Promise-based cache using an IndexedDb * WebWorker for generating map of package ids to names * Indexer class for building, loading, and saving map of ids to names * Added links to oplog entries of an entry's dependencies * This doesn't need to be decorated as async any longer * Implemented oplog searching * View and drop make no sense on package data payloads * Rudimentary search result truncation * Updated changelog * Updated HTML zip archive
* | memory leak fix (GetBatchHandle) (#215)Stefan Boberg2024-11-081-0/+4
| | | | | | This fixes a memory leak which would cause stale handles to accumulate until process shutdown. Each cache get operation would therefore leak some memory.
* | Make op key and file path matching in `zen oplog-mirror` case insensitive (#212)Dan Engelbrecht2024-11-061-3/+18
| |
* | project name list improvement (#211)Dan Engelbrecht2024-11-061-3/+33
| | | | | | * provide shorter project store name in list when not conflicting
* | oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-062-7/+11
| | | | | | * make sure we track added attachments properly in oplogs
* | project details cmd resolve prj oplog (#209)Dan Engelbrecht2024-11-063-35/+23
| | | | | | | | | | * auto-resolve project name if only one project exist * always try to resolve project and oplog name in project-details command * project-details -> project-op-details
* | Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-0612-150/+457
| | | | | | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* | sponsor process attach hardening (#208)Dan Engelbrecht2024-11-043-6/+11
| | | | | | * make sure to clear sponsor slot if pickup does not happen
* | workspace share security (#192)Dan Engelbrecht2024-10-2314-654/+1688
| | | | | | | | | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* | fix gc date (#204)Dan Engelbrecht2024-10-221-1/+1
| | | | | | * fix month conversion in ToDateTime
* | Use a smaller thread pool during pre-cache phase of GC to reduce memory ↵Dan Engelbrecht2024-10-221-7/+11
| | | | | | | | pressure (#205)
* | bucket size queries (#203)Dan Engelbrecht2024-10-218-14/+258
| | | | | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* | validate compact binary rpc requests before trying to parse them (#200)Dan Engelbrecht2024-10-161-1/+10
| |
* | safer path from handle (#195)Dan Engelbrecht2024-10-169-65/+104
| | | | | | * remove PathFromHandle that throws to give better context on failures
* | add --dryrun instructions to projectstore drop (#199)Dan Engelbrecht2024-10-161-9/+15
| |
* | make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-163-21/+21
| | | | | | * make sure to initialize stats in file, jupiter and zen remote stores
* | Mirror command: fix utf8 paths mapping to native code page (#197)Matt Peters2024-10-151-2/+2
| | | | | | | | | | * Mirror command: need to keep n unordered_set of filesystem::path rather than converting to string(), because string() does not work for unicode paths. * Mirror: Store files in unordered_set as u8string rather than std::filesystem::path; std::filesystem::path does not support unordered_sest hashing on linux.
* | Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-147-104/+221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
* | Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-117-37/+410
| | | | | | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
* | oplog mirror and vfs utf8 paths (#189)Dan Engelbrecht2024-10-102-6/+6
| | | | | | * assume "filename" fields in oplog entries are utf8 paths
* | Dashboard: alignment of complex stats & logo goes home (#187)Martin Ridgers2024-10-103-57/+78
| | | | | | | | | | | | | | * Unselectable logo text * Revisit alignment of complex request/bytes stats * ZenStore logo takes you home
* | don't read chunks into memory during cache batch fetch unless we may cache ↵Dan Engelbrecht2024-10-092-13/+14
| | | | | | | | | | them in memory (#188) * Don't read chunks into memory during cache batch fetch unless we may cache them in memory
* | Mrp/referencedset (#174)Matt Peters2024-10-047-15/+15
| | | | | | | | * zen.exe commands: Add initializers for all POD types; this fixes incorrect behavior for flags in debug * README.md: Fix typo in --no-dry-run line
* | add automatic decompression to cache-get (default on) and oplog-mirror ↵Dan Engelbrecht2024-10-044-3/+35
| | | | | | | | (default off) (#186)
* | improve naming and feedback in zen commands (#185)Dan Engelbrecht2024-10-033-13/+18
| | | | | | | | * rename cache-get opton attachmentid -> attachmenthash for improved clarity * add help info for --dry-run in DropProjectCommand
* | remove gc v1 (#121)Dan Engelbrecht2024-10-0320-3120/+115
| | | | | | | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* | - Improvement: Self-hosted dashboard (#181)Martin Ridgers2024-10-035-161/+1161
| | | | | | Self-hosted dashboard gets oplog entry view and a stats browser
* | cache get command (#183)Dan Engelbrecht2024-10-038-45/+200
| | | | | | | | * move TryParseObjectId and TryParseIoHash to Oid::TryParse and IoHash::TryParse respectively * zen cache-get command
* | simplified CleanDirectory implementation (#182)Stefan Boberg2024-10-031-26/+8
| | | | | | we use the std implementation for all normal cases now. The Windows-only path still exists for VFS cleanup
* | gc block size target max size (#180)Dan Engelbrecht2024-10-024-42/+67
| | | | | | | | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* | Porject -> ProjectStefan Boberg2024-10-021-1/+1
| |
* | optimize gc reference sort (#179)Dan Engelbrecht2024-10-012-37/+40
| | | | | | | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* | use alternate IoHash comparision function (#177)v5.5.8-pre5Dan Engelbrecht2024-09-301-4/+23
| | | | | | * Use alternate IoHash comparision function - reduces KeepUnusedReferences execution time by ~20%
* | gc command attachment options (#176)Dan Engelbrecht2024-09-3012-42/+100
| | | | | | * zen command - add options to control meta data cache when triggering gc
* | optimize startup time (#175)Dan Engelbrecht2024-09-306-41/+32
| | | | | | | | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-14 01:04:04 +0000