aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
* ODR violation fixStefan Boberg2024-12-031-2/+2
|
* reduce memory churn (#248)Stefan Boberg2024-12-025-11/+23
| | | | | * eliminated allocation in SetCurrentThreadName * reduced memory allocator activity in cache RPC response building * reduced allocations in compact binary building
* validate that root path exists for workspace before adding it (#250)Dan Engelbrecht2024-12-021-0/+5
|
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-022-12/+12
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-0227-34/+449
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-284-339/+447
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix oplog index path reading error (#246)Dan Engelbrecht2024-11-281-30/+41
| | | * when reading paths for oplog index, make sure we don't point our string view to potentially stale memory
* Dashboard CSS fixes and archival of a partial treemap view (#242)Martin Ridgers2024-11-284-6/+194
| | | | | | | | | | | * Input boxes' text was unreadable when using the dark theme * Change from margins to padding top/bottom - easier to reason about vertical styling. * A treemap. Not used anywhere and not finished. Submitting so it isn't lost * Prevent tables' first content columns from collapsing * Dashboardk .zip archive update
* set content type correctly for getchunkrange (#241)Dan Engelbrecht2024-11-281-3/+3
|
* add validation of payload responses in http client (#240)Dan Engelbrecht2024-11-271-220/+292
| | | if response payload does not validate properly do a retry if applicable
* use plain sorted array instead of map of vectors (#237)Dan Engelbrecht2024-11-272-22/+33
| | | | | * use plain sorted array instead of map of vectors * reserve vectors up front = 5% perf increase * don't do batch read of chunks if we have a single chunk -> 1% perf gain
* add missing shard lock in filecas compact (#239)Dan Engelbrecht2024-11-271-1/+6
| | | | * add missing shardlock during compact in filecas * add warning log when filecas fails to open a file it expects to be present
* skip rawsize/size in getchunksinfo if chunk is missing (#238)Dan Engelbrecht2024-11-261-2/+2
|
* Dashboard: display package data sizes in oplog entry and tree views. (#232)Martin Ridgers2024-11-268-42/+168
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Wrong divisor for friendly giga-values * We want Explorer style for kilo/kibi units; round up * var -> const - zero idea if this matters * Include sum of an entry's package data sizes in index * Method to enurate all properties of a loaded oplog index * Include bulkdata size in an oplog index * Found a space that was missing * Show package data sizes when viewing an oplog entry * Navigating a component tree would error out at the end of the chain * Parameterise friendly rounding * Added size and rawsize columns to oplog tree view * Sort of parameterised indexer's worker count and page size * Right-align size columns on entry view page * Updated frontend .zip archive * A changelog update
* snapshot crash (#234)Dan Engelbrecht2024-11-252-2/+102
| | | | * snapshot test * use proper IoBuffer constructor to take handle ownership
* fix oplog snapshot deadlock (#233)Dan Engelbrecht2024-11-251-31/+57
| | | | * store inlined chunk as temp files and store to Cid after oplog iteration is complete and ops updated * make sure we can get to the payload when doing `prep` for new ops
* /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | * don't provide size if compressed buffer validation fails
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-2512-28/+50
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-256-101/+197
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* add missing projectstore expire time in gc log (#227)Dan Engelbrecht2024-11-251-0/+1
|
* split zencore/memory.h -> memoryview.h, memcmp.h (#228)Stefan Boberg2024-11-2522-55/+69
| | | | | | | minor clean-up `zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-2558-169/+5988
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* fix inconsistencies in filecas due to failing to remove payload file during ↵Dan Engelbrecht2024-11-224-429/+438
| | | | | | | | GC (#224) make sure we rewrite filecas entries if chunk size changes (due to compression changes) hardening of move/write files in filecas if we encounter a filecas entry with mismatching size (due to pre-existing bug) we validate the file and update the index if we find a bad filecas file on disk we now attempt to remove it
* fixed off-by-one in GetPidStatus (Linux) which might cause spurious errors ↵Stefan Boberg2024-11-221-2/+2
| | | | (#226)
* fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | * bail attachment import on corrupt attachment block * changelog
* memory/string support cleanup and additions (#220)Stefan Boberg2024-11-199-181/+188
| | | | | | | | | * removed unused memory classes * added align.h alignment helpers used in upcoming changes * added char16_t StringLength * avoid memory alloc in SetCurrentThreadName * added command line parsing helpers to zencore/commandline.h * removed IoBuffer direct VirtualAlloc path
* some fixes to timestamp caching in logging (#219)Stefan Boberg2024-11-182-7/+13
| | | * some fixes to timestamp caching in logging
* fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* Dashboard: oplog tree view (#217)Martin Ridgers2024-11-1816-1385/+1682
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Turned tables and progress bars and friends into "widgets!" * A step to abstracting away a page's the internal DOM structure * Folded sector creation into Page and pivoted it to a widget host * Try and keep start/count as numbers regardless of input * No need for the entry table to be defined up front now * Add op count and log sixe to oplog list page * Cache left side toolbar object * Bounds count page start when building list of oplog entrie * Start/end navigation tools * Build rest of entry page while waiting for indexer to load * Consistent naming with other pages * Spacially consolidate fetching code * Hide fetch latency to speed up index generation workers * Extract dashboard structure from zen.js monolith * Fix breadcrumbs after restructuring * Add view link to actions cell of oplogs list * Generator to enumerate names of entries in indexer * Methods for simple traversal of component relations * is() to check if a component is of a certain type * Extend attr() to get and unset a component's attributes * Unsetting all styles of anchor tags was underisrable * Restore page name as id of container element * A tree view of an oplog * Move helper class out to private module scope * Small tweak to use left var that already exists * Changelog update * Updated frontend .zip archive
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-1518-453/+1494
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* fixed some issues with ZenServerInstance::SpawnServer (#218)Stefan Boberg2024-11-142-17/+27
| | | | * previously it would assign a child identifier twice in some cases, which would lead to confusing log output * added pid as context in debug logging when launching a process
* Self-hosted dashboard: Searchable oplog and links between oplog entry ↵v5.5.12-pre0Martin Ridgers2024-11-1110-372/+1000
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependencies (#213) * Consistent use of semicolons * Added fallback if oplog entry assumptions do not hold * 'marker' and 'expired' cells were incorrectly friendly * Two spaces when there should only be one * Robustness against .text(undefined) calls * A single step into JavaScript modules * Turned Fetcher into a module * Friendly into a module * Specialise Cbo field name comparison as TextDecoder() is very slow * Prefer is_named() over get_name() * Incorrect logic checking if a server reply was okay * Try and make sure it's always numbers that flow through Friendly * Added a progress bar component * Swap key and package hash columns * CbObject cloning * Dark and light themes depending on browser settings * Adjust styling of input boxes * Add theme swatches to test page * Turns out one can nest CSS selectors * Separate swatch for links/actions * Generate theme by lerping intermediate colours * Clearer progress bar * Chromium was complaining about label-less input elements * Promise-based cache using an IndexedDb * WebWorker for generating map of package ids to names * Indexer class for building, loading, and saving map of ids to names * Added links to oplog entries of an entry's dependencies * This doesn't need to be decorated as async any longer * Implemented oplog searching * View and drop make no sense on package data payloads * Rudimentary search result truncation * Updated changelog * Updated HTML zip archive
* memory leak fix (GetBatchHandle) (#215)Stefan Boberg2024-11-081-0/+4
| | | This fixes a memory leak which would cause stale handles to accumulate until process shutdown. Each cache get operation would therefore leak some memory.
* Make op key and file path matching in `zen oplog-mirror` case insensitive (#212)Dan Engelbrecht2024-11-061-3/+18
|
* project name list improvement (#211)Dan Engelbrecht2024-11-061-3/+33
| | | * provide shorter project store name in list when not conflicting
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-062-7/+11
| | | * make sure we track added attachments properly in oplogs
* project details cmd resolve prj oplog (#209)Dan Engelbrecht2024-11-063-35/+23
| | | | | * auto-resolve project name if only one project exist * always try to resolve project and oplog name in project-details command * project-details -> project-op-details
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-0612-150/+457
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* sponsor process attach hardening (#208)Dan Engelbrecht2024-11-043-6/+11
| | | * make sure to clear sponsor slot if pickup does not happen
* workspace share security (#192)Dan Engelbrecht2024-10-2314-654/+1688
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* fix gc date (#204)Dan Engelbrecht2024-10-221-1/+1
| | | * fix month conversion in ToDateTime
* Use a smaller thread pool during pre-cache phase of GC to reduce memory ↵Dan Engelbrecht2024-10-221-7/+11
| | | | pressure (#205)
* bucket size queries (#203)Dan Engelbrecht2024-10-218-14/+258
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* validate compact binary rpc requests before trying to parse them (#200)Dan Engelbrecht2024-10-161-1/+10
|
* safer path from handle (#195)Dan Engelbrecht2024-10-169-65/+104
| | | * remove PathFromHandle that throws to give better context on failures
* add --dryrun instructions to projectstore drop (#199)Dan Engelbrecht2024-10-161-9/+15
|
* make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-163-21/+21
| | | * make sure to initialize stats in file, jupiter and zen remote stores
* Mirror command: fix utf8 paths mapping to native code page (#197)Matt Peters2024-10-151-2/+2
| | | | | * Mirror command: need to keep n unordered_set of filesystem::path rather than converting to string(), because string() does not work for unicode paths. * Mirror: Store files in unordered_set as u8string rather than std::filesystem::path; std::filesystem::path does not support unordered_sest hashing on linux.
* Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-147-104/+221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
* Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-117-37/+410
| | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 22:46:09 +0000