aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
* move WriteToTempFile to basicfile.h (#283)Dan Engelbrecht2025-02-121-60/+40
| | | add helper constructors to BasicFile
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-127-240/+122
|
* Fix workspace shares reply array (#280)Tomasz Obrębski2025-02-031-3/+3
| | | | Workspace shares were sent in the '/ws' reply as repeating objects 'shares' instead of a 'shares' array of objects
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-224-25/+38
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-228-186/+159
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* add missing statsd.enabled and gc.projectstore.duration.seconds (#275)Dan Engelbrecht2025-01-161-10/+13
| | | | | - Bugfix: Fix parsing of workspace options in Lua config - Bugfix: Add missing Lua option for option `--gc-projectstore-duration-seconds` - Bugfix: Add missing Lua mapping option to `--statsd` command line option
* export oplog to zen target url (#274)Dan Engelbrecht2025-01-161-1/+1
| | | | * fix oplog target url for oplog export to remote zenserver * verify that chunking is allowed before chunking loose files
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-1614-933/+17
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-1/+4
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* don't add overhead of verifying oplog presence on disk for "getchunks" rpc ↵Dan Engelbrecht2024-12-181-3/+5
| | | | call (#269)
* remove all referenced attachments in op from pending chunk references (#267)Dan Engelbrecht2024-12-171-5/+7
|
* more memory tagging and fixes (#263)Stefan Boberg2024-12-162-2/+139
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Implemented temporary workaround for long paths in ComputeOpKeyStefan Boberg2024-12-131-3/+72
| | | | also added a test case to exercise the logic
* fixed XXH3_128Stream so it initializes the state properlyStefan Boberg2024-12-131-1/+1
| | | | the old version is still present for now, with a _deprecated suffix
* added ComputeOpKey so all instances of mapping key -> Oid is in a single placeStefan Boberg2024-12-131-20/+19
|
* Dashboard: view -> list rename, table style fix, file name appended to ↵Martin Ridgers2024-12-125-5/+8
| | | | | | | | | | | | | | | downloads (#264) * Single-column tables could overflow their maximum width * Suffix oplog entry data's file name when downloading * Renamed "view" link to "list" * Ensure all undesirable characters are removed from page name * Updated embedded frontend Zip archive * Wrote some entries into the changelog
* Builds API remote project store (#258)Dan Engelbrecht2024-12-1210-219/+1170
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* Memory tracking improvements (#262)Stefan Boberg2024-12-111-0/+4
| | | | | * added LLM tag to properly tag RPC allocations * annotated some more httpsys functions with memory tags * only emit memory scope events if the active tag is different from the new tag
* reduce oplog job name (#261)Dan Engelbrecht2024-12-111-2/+2
| | | * reduce length of oplog import/export job name
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-1241/+1450
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* global open process cache for projectstore (#257)Dan Engelbrecht2024-12-056-31/+64
| | | | * move openprocess cache to central location * enable openprocesscache in projectstore so "getchunks" can send filehandles when requested
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-055-216/+1780
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* ODR fixStefan Boberg2024-12-031-3/+3
|
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-022-12/+12
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-026-3/+72
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-282-334/+428
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix oplog index path reading error (#246)Dan Engelbrecht2024-11-281-30/+41
| | | * when reading paths for oplog index, make sure we don't point our string view to potentially stale memory
* Dashboard CSS fixes and archival of a partial treemap view (#242)Martin Ridgers2024-11-284-6/+194
| | | | | | | | | | | * Input boxes' text was unreadable when using the dark theme * Change from margins to padding top/bottom - easier to reason about vertical styling. * A treemap. Not used anywhere and not finished. Submitting so it isn't lost * Prevent tables' first content columns from collapsing * Dashboardk .zip archive update
* set content type correctly for getchunkrange (#241)Dan Engelbrecht2024-11-281-3/+3
|
* skip rawsize/size in getchunksinfo if chunk is missing (#238)Dan Engelbrecht2024-11-261-2/+2
|
* Dashboard: display package data sizes in oplog entry and tree views. (#232)Martin Ridgers2024-11-268-42/+168
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Wrong divisor for friendly giga-values * We want Explorer style for kilo/kibi units; round up * var -> const - zero idea if this matters * Include sum of an entry's package data sizes in index * Method to enurate all properties of a loaded oplog index * Include bulkdata size in an oplog index * Found a space that was missing * Show package data sizes when viewing an oplog entry * Navigating a component tree would error out at the end of the chain * Parameterise friendly rounding * Added size and rawsize columns to oplog tree view * Sort of parameterised indexer's worker count and page size * Right-align size columns on entry view page * Updated frontend .zip archive * A changelog update
* snapshot crash (#234)Dan Engelbrecht2024-11-251-1/+6
| | | | * snapshot test * use proper IoBuffer constructor to take handle ownership
* fix oplog snapshot deadlock (#233)Dan Engelbrecht2024-11-251-31/+57
| | | | * store inlined chunk as temp files and store to Cid after oplog iteration is complete and ops updated * make sure we can get to the payload when doing `prep` for new ops
* /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | * don't provide size if compressed buffer validation fails
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-253-10/+19
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-253-44/+110
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* split zencore/memory.h -> memoryview.h, memcmp.h (#228)Stefan Boberg2024-11-251-1/+1
| | | | | | | minor clean-up `zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-254-12/+33
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | * bail attachment import on corrupt attachment block * changelog
* fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* Dashboard: oplog tree view (#217)Martin Ridgers2024-11-1816-1385/+1682
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Turned tables and progress bars and friends into "widgets!" * A step to abstracting away a page's the internal DOM structure * Folded sector creation into Page and pivoted it to a widget host * Try and keep start/count as numbers regardless of input * No need for the entry table to be defined up front now * Add op count and log sixe to oplog list page * Cache left side toolbar object * Bounds count page start when building list of oplog entrie * Start/end navigation tools * Build rest of entry page while waiting for indexer to load * Consistent naming with other pages * Spacially consolidate fetching code * Hide fetch latency to speed up index generation workers * Extract dashboard structure from zen.js monolith * Fix breadcrumbs after restructuring * Add view link to actions cell of oplogs list * Generator to enumerate names of entries in indexer * Methods for simple traversal of component relations * is() to check if a component is of a certain type * Extend attr() to get and unset a component's attributes * Unsetting all styles of anchor tags was underisrable * Restore page name as id of container element * A tree view of an oplog * Move helper class out to private module scope * Small tweak to use left var that already exists * Changelog update * Updated frontend .zip archive
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-157-213/+878
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* Self-hosted dashboard: Searchable oplog and links between oplog entry ↵v5.5.12-pre0Martin Ridgers2024-11-1110-372/+1000
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependencies (#213) * Consistent use of semicolons * Added fallback if oplog entry assumptions do not hold * 'marker' and 'expired' cells were incorrectly friendly * Two spaces when there should only be one * Robustness against .text(undefined) calls * A single step into JavaScript modules * Turned Fetcher into a module * Friendly into a module * Specialise Cbo field name comparison as TextDecoder() is very slow * Prefer is_named() over get_name() * Incorrect logic checking if a server reply was okay * Try and make sure it's always numbers that flow through Friendly * Added a progress bar component * Swap key and package hash columns * CbObject cloning * Dark and light themes depending on browser settings * Adjust styling of input boxes * Add theme swatches to test page * Turns out one can nest CSS selectors * Separate swatch for links/actions * Generate theme by lerping intermediate colours * Clearer progress bar * Chromium was complaining about label-less input elements * Promise-based cache using an IndexedDb * WebWorker for generating map of package ids to names * Indexer class for building, loading, and saving map of ids to names * Added links to oplog entries of an entry's dependencies * This doesn't need to be decorated as async any longer * Implemented oplog searching * View and drop make no sense on package data payloads * Rudimentary search result truncation * Updated changelog * Updated HTML zip archive
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-062-7/+11
| | | * make sure we track added attachments properly in oplogs
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-066-71/+120
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* sponsor process attach hardening (#208)Dan Engelbrecht2024-11-041-0/+1
| | | * make sure to clear sponsor slot if pickup does not happen
* workspace share security (#192)Dan Engelbrecht2024-10-235-148/+268
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* bucket size queries (#203)Dan Engelbrecht2024-10-211-0/+115
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-163-21/+21
| | | * make sure to initialize stats in file, jupiter and zen remote stores
* Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-147-104/+221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-22 08:34:56 +0000