aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore
Commit message (Collapse)AuthorAgeFilesLines
...
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-127-240/+122
|
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-221-4/+4
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-223-109/+95
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* export oplog to zen target url (#274)Dan Engelbrecht2025-01-161-1/+1
| | | | * fix oplog target url for oplog export to remote zenserver * verify that chunking is allowed before chunking loose files
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-164-4/+4
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-1/+4
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* don't add overhead of verifying oplog presence on disk for "getchunks" rpc ↵Dan Engelbrecht2024-12-181-3/+5
| | | | call (#269)
* remove all referenced attachments in op from pending chunk references (#267)Dan Engelbrecht2024-12-171-5/+7
|
* more memory tagging and fixes (#263)Stefan Boberg2024-12-162-2/+139
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Implemented temporary workaround for long paths in ComputeOpKeyStefan Boberg2024-12-131-3/+72
| | | | also added a test case to exercise the logic
* fixed XXH3_128Stream so it initializes the state properlyStefan Boberg2024-12-131-1/+1
| | | | the old version is still present for now, with a _deprecated suffix
* added ComputeOpKey so all instances of mapping key -> Oid is in a single placeStefan Boberg2024-12-131-20/+19
|
* Builds API remote project store (#258)Dan Engelbrecht2024-12-128-219/+971
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* reduce oplog job name (#261)Dan Engelbrecht2024-12-111-2/+2
| | | * reduce length of oplog import/export job name
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-1241/+1450
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* global open process cache for projectstore (#257)Dan Engelbrecht2024-12-052-14/+35
| | | | * move openprocess cache to central location * enable openprocesscache in projectstore so "getchunks" can send filehandles when requested
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-054-215/+1778
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-022-12/+12
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+11
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-282-334/+428
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix oplog index path reading error (#246)Dan Engelbrecht2024-11-281-30/+41
| | | * when reading paths for oplog index, make sure we don't point our string view to potentially stale memory
* set content type correctly for getchunkrange (#241)Dan Engelbrecht2024-11-281-3/+3
|
* skip rawsize/size in getchunksinfo if chunk is missing (#238)Dan Engelbrecht2024-11-261-2/+2
|
* snapshot crash (#234)Dan Engelbrecht2024-11-251-1/+6
| | | | * snapshot test * use proper IoBuffer constructor to take handle ownership
* fix oplog snapshot deadlock (#233)Dan Engelbrecht2024-11-251-31/+57
| | | | * store inlined chunk as temp files and store to Cid after oplog iteration is complete and ops updated * make sure we can get to the payload when doing `prep` for new ops
* /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | * don't provide size if compressed buffer validation fails
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-8/+15
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-253-44/+110
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | * bail attachment import on corrupt attachment block * changelog
* fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-154-213/+864
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-062-7/+11
| | | * make sure we track added attachments properly in oplogs
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-064-68/+105
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-163-21/+21
| | | * make sure to initialize stats in file, jupiter and zen remote stores
* Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-141-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
* Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-115-35/+397
| | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
* remove gc v1 (#121)Dan Engelbrecht2024-10-032-291/+1
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc command attachment options (#176)Dan Engelbrecht2024-09-302-9/+8
| | | * zen command - add options to control meta data cache when triggering gc
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-6/+9
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* gc unused refactor (#165)Dan Engelbrecht2024-09-231-28/+21
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* End was clamped to the wrong side if Count was defaultedMartin Ridgers2024-09-191-1/+1
|
* Reinstate member initialisation that was there originallyMartin Ridgers2024-09-191-2/+2
|
* Poorly cherry-pickMartin Ridgers2024-09-191-2/+2
|
* Surfaced oplog paging to HTTP handlerMartin Ridgers2024-09-191-3/+18
|
* clang-format on branch's changed filesMartin Ridgers2024-09-192-106/+113
|
* Mandatory IterateOplog() paging argumentMartin Ridgers2024-09-193-16/+13
|
* Removed redundant initialisationMartin Ridgers2024-09-191-2/+2
|
* A clang-format passMartin Ridgers2024-09-191-3/+3
|
* Appease Apple Clang's historical lack of std::span constructorsMartin Ridgers2024-09-191-6/+2
|
* Forgot there was a std::clampMartin Ridgers2024-09-191-7/+4
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-21 06:50:51 +0000