aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/httpprojectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-141-3307/+0
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-061-2/+2
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-3/+21
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-031-8/+5
| | | * move remoteproject code to remotestorelib
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-021-2/+2
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-021-20/+875
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* projectstore refactor (#531)Dan Engelbrecht2025-09-301-172/+310
| | | | | | | | | | | | | | | | | * convert ProjectStore::GetProjectFiles to not use http return codes * convert ProjectStore::GetProjectChunkInfos to not use http return codes * convert ProjectStore::GetChunkInfo to not use http return codes * convert ProjectStore::GetChunkRange to not use http return codes * convert ProjectStore::GetChunk to not use http return codes * convert ProjectStore::PutChunk to not use http return codes * convert ProjectStore::WriteOplog to not use http return codes * convert ProjectStore::ReadOplog to not use http return codes
* new in-memory storage strategy for oplogs (#490)Dan Engelbrecht2025-09-151-48/+52
| | | - Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
* faster oplog entries with referenceset (#488)Dan Engelbrecht2025-09-081-23/+48
| | | | - Improvement: Faster project store `/entries` endpoint, 10-15% faster when using a reference set to limit entries fix missing space after task name in pretty progressbar
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-87/+117
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-041-3/+1
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* projectstore lifetime improvements (#481)Dan Engelbrecht2025-08-291-19/+19
| | | | | - Improvement: Oplogs that have not been touched for 15 min are unloaded from memory during GC pass, oplogs are reloaded on demand - Improvement: Oplogs read for GC/Validation operations are loaded using a lightweight mode reducing memory usage and load times - Improvement: Cleaned up logging for oplogs/projects to reduce clutter
* revert multi-cid store (#475)Dan Engelbrecht2025-08-261-134/+33
|
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-201-33/+134
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* builds url discovery (#334)Dan Engelbrecht2025-04-011-3/+19
| | | | | - Feature: Added `--host` option to use Jupiters list of cloud host and zen servers to resolve best hosts - Feature: Use local zenserver as builds cache if it has the `builds` service enabled and `--cloud-discovery-host` is provided and no remote zenserver cache hosts can be found - Improvement: Added `--override-host` option as a replacement for `--url` (`--url` still works, but `--override-host` is preferred)
* Descriptive type conversion messageszousar2025-03-261-15/+39
| | | | Handling decompression or validation errors with more descriptive messages
* Moved AcceptType Handling From ProjectStore To HttpProjectStorezousar2025-03-251-2/+32
|
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-061-14/+18
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* remove all referenced attachments in op from pending chunk references (#267)Dan Engelbrecht2024-12-171-5/+7
|
* more memory tagging and fixes (#263)Stefan Boberg2024-12-161-2/+4
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-051-6/+6
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-021-5/+5
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+11
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-251-2/+3
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-151-16/+141
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-061-0/+1
| | | * make sure we track added attachments properly in oplogs
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-061-11/+11
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-111-27/+48
| | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
* Poorly cherry-pickMartin Ridgers2024-09-191-2/+2
|
* Surfaced oplog paging to HTTP handlerMartin Ridgers2024-09-191-3/+18
|
* clang-format on branch's changed filesMartin Ridgers2024-09-191-1/+2
|
* Mandatory IterateOplog() paging argumentMartin Ridgers2024-09-191-2/+2
|
* add compacting of oplogs as part of GC (#106)Dan Engelbrecht2024-08-121-12/+12
| | | | * add compacting of oplogs as part of GC * force retain of LSN unless we have less than 16 miln entries left
* make oplog lsn unsigned (#107)Dan Engelbrecht2024-08-121-14/+15
| | | * change oplog lsn to uint32
* project/oplog delete improvements (#105)Dan Engelbrecht2024-08-121-4/+11
| | | | | * make oplog/project folder removeal more robust * report back error to http caller if removal fails
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-261-5/+0
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* add "fieldnames" query param for GetProjectFiles/GetProjectChunkInfos (#29)Dan Engelbrecht2024-03-281-3/+45
| | | | | | | | - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/chunkinfos` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `rawhash` and `rawsize` (translates to `?fieldnames=id,rawhash,rawsize`) - Use `?fieldnames=*` to get all the fields - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/files` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `clientpath` and `serverpath` (translates to `?fieldnames=id,clientpath,serverpath`), `filter=client` only applies if `fieldnames` is not given as a parameter - Use `?fieldnames=*` to get all the fields
* add filter to projectstore entries request (#25)Dan Engelbrecht2024-03-261-6/+65
| | | | | | | * Add HttpServerRequest::Decode to decode http request parameters * Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited * Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header
* consistent paths encoding (#24)Dan Engelbrecht2024-03-251-14/+16
| | | | * Don't encode filesystem path to UTF8 unless stored in compactbinary string * Be consistent where we encode/decode paths to UTF8
* non memory copy compressed range (#13)Dan Engelbrecht2024-03-201-4/+5
| | | | | * Add CompressedBuffer::GetRange that references source data rather than make a memory copy * Use Compressed.CopyRange in project store GetChunkRange * docs for CompressedBuffer::CopyRange and CompressedBuffer::GetRange
* Change naming to ChunkInfos instead of Chunkszousar2023-12-061-5/+5
|
* Add endpoint for all chunk infoszousar2023-12-011-0/+40
| | | | Add endpoint for querying all chunk infos in an oplog.
* faster oplog iteration (#471)Dan Engelbrecht2023-10-131-14/+13
| | | | | * use a CbObjectView instead of CbObject to avoid creating IOBufferCore instances * use BasicFileBuffer directly where possible * changelog
* adding more stats (#429)Dan Engelbrecht2023-09-281-0/+6
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* More statistics for Cache, Project Store and Cid Store (#405)Dan Engelbrecht2023-09-141-0/+3
| | | | | Cache: requestcount, badrequestcount, writes Project Store: requestcount Cid Store: cidhits, cidmisses, cidwrites
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-0/+1
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* project store gc deadlock (#381)Dan Engelbrecht2023-08-311-2/+2
| | | | * Trying to recursively take a shared RWLock while another thread is waiting for a exclusive lock results in a deadlock. Forward the shared lock so we don't have to grab it recursively. * changelog
* add helper functions in HttpProjectService to help with readability (#378)Dan Engelbrecht2023-08-291-1195/+1306
|
* add update/delete endpoint for project and oplog (#353)Dan Engelbrecht2023-08-181-2/+89
| | | | | | | | | | | | | * add update endpoint for project store project * add update endpoint for oplog * changelog * Zen command line tool `project-update` Zen command line tool `project-delete` Zen command line tool `oplog-update` Zen command line tool `oplog-delete` * add --force-update option to project/oplog create remove project/oplog update commnad
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 12:57:43 +0000