aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/projectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-021-8098/+0
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-021-1610/+415
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* Zs/OIDC exe path handling (#538)Zousar Shaker2025-10-021-4/+10
| | | | | * Fix handling of oidc-exe-path The path was only being set if the file DIDN'T exist on the server (along with the warning about it not existing). Now set if it DOES exist, and the warning emitted if it DOESN'T.
* Fix bug in ReplayLogEntries for large op blobs (#537)Matt Peters2025-10-011-0/+1
| | | | | - Bugfix: Fixed issue where large ops in oplog would fail to read properly ReplayLogEntries: Fix bug in the case where an op is too big for OpBl…obsBuffer. The OpBufferView needs to be pointed to the temporary buffer.
* skip dropped oplogs in ScanForOplogs() (#534)Dan Engelbrecht2025-09-301-0/+5
|
* projectstore refactor (#531)Dan Engelbrecht2025-09-301-473/+287
| | | | | | | | | | | | | | | | | * convert ProjectStore::GetProjectFiles to not use http return codes * convert ProjectStore::GetProjectChunkInfos to not use http return codes * convert ProjectStore::GetChunkInfo to not use http return codes * convert ProjectStore::GetChunkRange to not use http return codes * convert ProjectStore::GetChunk to not use http return codes * convert ProjectStore::PutChunk to not use http return codes * convert ProjectStore::WriteOplog to not use http return codes * convert ProjectStore::ReadOplog to not use http return codes
* gracefully handle missing chunks when exporting an oplog (#526)Dan Engelbrecht2025-09-291-23/+47
| | | | - Improvement: Gracefully handle missing chunks when exporting an oplog - Bugfix: Add object validation when replaying oplog to avoid crash on malformed data
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-3/+2
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-227/+375
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* limit number of chunks per block (#508)Dan Engelbrecht2025-09-231-47/+66
| | | | | - Improvement: Chunk block generation for `zen oplog-export` and `zen builds upload` command are now limited to max 4000 chunks per block zen builds list-block testing function to check results for recently used blocks in a context
* dont set Size or RawSize when fetching chunkinfos if the payload can't be ↵Dan Engelbrecht2025-09-221-2/+7
| | | | found/validated (#507)
* Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-221-2/+2
| | | | command to use unattended mode when launching oidc-token.exe (#506)
* new in-memory storage strategy for oplogs (#490)Dan Engelbrecht2025-09-151-417/+598
| | | - Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-13/+16
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* faster oplog entries with referenceset (#488)Dan Engelbrecht2025-09-081-8/+2
| | | | - Improvement: Faster project store `/entries` endpoint, 10-15% faster when using a reference set to limit entries fix missing space after task name in pretty progressbar
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-65/+73
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-041-81/+178
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* projectstore lifetime improvements (#481)Dan Engelbrecht2025-08-291-317/+488
| | | | | - Improvement: Oplogs that have not been touched for 15 min are unloaded from memory during GC pass, oplogs are reloaded on demand - Improvement: Oplogs read for GC/Validation operations are loaded using a lightweight mode reducing memory usage and load times - Improvement: Cleaned up logging for oplogs/projects to reduce clutter
* revert multi-cid store (#475)Dan Engelbrecht2025-08-261-111/+58
|
* oplog output improvements (#474)Dan Engelbrecht2025-08-221-8/+12
| | | | | | * include session id in oplog import/export output * use Mbit/sec for transfer rates * fix elapsed time conversion in buildremotprojectstore * changelog
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-201-63/+121
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-3/+3
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-3/+5
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* list build part content (#462)Dan Engelbrecht2025-08-111-2/+2
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-67/+104
| | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
* missing chunks bugfix (#424)Dan Engelbrecht2025-06-091-1/+1
| | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
* pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-051-3/+6
| | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* faster oplog validate (#408)Dan Engelbrecht2025-05-301-9/+16
| | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-23/+17
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-121-54/+7
| | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
* optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-16/+8
| | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
* iterate chunks crash fix (#376)Dan Engelbrecht2025-05-021-1/+1
| | | * Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
* `zen oplog-export`, `zen oplog-import` for `--url` (cloud) and `--builds` ↵Dan Engelbrecht2025-04-031-0/+22
| | | | (builds) option now has `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token (#340)
* long filename support (#330)Dan Engelbrecht2025-03-311-53/+53
| | | - Bugfix: Long file paths now works correctly on Windows
* Moved AcceptType Handling From ProjectStore To HttpProjectStorezousar2025-03-251-58/+3
|
* Add CookPackageArtifacts attachment to web uizousar2025-03-211-4/+39
|
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-061-14/+31
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* builds upload command (#278)Dan Engelbrecht2025-02-261-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-8/+8
|
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-221-4/+4
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-161-1/+1
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* don't add overhead of verifying oplog presence on disk for "getchunks" rpc ↵Dan Engelbrecht2024-12-181-3/+5
| | | | call (#269)
* more memory tagging and fixes (#263)Stefan Boberg2024-12-161-0/+135
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Implemented temporary workaround for long paths in ComputeOpKeyStefan Boberg2024-12-131-3/+72
| | | | also added a test case to exercise the logic
* fixed XXH3_128Stream so it initializes the state properlyStefan Boberg2024-12-131-1/+1
| | | | the old version is still present for now, with a _deprecated suffix
* added ComputeOpKey so all instances of mapping key -> Oid is in a single placeStefan Boberg2024-12-131-20/+19
|
* Builds API remote project store (#258)Dan Engelbrecht2024-12-121-3/+71
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* reduce oplog job name (#261)Dan Engelbrecht2024-12-111-2/+2
| | | * reduce length of oplog import/export job name
* global open process cache for projectstore (#257)Dan Engelbrecht2024-12-051-13/+27
| | | | * move openprocess cache to central location * enable openprocesscache in projectstore so "getchunks" can send filehandles when requested
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-051-193/+1738
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 05:49:57 +0000