aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore
Commit message (Collapse)AuthorAgeFilesLines
* `zen oplog-export`, `zen oplog-import` for `--url` (cloud) and `--builds` ↵Dan Engelbrecht2025-04-035-24/+64
| | | | (builds) option now has `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token (#340)
* added --find-max-block-count option to builds upload (#337)Dan Engelbrecht2025-04-021-1/+1
|
* builds url discovery (#334)Dan Engelbrecht2025-04-012-5/+28
| | | | | - Feature: Added `--host` option to use Jupiters list of cloud host and zen servers to resolve best hosts - Feature: Use local zenserver as builds cache if it has the `builds` service enabled and `--cloud-discovery-host` is provided and no remote zenserver cache hosts can be found - Improvement: Added `--override-host` option as a replacement for `--url` (`--url` still works, but `--override-host` is preferred)
* long filename support (#330)Dan Engelbrecht2025-03-313-61/+61
| | | - Bugfix: Long file paths now works correctly on Windows
* Descriptive type conversion messageszousar2025-03-261-15/+39
| | | | Handling decompression or validation errors with more descriptive messages
* Moved AcceptType Handling From ProjectStore To HttpProjectStorezousar2025-03-253-61/+35
|
* Add CookPackageArtifacts attachment to web uizousar2025-03-211-4/+39
|
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-062-28/+49
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* builds upload command (#278)Dan Engelbrecht2025-02-266-43/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-252-67/+19
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* move WriteToTempFile to basicfile.h (#283)Dan Engelbrecht2025-02-121-60/+40
| | | add helper constructors to BasicFile
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-127-240/+122
|
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-221-4/+4
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-223-109/+95
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* export oplog to zen target url (#274)Dan Engelbrecht2025-01-161-1/+1
| | | | * fix oplog target url for oplog export to remote zenserver * verify that chunking is allowed before chunking loose files
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-164-4/+4
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-1/+4
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* don't add overhead of verifying oplog presence on disk for "getchunks" rpc ↵Dan Engelbrecht2024-12-181-3/+5
| | | | call (#269)
* remove all referenced attachments in op from pending chunk references (#267)Dan Engelbrecht2024-12-171-5/+7
|
* more memory tagging and fixes (#263)Stefan Boberg2024-12-162-2/+139
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Implemented temporary workaround for long paths in ComputeOpKeyStefan Boberg2024-12-131-3/+72
| | | | also added a test case to exercise the logic
* fixed XXH3_128Stream so it initializes the state properlyStefan Boberg2024-12-131-1/+1
| | | | the old version is still present for now, with a _deprecated suffix
* added ComputeOpKey so all instances of mapping key -> Oid is in a single placeStefan Boberg2024-12-131-20/+19
|
* Builds API remote project store (#258)Dan Engelbrecht2024-12-128-219/+971
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* reduce oplog job name (#261)Dan Engelbrecht2024-12-111-2/+2
| | | * reduce length of oplog import/export job name
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-1241/+1450
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* global open process cache for projectstore (#257)Dan Engelbrecht2024-12-052-14/+35
| | | | * move openprocess cache to central location * enable openprocesscache in projectstore so "getchunks" can send filehandles when requested
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-054-215/+1778
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-022-12/+12
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+11
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-282-334/+428
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix oplog index path reading error (#246)Dan Engelbrecht2024-11-281-30/+41
| | | * when reading paths for oplog index, make sure we don't point our string view to potentially stale memory
* set content type correctly for getchunkrange (#241)Dan Engelbrecht2024-11-281-3/+3
|
* skip rawsize/size in getchunksinfo if chunk is missing (#238)Dan Engelbrecht2024-11-261-2/+2
|
* snapshot crash (#234)Dan Engelbrecht2024-11-251-1/+6
| | | | * snapshot test * use proper IoBuffer constructor to take handle ownership
* fix oplog snapshot deadlock (#233)Dan Engelbrecht2024-11-251-31/+57
| | | | * store inlined chunk as temp files and store to Cid after oplog iteration is complete and ops updated * make sure we can get to the payload when doing `prep` for new ops
* /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | * don't provide size if compressed buffer validation fails
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-8/+15
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-253-44/+110
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | * bail attachment import on corrupt attachment block * changelog
* fix oplog chunk batch get (#221)Dan Engelbrecht2024-11-181-5/+7
| | | | * fix batch request not handling missing chunks correctly * fix CorrelationId in oplog batch chunk fetch
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-154-213/+864
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-062-7/+11
| | | * make sure we track added attachments properly in oplogs
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-064-68/+105
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-163-21/+21
| | | * make sure to initialize stats in file, jupiter and zen remote stores
* Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-141-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
* Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-115-35/+397
| | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
* remove gc v1 (#121)Dan Engelbrecht2024-10-032-291/+1
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc command attachment options (#176)Dan Engelbrecht2024-09-302-9/+8
| | | * zen command - add options to control meta data cache when triggering gc
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-6/+9
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-27 08:09:46 +0000