aboutsummaryrefslogtreecommitdiff
path: root/src/zencore/filesystem.cpp
Commit message (Collapse)AuthorAgeFilesLines
* speed up tests (#555)Dan Engelbrecht2025-10-061-3/+10
| | | | | | | | | | | | * faster FileSystemTraversal test * faster jobqueue test * faster NamedEvent test * faster cache tests * faster basic http tests * faster blockstore test * faster cache store tests * faster compactcas tests * more responsive zenserver launch * tweak worker pool sizes in tests
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-0/+21
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-20/+22
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* Merge pull request #452 from ue-foundation/lm/full-service-installLiam Mitchell2025-08-061-0/+2
|\ | | | | Add --full option to service install, which will handle stop/uninstall if necessary, and copy binaries to install location
| * Fix permissions and ownership issues with service binary copy and remove ↵Liam Mitchell2025-07-251-0/+2
| | | | | | | | unnecessary alias from unit file
* | pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-051-0/+230
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* | fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-031-7/+16
| | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
* | http client streaming upload (#413)Dan Engelbrecht2025-06-021-7/+119
| | | | | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* | parallel work handle dispatch exception (#400)Dan Engelbrecht2025-05-161-0/+5
| | | | | | - Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception
* | sparse win file write (#398)Dan Engelbrecht2025-05-161-0/+28
| | | | | | * Added `--use-sparse-files` option to `zen builds` command improving write performance of large files. Enabled by default.
* | add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-221-16/+0
| |
* | use FindClose (#357)Dan Engelbrecht2025-04-111-1/+1
| | | | | | * use FindClose
* | filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-34/+67
| | | | | | | | | | | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* | multithreaded clean (#331)Dan Engelbrecht2025-03-311-4/+51
| | | | | | | | - Improvement: Faster cleaning of directories - Improvement: Faster initial scanning of local state
* | long filename support (#330)Dan Engelbrecht2025-03-311-127/+748
| | | | | | - Bugfix: Long file paths now works correctly on Windows
* | optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-3/+6
|/ | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* fix quoted command lines arguments (#306)Dan Engelbrecht2025-03-121-0/+13
| | | Handling of quotes and quotes with leading backslash for command line parsing - UE-231677
* fix mac/linux builds command (#303)Dan Engelbrecht2025-03-121-15/+20
| | | * fix linux/mac version of GetModificationTickFromPath and CopyFile
* builds download incremental (#290)Dan Engelbrecht2025-03-011-2/+5
| | | | | * incremental download * merge rebuild state and output state building * fix writing when > 1 zero size file
* builds upload command (#278)Dan Engelbrecht2025-02-261-0/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-6/+20
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-221-19/+145
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-051-1/+23
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-251-0/+1
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* fix inconsistencies in filecas due to failing to remove payload file during ↵Dan Engelbrecht2024-11-221-2/+2
| | | | | | | | GC (#224) make sure we rewrite filecas entries if chunk size changes (due to compression changes) hardening of move/write files in filecas if we encounter a filecas entry with mismatching size (due to pre-existing bug) we validate the file and update the index if we find a bad filecas file on disk we now attempt to remove it
* workspace share security (#192)Dan Engelbrecht2024-10-231-0/+30
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* safer path from handle (#195)Dan Engelbrecht2024-10-161-15/+10
| | | * remove PathFromHandle that throws to give better context on failures
* simplified CleanDirectory implementation (#182)Stefan Boberg2024-10-031-26/+8
| | | we use the std implementation for all normal cases now. The Windows-only path still exists for VFS cleanup
* handle "path not found" as well as "file not found" in directory travers on ↵Dan Engelbrecht2024-08-201-1/+1
| | | | windows (#129)
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-261-1/+41
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* improved assert (#37)Dan Engelbrecht2024-04-041-1/+1
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* fix potential partially written files (#2)Dan Engelbrecht2024-03-131-1/+9
| | | | * Make sure WriteFile() does not leave incomplete files * use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error
* Make sure that PathFromHandle don't hide true error when throwing exceptions ↵Dan Engelbrecht2023-12-111-6/+6
| | | | | | | | | (#600) * Make sure that PathFromHandle don't hide true error when throwing exceptions * changelog * return error info in PathFromHandle if path fails to resolve
* optimized index snapshot reading/writing (#561)Stefan Boberg2023-11-271-2/+23
| | | | | the previous implementation of in-memory index snapshots serialise data to memory before writing to disk and vice versa when reading. This leads to some memory spikes which end up pushing useful data out of system cache and also cause stalls on I/O operations. this change moves more code to a streaming serialisation approach which scales better from a memory usage perspective and also performs much better
* fix block cloning copy argument validation (#560)Stefan Boberg2023-11-221-0/+30
|
* zen run command (#552)Stefan Boberg2023-11-211-0/+119
| | | | | | | | | | initial version -- this is primarily intended to be used for running stress tests and/or benchmarks example usage: `zen run -n 10 -- zenserver-test` `zen run -n 10 -- zenserver-test --ts=core.assert` run zenserver-test 10 times (testing only the `core.assert` test suite) `zen run --time 600 --basepath=d:\test_dir\test1 -- zenserver-test` keeps spawning new instances for 10 minutes (600 seconds)
* moved process handling code into separate h/cpp (#555)Stefan Boberg2023-11-201-0/+1
|
* add zenserver state snapshot support (#543)Stefan Boberg2023-11-161-0/+97
| | | | | this introduces a --snapshot-dir command line option to zenserver which specifies a directory which will be propagated to the persistence root directory on start-up. This is most powerful with file systems which support block cloning, such as ReFS on Windows. This allows even very large state snapshots to be used repeatedly without having to worry about mutating the original dataset on disk. When using ReFS the state copy for even large state directories can be very fast since the duration is primarily proportional to the number of files in the tree rather than the size of the files being cloned. The storage requirements are also minimal as all data will be handled in a copy-on-write manner.
* gc history log (#519)Dan Engelbrecht2023-11-131-0/+64
| | | | | - Feature: Writes a `gc.log` with settings and detailed result after each GC execution (version 2 only) - Break out file name rotate to allow access for gclog - CompactBinaryToJson(MemoryView Data, StringBuilderBase& InBuilder)
* If a directory is deleted while we try to traverse it, skip it and continue ↵Dan Engelbrecht2023-11-101-0/+11
| | | | (#528)
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-201-8/+67
| | | currently, only Windows (using Projected File System) is supported
* Make sure error logging or destructors don't throw exception when trying to ↵Dan Engelbrecht2023-09-121-13/+44
| | | | | get file name from handle (#393) - Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle
* Extend http client (#387)Dan Engelbrecht2023-09-081-0/+63
| | | * extend http client with configuration, headers, parameters and disk streaming upload/download
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-1/+1
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* lua config file improvements (#379)Dan Engelbrecht2023-08-311-0/+4
| | | | | | - Bugfix: All options given on command line now overrides lua config file settings - Improvement: All options available from command line can now be configured in the lua config file (with a few exceptions such as `owner-pid`, `install` and `uninstall`) We can now also generate a Lua config (just string so far) that includes all options given on command line together with any optional Lua input config file. The idea here is to be able to save config options in the future.
* Run clang tidy on modified filezousar2023-08-181-4/+4
|
* Reduce log level for RLIMIT messagezousar2023-08-181-1/+1
|
* * Added Guid::FromStringStefan Boberg2023-06-301-8/+61
| | | | | | | * Added LoadCompactBinaryObject from file to compactbinaryfile.cpp/h * Added SaveCompactBinary(BinaryWriter& Ar, ...) functions * Added ZEN_PLATFORM_NAME define * Added SystemMetrics functionality to query system properties (see zencore/system.h)
* Make sure GetEnvVariable can handle values that are longer than 1023 ↵Dan Engelbrecht2023-06-131-4/+18
| | | | characters (#332)
* Remove ATL header usage (#306)Stefan Boberg2023-05-151-30/+30
| | | ATL has been used here and there as a convenience. Given that this is a legacy component and not always something which gets installed along with the compiler we hereby remove the dependency altogether in favour of our own simple wrappers
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 07:17:59 +0000