seaweedfs/weed/server
Chris Lu c284e51d20
fix: multipart upload ETag calculation (#8238)
* fix multipart etag

* address comments

* clean up

* clean up

* optimization

* address comments

* unquoted etag

* dedup

* upgrade

* clean

* etag

* return quoted tag

* quoted etag

* debug

* s3api: unify ETag retrieval and quoting across handlers

Refactor newListEntry to take *S3ApiServer and use getObjectETag,
and update setResponseHeaders to use the same logic. This ensures
consistent ETags are returned for both listing and direct access.

* s3api: implement ListObjects deduplication for versioned buckets

Handle duplicate entries between the main path and the .versions
directory by prioritizing the latest version when bucket versioning
is enabled.

* s3api: cleanup stale main file entries during versioned uploads

Add explicit deletion of pre-existing "main" files when creating new
versions in versioned buckets. This prevents stale entries from
appearing in bucket listings and ensures consistency.

* s3api: fix cleanup code placement in versioned uploads

Correct the placement of rm calls in completeMultipartUpload and
putVersionedObject to ensure stale main files are properly deleted
during versioned uploads.

* s3api: improve getObjectETag fallback for empty ExtETagKey

Ensure that when ExtETagKey exists but contains an empty value,
the function falls through to MD5/chunk-based calculation instead
of returning an empty string.

* s3api: fix test files for new newListEntry signature

Update test files to use the new newListEntry signature where the
first parameter is *S3ApiServer. Created mockS3ApiServer to properly
test owner display name lookup functionality.

* s3api: use filer.ETag for consistent Md5 handling in getEtagFromEntry

Change getEtagFromEntry fallback to use filer.ETag(entry) instead of
filer.ETagChunks to ensure legacy entries with Attributes.Md5 are
handled consistently with the rest of the codebase.

* s3api: optimize list logic and fix conditional header logging

- Hoist bucket versioning check out of per-entry callback to avoid
  repeated getVersioningState calls
- Extract appendOrDedup helper function to eliminate duplicate
  dedup/append logic across multiple code paths
- Change If-Match mismatch logging from glog.Errorf to glog.V(3).Infof
  and remove DEBUG prefix for consistency

* s3api: fix test mock to properly initialize IAM accounts

Fixed nil pointer dereference in TestNewListEntryOwnerDisplayName by
directly initializing the IdentityAccessManagement.accounts map in the
test setup. This ensures newListEntry can properly look up account
display names without panicking.

* cleanup

* s3api: remove premature main file cleanup in versioned uploads

Removed incorrect cleanup logic that was deleting main files during
versioned uploads. This was causing test failures because it deleted
objects that should have been preserved as null versions when
versioning was first enabled. The deduplication logic in listing is
sufficient to handle duplicate entries without deleting files during
upload.

* s3api: add empty-value guard to getEtagFromEntry

Added the same empty-value guard used in getObjectETag to prevent
returning quoted empty strings. When ExtETagKey exists but is empty,
the function now falls through to filer.ETag calculation instead of
returning "".

* s3api: fix listing of directory key objects with matching prefix

Revert prefix handling logic to use strings.TrimPrefix instead of
checking HasPrefix with empty string result. This ensures that when a
directory key object exactly matches the prefix (e.g. prefix="dir/",
object="dir/"), it is correctly handled as a regular entry instead of
being skipped or incorrectly processed as a common prefix. Also fixed
missing variable definition.

* s3api: refactor list inline dedup to use appendOrDedup helper

Refactored the inline deduplication logic in listFilerEntries to use the
shared appendOrDedup helper function. This ensures consistent behavior
and reduces code duplication.

* test: fix port allocation race in s3tables integration test

Updated startMiniCluster to find all required ports simultaneously using
findAvailablePorts instead of sequentially. This prevents race conditions
where the OS reallocates a port that was just released, causing multiple
services (e.g. Filer and Volume) to be assigned the same port and fail
to start.
2026-02-06 21:54:43 -08:00
..
constants Nit: use time.Durations instead of constants in seconds. (#7438) 2025-11-04 13:02:22 -08:00
filer_ui chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
master_ui Fix/parse upload filename (#6241) 2024-11-15 11:25:18 -08:00
postgres Message Queue: Add sql querying (#7185) 2025-09-09 01:01:03 -07:00
volume_server_ui fix: EC UI template error when viewing shard details (#7955) 2026-01-03 22:45:48 -08:00
common_test.go jwt check the base file id 2019-03-03 10:17:44 -08:00
common.go fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635) 2025-12-05 15:59:12 -08:00
filer_grpc_server_admin.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_grpc_server_dlm.go reduce logs 2025-12-26 13:26:25 -08:00
filer_grpc_server_kv.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_grpc_server_remote.go s3: fix remote object not caching (#7790) 2025-12-16 12:41:04 -08:00
filer_grpc_server_rename.go Prevent bucket renaming in filer, fuse mount, and S3 (#8048) 2026-01-16 19:48:09 -08:00
filer_grpc_server_sub_meta.go avoid repeated reading disk (#7369) 2025-10-24 01:26:42 -07:00
filer_grpc_server_traverse_meta_test.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_grpc_server_traverse_meta.go Add error list each entry func (#7485) 2025-11-25 19:35:19 -08:00
filer_grpc_server.go filer: preserve existing TTL during CreateEntry/UpdateEntry gRPC calls 2026-01-29 14:42:09 -08:00
filer_jwt_test.go fix Filer startup failure due to JWT on / path #8149 (#8167) 2026-01-29 21:45:15 -08:00
filer_server_handlers_copy.go use one http client 2025-08-11 16:45:13 -07:00
filer_server_handlers_iam_grpc.go Implement IAM propagation to S3 servers (#8130) 2026-01-26 22:59:43 -08:00
filer_server_handlers_proxy.go Changes logging function (#6919) 2025-06-24 08:44:06 -07:00
filer_server_handlers_read_dir.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_server_handlers_read.go S3: set identity to request context, and remove obsolete code (#7523) 2025-11-21 14:46:32 -08:00
filer_server_handlers_tagging.go Changes logging function (#6919) 2025-06-24 08:44:06 -07:00
filer_server_handlers_write_autochunk.go filer write request use context without cancellation (#7567) 2025-11-28 11:52:57 -08:00
filer_server_handlers_write_cipher.go convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
filer_server_handlers_write_merge.go S3 API: Add SSE-KMS (#7144) 2025-08-21 08:28:07 -07:00
filer_server_handlers_write_upload.go fix: multipart upload ETag calculation (#8238) 2026-02-06 21:54:43 -08:00
filer_server_handlers_write.go Reduce memory allocations in hot paths (#7725) 2025-12-12 12:51:48 -08:00
filer_server_handlers.go fix Filer startup failure due to JWT on / path #8149 (#8167) 2026-01-29 21:45:15 -08:00
filer_server_rocksdb.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
filer_server_tus_handlers.go Add TUS protocol support for resumable uploads (#7592) 2025-12-14 21:56:07 -08:00
filer_server_tus_session.go Add TUS protocol support for resumable uploads (#7592) 2025-12-14 21:56:07 -08:00
filer_server.go Fix: filer not yet available in s3.configure (#8198) 2026-02-03 17:43:58 -08:00
master_grpc_server_admin.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
master_grpc_server_assign.go Fix Maintenance Task Sorting and Refactor Log Persistence (#8199) 2026-02-04 08:48:55 -08:00
master_grpc_server_cluster.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
master_grpc_server_collection.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
master_grpc_server_raft_test.go Add cluster.raft.leader.transfer command for graceful leader change (#7819) 2025-12-19 00:15:39 -08:00
master_grpc_server_raft.go fix: admin does not show all master servers #7999 (#8002) 2026-01-11 12:31:46 -08:00
master_grpc_server_volume.go Fix imbalance detection disk type grouping and volume grow errors (#8097) 2026-01-23 12:25:11 -08:00
master_grpc_server.go Have masters update DataNode details based on state heartbeats from volume servers. (#8017) 2026-01-29 21:51:46 -08:00
master_server_handlers_admin.go follow grow volume option version 2025-06-19 13:54:54 -07:00
master_server_handlers_ui.go hide millseconds in up time (#7553) 2025-11-26 08:01:19 -08:00
master_server_handlers.go add disable volume_growth flag (#7196) 2025-09-04 05:39:56 -07:00
master_server.go Prevent split-brain: Persistent ClusterID and Join Validation (#8022) 2026-01-18 14:02:34 -08:00
raft_common.go Prevent split-brain: Persistent ClusterID and Join Validation (#8022) 2026-01-18 14:02:34 -08:00
raft_hashicorp.go Prevent split-brain: Persistent ClusterID and Join Validation (#8022) 2026-01-18 14:02:34 -08:00
raft_server_handlers.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
raft_server.go Prevent split-brain: Persistent ClusterID and Join Validation (#8022) 2026-01-18 14:02:34 -08:00
volume_grpc_admin.go Add a version token on RPCs to read/update volume server states. (#8191) 2026-02-06 10:58:43 -08:00
volume_grpc_batch_delete.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_client_to_master.go Boostrap persistent state for volume servers. (#7984) 2026-01-12 10:49:59 -08:00
volume_grpc_copy_incremental.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
volume_grpc_copy.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_erasure_coding.go Implement RPC skeleton for regular/EC volumes scrubbing. (#8187) 2026-02-02 17:55:04 -08:00
volume_grpc_query.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
volume_grpc_read_all.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
volume_grpc_read_write.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_remote.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_scrub.go Contrib/volume scrub local (#8226) 2026-02-05 14:44:12 -08:00
volume_grpc_state.go Add a version token on RPCs to read/update volume server states. (#8191) 2026-02-06 10:58:43 -08:00
volume_grpc_tail.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_tier_download.go avoid load volume file with BytesOffset mismatch (#3841) 2022-10-14 00:18:09 -07:00
volume_grpc_tier_upload.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_vacuum.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_server_handlers_admin.go fix: volume server healthz now checks local conditions only (#7610) 2025-12-02 23:19:14 -08:00
volume_server_handlers_helper.go directory structure change to work with glide 2016-06-02 18:09:14 -07:00
volume_server_handlers_read.go fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635) 2025-12-05 15:59:12 -08:00
volume_server_handlers_ui.go hide millseconds in up time (#7553) 2025-11-26 08:01:19 -08:00
volume_server_handlers_write.go fix: multipart upload ETag calculation (#8238) 2026-02-06 21:54:43 -08:00
volume_server_handlers.go fix: JWT validation failures during replication (#7788) (#7795) 2025-12-16 13:42:18 -08:00
volume_server_test.go Add a version token on RPCs to read/update volume server states. (#8191) 2026-02-06 10:58:43 -08:00
volume_server.go Add a version token on RPCs to read/update volume server states. (#8191) 2026-02-06 10:58:43 -08:00
webdav_server.go mount: improve read throughput with parallel chunk fetching (#7569) 2025-11-29 10:06:11 -08:00
wrapped_webdav_fs.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00