gian/seaweedfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
fix/execution-plan-disk-id
seaweedfs/weed/server
History
Chris Lu 68794fb94c
fix(ec_distribute): remove partial files on copy stream error (#9543) 
* fix(ec_distribute): remove partial files on copy stream error

writeToFile opens the destination with O_TRUNC and streams into it. On
a mid-stream receive / write / cancellation error it returned the
failure but left the destination behind in whatever state had been
written so far — typically 0 bytes when the source errored before
sending any FileContent. VolumeEcShardsCopy distributes .ecx by
calling doCopyFile, so this same stub-leaving behaviour produced the
0-byte .ecx files seen on EC encoding failures: the source claims a
non-zero ModifiedTsNs (so the existing "source not found" cleanup
doesn't fire), the stream then errors immediately, and the receiver
ends up with a 0-byte .ecx that downstream code mistook for a valid
empty index.

Clean up the partial file on every error path that returns from the
streaming loop (receive, write, and cancellation). Skip cleanup when
isAppend=true so resumable appends keep their existing content. As
defense in depth, VolumeEcShardsCopy also stats the .ecx after copy
and removes / errors on a 0-byte result so the orchestrator can pick
a different source.

The Rust volume server has only the source side of CopyFile (no
client-side stream-to-disk consumer) and no .ecx subsystem yet, so
this fix has no Rust mirror.

* fix(ec_distribute): close file before remove, fail fast on stat error

Address review feedback:

- writeToFile's mid-stream removeIncomplete called os.Remove while the
  destination file handle was still open. On Windows os.Remove fails
  while a handle is open, so the cleanup wouldn't run there. Wrap the
  handle close in a once-only helper, call it from removeIncomplete
  and from the existing "source not found" cleanup, and keep a deferred
  close as the safety net for the normal-return path.
- VolumeEcShardsCopy's post-copy .ecx check silently passed when
  os.Stat returned an error: doCopyFile had reported success but if
  the file was already gone, unreadable, or somehow a directory, the
  orchestrator only learned at mount time with no useful context.
  Treat any non-nil stat error and any directory result as a copy
  failure here and surface it immediately.
2026-05-18 15:19:51 -07:00
..
constants Nit: use time.Durations instead of constants in seconds. (#7438) 2025-11-04 13:02:22 -08:00
filer_ui Fix UI prefix url encoding (#9344) 2026-05-06 19:14:36 -07:00
master_ui feat: improve aio support for admin/volume ingress and fix UI links (#8679) 2026-03-18 13:20:55 -07:00
nfs fix(tests): make 32-bit GOARCH tests build and run (#9507) 2026-05-14 20:55:37 -07:00
postgres chore: remove ~50k lines of unreachable dead code (#8913) 2026-04-03 16:04:27 -07:00
volume_server_ui fix: EC UI template error when viewing shard details (#7955) 2026-01-03 22:45:48 -08:00
common_test.go jwt check the base file id 2019-03-03 10:17:44 -08:00
common.go fix(filer): return 503 + Retry-After when remote object not cached yet (#9236) 2026-04-27 01:58:33 -07:00
filer_grpc_server_admin.go cluster: restrict Ping RPC to known peers of the requested type (#9445) 2026-05-12 13:00:52 -07:00
filer_grpc_server_dlm_test.go dlm: resilient distributed locks via consistent hashing + backup replication (#8860) 2026-03-30 23:29:56 -07:00
filer_grpc_server_dlm.go dlm: resilient distributed locks via consistent hashing + backup replication (#8860) 2026-03-30 23:29:56 -07:00
filer_grpc_server_kv.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_grpc_server_mount_peer_test.go chore(filer): remove -mount.p2p flag; registry is always on (#9183) 2026-04-21 23:00:11 -07:00
filer_grpc_server_mount_peer.go chore(filer): remove -mount.p2p flag; registry is always on (#9183) 2026-04-21 23:00:11 -07:00
filer_grpc_server_remote.go fix(filer/remote): keep re-cache work alive past caller cancellation (#9174) (#9193) 2026-04-22 17:56:15 -07:00
filer_grpc_server_rename_test.go Adjust rename events metadata format (#8854) 2026-03-30 18:25:11 -07:00
filer_grpc_server_rename.go [nfs] Add NFS (#9067) 2026-04-14 20:48:24 -07:00
filer_grpc_server_stream_mutate_bench_test.go perf(filer): parallelize StreamMutateEntry with path-keyed scheduler (#9171) 2026-04-21 11:25:09 -07:00
filer_grpc_server_stream_mutate_scheduler_test.go perf(filer): parallelize StreamMutateEntry with path-keyed scheduler (#9171) 2026-04-21 11:25:09 -07:00
filer_grpc_server_stream_mutate_scheduler.go perf(filer): parallelize StreamMutateEntry with path-keyed scheduler (#9171) 2026-04-21 11:25:09 -07:00
filer_grpc_server_stream_mutate.go perf(filer): parallelize StreamMutateEntry with path-keyed scheduler (#9171) 2026-04-21 11:25:09 -07:00
filer_grpc_server_sub_meta_test.go fix(tests): make tests pass on 32-bit architectures (#9168) (#9170) 2026-04-20 22:48:01 -07:00
filer_grpc_server_sub_meta.go fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9039) 2026-04-11 23:12:54 -07:00
filer_grpc_server_test.go fix(filer): apply default disk type after location-prefix resolution in gRPC AssignVolume (#8836) 2026-03-29 14:18:24 -07:00
filer_grpc_server_traverse_meta_test.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_grpc_server_traverse_meta.go Add error list each entry func (#7485) 2025-11-25 19:35:19 -08:00
filer_grpc_server.go feat: pass expected_data_size from clients for size-aware assignment (#9032) 2026-04-11 11:30:47 -07:00
filer_jwt_test.go filer: scope JWT allowed_prefixes to path components (#9439) 2026-05-12 10:10:48 -07:00
filer_server_handlers_copy_test.go Use filer-side copy for mounted whole-file copy_file_range (#8747) 2026-03-23 18:35:15 -07:00
filer_server_handlers_copy.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
filer_server_handlers_iam_grpc_test.go filer/iam-grpc: make admin Bearer auth opt-in (fixes #9509) (#9514) 2026-05-15 13:15:20 -07:00
filer_server_handlers_iam_grpc.go filer/iam-grpc: make admin Bearer auth opt-in (fixes #9509) (#9514) 2026-05-15 13:15:20 -07:00
filer_server_handlers_proxy_test.go fix(filer): limit concurrent proxy reads per volume server (#8608) 2026-03-11 23:32:09 -07:00
filer_server_handlers_proxy.go chore: remove ~50k lines of unreachable dead code (#8913) 2026-04-03 16:04:27 -07:00
filer_server_handlers_read_dir.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
filer_server_handlers_read.go fix(filer): return 503 + Retry-After when remote object not cached yet (#9236) 2026-04-27 01:58:33 -07:00
filer_server_handlers_tagging.go Changes logging function (#6919) 2025-06-24 08:44:06 -07:00
filer_server_handlers_write_autochunk.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
filer_server_handlers_write_merge.go S3 API: Add SSE-KMS (#7144) 2025-08-21 08:28:07 -07:00
filer_server_handlers_write_upload.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
filer_server_handlers_write.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
filer_server_handlers.go filer: scope JWT allowed_prefixes to path components (#9439) 2026-05-12 10:10:48 -07:00
filer_server_rocksdb.go go fix 2026-02-20 18:42:00 -08:00
filer_server_tus_handlers.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
filer_server_tus_session.go Add TUS protocol support for resumable uploads (#7592) 2025-12-14 21:56:07 -08:00
filer_server.go chore(filer): remove -mount.p2p flag; registry is always on (#9183) 2026-04-21 23:00:11 -07:00
master_grpc_server_admin_ping_test.go cluster: restrict Ping RPC to known peers of the requested type (#9445) 2026-05-12 13:00:52 -07:00
master_grpc_server_admin.go cluster: restrict Ping RPC to known peers of the requested type (#9445) 2026-05-12 13:00:52 -07:00
master_grpc_server_assign.go feat: pass expected_data_size from clients for size-aware assignment (#9032) 2026-04-11 11:30:47 -07:00
master_grpc_server_cluster.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00
master_grpc_server_collection.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
master_grpc_server_raft_test.go Add cluster.raft.leader.transfer command for graceful leader change (#7819) 2025-12-19 00:15:39 -08:00
master_grpc_server_raft.go fix(admin): fix master leader link showing incorrect port in Admin UI (#8924) 2026-04-04 11:50:43 -07:00
master_grpc_server_test.go dlm: resilient distributed locks via consistent hashing + backup replication (#8860) 2026-03-30 23:29:56 -07:00
master_grpc_server_volume.go fix(ec): make multi-disk same-server EC reads work + full-lifecycle integration test (#9487) 2026-05-13 13:56:20 -07:00
master_grpc_server.go mini: quieter startup with a docker-compose-style progress board (#9524) 2026-05-17 19:13:09 -07:00
master_server_handlers_admin.go fix: generate topology uuid uniformly in single-master mode (#8405) 2026-02-22 23:45:48 -08:00
master_server_handlers_ui.go hide millseconds in up time (#7553) 2025-11-26 08:01:19 -08:00
master_server_handlers.go feat: pass expected_data_size from clients for size-aware assignment (#9032) 2026-04-11 11:30:47 -07:00
master_server.go Revert #9443 — heartbeat peer binding breaks hostname-based clusters (#9474) 2026-05-12 18:22:21 -07:00
raft_common.go fix: improve raft leader election reliability and failover speed (#8692) 2026-03-18 23:28:07 -07:00
raft_hashicorp_test.go Normalize hashicorp raft peer ids (#8253) 2026-02-09 07:46:34 -08:00
raft_hashicorp.go fix: improve raft leader election reliability and failover speed (#8692) 2026-03-18 23:28:07 -07:00
raft_server_handlers.go fix(wdclient,volume): compare master leader with ServerAddress.Equals (#9089) 2026-04-15 12:29:31 -07:00
raft_server.go go fmt 2026-04-10 17:31:14 -07:00
volume_grpc_admin.go fix(ec): verify full shard set before deleting source volume (#9490) (#9493) 2026-05-13 19:29:24 -07:00
volume_grpc_batch_delete_test.go volume: require admin auth on BatchDelete (#9438) 2026-05-11 13:50:48 -07:00
volume_grpc_batch_delete.go volume: require admin auth on BatchDelete (#9438) 2026-05-11 13:50:48 -07:00
volume_grpc_client_to_master.go fix(ec): blanket-clean every destination over the full shard range (#9512) 2026-05-17 11:31:37 -07:00
volume_grpc_copy_incremental.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
volume_grpc_copy_writefile_test.go fix(ec_distribute): remove partial files on copy stream error (#9543) 2026-05-18 15:19:51 -07:00
volume_grpc_copy.go fix(ec_distribute): remove partial files on copy stream error (#9543) 2026-05-18 15:19:51 -07:00
volume_grpc_erasure_coding_test.go iceberg: wire pagination for list namespaces/tables REST APIs (#8275) 2026-02-09 21:46:55 -08:00
volume_grpc_erasure_coding.go fix(ec_distribute): remove partial files on copy stream error (#9543) 2026-05-18 15:19:51 -07:00
volume_grpc_query.go move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
volume_grpc_read_all.go volume: require admin auth on ReadAllNeedles and VolumeNeedleStatus (#9437) 2026-05-11 13:50:19 -07:00
volume_grpc_read_write.go Export gRPC file_{read,write}_failures metrics on volume servers. (#9177) 2026-04-22 11:22:21 -07:00
volume_grpc_remote_test.go volume: gate FetchAndWriteNeedle behind admin auth and refuse internal endpoints (#9441) 2026-05-12 10:11:20 -07:00
volume_grpc_remote.go volume: gate FetchAndWriteNeedle behind admin auth and refuse internal endpoints (#9441) 2026-05-12 10:11:20 -07:00
volume_grpc_scrub.go fix(volume): add authentication to destructive gRPC admin endpoints (#8876) 2026-05-04 21:14:55 -07:00
volume_grpc_state.go Add a version token on RPCs to read/update volume server states. (#8191) 2026-02-06 10:58:43 -08:00
volume_grpc_tail.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_tier_download.go avoid load volume file with BytesOffset mismatch (#3841) 2022-10-14 00:18:09 -07:00
volume_grpc_tier_upload.go Block RPC write operations on volume servers when maintenance mode is enabled (#8115) 2026-02-02 13:21:02 -08:00
volume_grpc_vacuum.go fix(volume): add authentication to destructive gRPC admin endpoints (#8876) 2026-05-04 21:14:55 -07:00
volume_server_handlers_admin.go chore: remove ~50k lines of unreachable dead code (#8913) 2026-04-03 16:04:27 -07:00
volume_server_handlers_helper.go directory structure change to work with glide 2016-06-02 18:09:14 -07:00
volume_server_handlers_read.go Export file_read_invalid_needles metric for REST read requests on invalid file IDs. (#9241) 2026-04-27 12:22:42 -07:00
volume_server_handlers_ui.go hide millseconds in up time (#7553) 2025-11-26 08:01:19 -08:00
volume_server_handlers_write.go Export REST file_{read,write}_failures metrics on volume servers (#9215) 2026-04-24 11:45:21 -07:00
volume_server_handlers.go fix: JWT validation failures during replication (#7788) (#7795) 2025-12-16 13:42:18 -08:00
volume_server.go cluster: restrict Ping RPC to known peers of the requested type (#9445) 2026-05-12 13:00:52 -07:00
webdav_server.go fix(mount): remove fid pool to stop master over-allocating volumes (#9111) 2026-04-16 15:51:13 -07:00
wrapped_webdav_fs.go chore: execute goimports to format the code (#7983) 2026-01-07 13:06:08 -08:00