docs(s3/lifecycle): refresh DESIGN.md as-built (#9491)
* docs(s3/lifecycle): refresh DESIGN.md as-built + add wiki pages
DESIGN.md was written as a phased implementation plan ("Phase 2 will
ship X, Phase 4 will ship Y"). All phases are now merged, plus the
post-cutover changes from #9477/#9481/#9484/#9485/#9486 substantially
changed the worker model (single subscription, walker throttle,
observability gauges). Rewrite the doc in present tense describing
what's actually there.
Net changes vs the prior plan-style doc:
- Algorithm pseudo-code reflects the single-subscription fan-out plus
walkedThisPass within-pass guard.
- Walker invocation table replaces the implicit "two distinct calls"
prose with three call sites (recovery / steady-state / empty-replay)
and their throttle gates.
- New section on the subscription model (one Reader, ShardPredicate,
fan-out by ev.ShardID).
- New section on cursor.LastWalkedNs and the WalkerInterval throttle.
- Observability section: gauges, heartbeat tokens, what each means.
- "Implementation history" table maps phases to merged PRs.
- "Future work" lists the four optimizations we deferred (long-lived
subscription, bucket-coordinated walker, per-bucket lag metric,
filer meta-log retention).
Drop the "Phase N — ..." narrative from the bottom; the PR history
table is the durable artifact now.
Add wiki pages under docs/wiki/s3-lifecycle/ as source-of-truth for
the operator-facing docs. README explains the sync workflow with the
external seaweedfs.wiki.git repo. Five pages:
- Home.md — landing page, supported rule shapes, what the worker does
- Operator-Guide.md — config knobs, when to change each, walker
interval recommendations by cluster size
- Monitoring.md — Prometheus metric reference + heartbeat token table
+ suggested PromQL alerts
- Troubleshooting.md — stuck cursor, walker stuck, failure outcomes,
cursor schema for manual inspection
- Architecture.md — high-level overview for newcomers; sits between
Home.md (operator) and DESIGN.md (developer)
* docs(s3/lifecycle): address PR review feedback on docs
Coderabbit + gemini findings on #9491:
- Monitoring.md: clarify the "matches all dispatched" phrasing; note
that LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED is the proto zero-value
(shouldn't appear in healthy systems); filter PromQL alerts to
ignore zero-valued gauges so fresh-install heartbeats don't trip.
- Operator-Guide.md, Troubleshooting.md: clarify weed shell -master
format as host:http_port.grpc_port (SeaweedFS ServerAddress).
- Troubleshooting.md: pause the s3_lifecycle job in the admin UI
before manually editing a cursor file, otherwise the worker's
save races with the operator's edit.
- Architecture.md, Home.md, Operator-Guide.md, Monitoring.md,
Troubleshooting.md, DESIGN.md: add language tags (`text`) to
fenced code blocks for markdownlint MD040 compliance.
- DESIGN.md: standardize on the S3 spec rule names
(`ExpiredObjectDeleteMarker`, `NewerNoncurrentVersions`,
`AbortIncompleteMultipartUpload`) and add a one-line note mapping
them to the engine's `ActionKind*` constants.
- README.md: prepend `cd "$(git rev-parse --show-toplevel)"` to the
sync workflow so the `cp` commands' repo-root-relative paths work
whether the operator's shell is at the repo root or at
docs/wiki/s3-lifecycle/.
- Home.md: was lagging the wiki-repo merged version (had the older
pre-merge content). Re-sync from the wiki repo so source matches.
* docs(s3/lifecycle): remove wiki pages from PR
The wiki pages belong in seaweedfs.wiki.git, not the main repo. The
source-of-truth concern that motivated adding them here is real but
the cost — every code-review touchpoint requires reviewers to load
operator-facing pages too — outweighs it. The wiki pages are already
pushed locally (~/dev/seaweedfs.wiki); they'll publish on the
operator-side workflow.
This PR remains scoped to DESIGN.md (the developer-facing reference
that does belong with the code).
* docs(s3/lifecycle): drop Implementation history section
git log is the durable record of what shipped when; the prose table
duplicates it and goes stale faster than commit metadata.
* docs(s3/lifecycle): soften 'exactly once per run' in Goal
The prior phrasing overstated the guarantee versus the failure model
documented later in the same file. Reword to: 'process due objects
each pass; retryable/blocked outcomes get retried from the cursor on
later runs.' Surfaces the head-of-line-blocking semantics up front so
the rest of the doc reads consistently.
Also: drop the stale 'see docs/wiki/s3-lifecycle/' pointer — those
pages live in the wiki repo, not the main repo.
This commit is contained in:
Chris Lu
GitHub
parent
813f1351f8
commit
e9bcb8f4ad
@ -1,492 +1,270 @@
|
||||
# S3 Lifecycle (Design)
|
||||
|
||||
Replaces the current streaming + heap design with a daily meta-log replay. The worker becomes "start, do today's work, stop" — matching the operator's mental model and removing the per-shard heap that grows O(events × TTL_days).
|
||||
The S3 lifecycle worker replaces the streaming + heap design with a daily meta-log replay. The worker runs as a scheduled job: "start, do today's work, stop" — no long-running per-shard goroutines, no future-buffered match heap.
|
||||
|
||||
This document is the as-built reference. For operator-facing guides, see the [SeaweedFS wiki](https://github.com/seaweedfs/seaweedfs/wiki).
|
||||
|
||||
## Goal
|
||||
|
||||
For each bucket lifecycle rule with TTL `D` days, delete every object whose age exceeds `D`, exactly once per day, with the worker exiting when the pass completes. No future-buffered Matches in memory. No tick-driven dispatch loop. Cluster-wide delete rate cap allocated per worker.
|
||||
For each bucket lifecycle rule with TTL `D` days, the worker processes every object whose age exceeds `D` during each scheduled run, dispatching deletes via `LifecycleDelete`. Events whose dispatch returned a retryable / blocked outcome are reprocessed from the persisted cursor on later runs — head-of-line blocking is intentional rather than a per-key retry queue. No future-buffered Matches in memory. Cluster-wide delete rate cap allocated per worker. The worker exits when the pass completes.
|
||||
|
||||
## Algorithm
|
||||
|
||||
One pass per day, per shard:
|
||||
One pass per scheduled invocation. The pass spawns one filer meta-log subscription, fans events out to per-shard processors by `ShardID(bucket, key)`, and drains each shard's events through the router + dispatcher:
|
||||
|
||||
```text
|
||||
daily_run(shardID):
|
||||
persisted = load_cursor(shardID) // { TsNs, rule_set_hash, promoted_hash }
|
||||
earliest_available = metalog.EarliestAvailableTsNs(shardID)
|
||||
retention_window = duration(now - earliest_available) // time.Duration; sole window input to engine
|
||||
snap = engine.CurrentSnapshot() // base; per-action active/Mode reflect compile-time state, not partition
|
||||
replay, walk = snap.RulesForShard(shardID, retention_window)
|
||||
if replay == nil and walk == nil: return // bucket has no rules
|
||||
rsh = engine.ReplayContentHash(snap) // content over ExpirationDays/NoncurrentDays/AbortMPU rules; partition-independent
|
||||
promoted = engine.PromotedHash(snap, retention_window) // rules currently in walk due to scan_only
|
||||
dailyrun.Run(ctx, cfg):
|
||||
runNow = cfg.Now() // frozen for the whole pass
|
||||
snap = cfg.Engine.Snapshot()
|
||||
rsh = engine.ReplayContentHash(snap)
|
||||
maxTTL = engine.MaxEffectiveTTL(snap)
|
||||
|
||||
needs_recovery_walk = (
|
||||
persisted is fresh // cold start
|
||||
or persisted.rule_set_hash != rsh // replay-rule edit
|
||||
or persisted.promoted_hash != promoted // partition flip in either direction
|
||||
or (replay != nil and persisted.TsNs < earliest_available) // retention loss; meaningless if pure walker bucket
|
||||
)
|
||||
if rsh != [32]byte{}: // replay-eligible rules present
|
||||
globalStartTsNs = min over cfg.Shards of (persisted cursor or runNow - maxTTL)
|
||||
reader = subscribeMeta(ShardPredicate ∈ cfg.Shards, StartTsNs = globalStartTsNs)
|
||||
fanOut(reader.Events → shardEvents[shardID]) until ev.TsNs > runNow
|
||||
|
||||
if needs_recovery_walk:
|
||||
// Catch already-due objects across ALL rules — both walker-bound and
|
||||
// replay-bound. Replay-bound rules may have objects whose mtime is
|
||||
// older than (now - max_ttl) that the steady-state replay window cannot
|
||||
// reach. The recovery view force-activates every action so the walker
|
||||
// doesn't skip event-driven rules that the base snapshot has marked
|
||||
// inactive (e.g., pre-BootstrapComplete).
|
||||
walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)
|
||||
if replay == nil:
|
||||
save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
else:
|
||||
save_cursor(shardID, { TsNs: now - engine.MaxEffectiveTTL(replay), rule_set_hash: rsh, promoted_hash: promoted })
|
||||
spawn one goroutine per shard:
|
||||
runShard(ctx, cfg, snap, runNow, shardID, shardEvents[shardID])
|
||||
|
||||
wait all
|
||||
teardown reader + fan-out
|
||||
|
||||
runShard(ctx, cfg, snap, runNow, shardID, events):
|
||||
persisted, found = cfg.Persister.Load(shardID)
|
||||
retentionWindow = cfg.RetentionWindow or maxTTL // see "retention" below
|
||||
promoted = engine.PromotedHash(snap, retentionWindow)
|
||||
|
||||
if rsh == [32]byte{}: // pure walker bucket
|
||||
if walkerDue and Walker:
|
||||
cfg.Walker(walkView) // RulesForShard.walk
|
||||
lastWalkedNs = runNow.UnixNano()
|
||||
save cursor (TsNs=0, rsh, promoted, lastWalkedNs)
|
||||
return
|
||||
|
||||
// --- Steady-state path. ----------------------------------------------
|
||||
if walk != nil:
|
||||
walker.RunForShard(ctx, shardID, walk, limiter) // walker-bound actions only
|
||||
mustWalkRecovery = found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
|
||||
mustWalkColdStart = !found
|
||||
|
||||
if replay == nil:
|
||||
save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
return
|
||||
if mustWalkRecovery or mustWalkColdStart:
|
||||
cfg.Walker(engine.RecoveryView(snap)) // every rule, force-active
|
||||
walkedThisPass = true
|
||||
lastWalkedNs = runNow.UnixNano()
|
||||
if mustWalkRecovery:
|
||||
save cursor (TsNs = runNow - maxTTL, rsh, promoted, lastWalkedNs) // rewind
|
||||
return
|
||||
|
||||
max_ttl = engine.MaxEffectiveTTL(replay)
|
||||
cursor = max(persisted.TsNs, now - max_ttl)
|
||||
done = {r: false for r in replay.activeRules()}
|
||||
last_ok = persisted.TsNs
|
||||
if Walker and !walkedThisPass and walkerDue:
|
||||
cfg.Walker(walkView) // steady-state walker
|
||||
lastWalkedNs = runNow.UnixNano()
|
||||
|
||||
for ev in metalog.from(shardID, cursor): // ascending TsNs
|
||||
if all(done.values()): break
|
||||
halt = false
|
||||
matches = router.Route(ctx, replay, ev, now, lister) // existing signature; replay is *engine.Snapshot
|
||||
for m in matches:
|
||||
r = replay.lookup(m.RuleHash)
|
||||
if r == nil or done[r]: continue
|
||||
if m.DueTime > now:
|
||||
done[r] = true // monotonic for replay kinds
|
||||
continue
|
||||
limiter.wait()
|
||||
outcome = dispatch_delete(m.Bucket, m.ObjectKey, m.VersionID,
|
||||
expected_mtime=m.Identity.mtime, ruleHash=m.RuleHash)
|
||||
if outcome is unresolved (RETRY_LATER, transport error after in-run retries, BLOCKED):
|
||||
halt = true; break
|
||||
if halt: break
|
||||
last_ok = ev.TsNs
|
||||
|
||||
save_cursor(shardID, { TsNs: last_ok, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
startTsNs = !found ? runNow - maxTTL : persisted.TsNs // steady state honors cursor
|
||||
lastOK, _, drainErr = drainShardEvents(ctx, cfg, runNow, shardID, snap, startTsNs, events)
|
||||
save cursor (TsNs = lastOK, rsh, promoted, lastWalkedNs)
|
||||
```
|
||||
|
||||
Two distinct walker invocations, deliberately:
|
||||
Walker invocations — three call sites, distinct conditions:
|
||||
|
||||
| Branch | Walker input | Why |
|
||||
|---|---|---|
|
||||
| Recovery (cold-start, replay-rule edit, retention loss, partition flip) | `engine.RecoveryView(snap)` — every action clone force-`markActive()`'d, `Mode` preserved | Catch already-due objects across all rules — including replay-bound rules whose subjects are older than `max_ttl` (replay scan window cannot reach), *and* event-driven rules the base snapshot may mark inactive (`compile.go:73-74` requires `BootstrapComplete`). |
|
||||
| Steady-state | `walk` (walker-only actions) | Replay handles its own rules via the meta-log scan. Walker only runs for actions that have no replay path. |
|
||||
| Branch | View | Trigger | Throttle |
|
||||
|---|---|---|---|
|
||||
| Recovery | `engine.RecoveryView(snap)` | `mustWalkColdStart` or `mustWalkRecovery` | Unconditional |
|
||||
| Steady state | `RulesForShard.walk` | `!walkedThisPass && walkerDue` | `WalkerInterval` |
|
||||
| Empty replay | `RulesForShard.walk` | `rsh == [32]byte{}` and `walkerDue` | `WalkerInterval` |
|
||||
|
||||
`router.Route` and `walker.RunForShard` keep their existing `*engine.Snapshot` parameter; `replay`, `walk`, and `snap` are all `*Snapshot` values.
|
||||
`walkerDue` returns true when `WalkerInterval == 0`, or when no walk has happened yet (`LastWalkedNs == 0`), or when `runNow - LastWalkedNs >= WalkerInterval`. Within-pass double-fire suppression lives in `runShard`'s `walkedThisPass` flag, not in `walkerDue` — the recovery branch fires the walker with `RecoveryView` (a superset of every per-shard partition), so the steady-state branch must not re-walk in the same pass.
|
||||
|
||||
The single API surface is:
|
||||
## Engine surface
|
||||
|
||||
```go
|
||||
// In engine:
|
||||
func CurrentSnapshot() *Snapshot // base
|
||||
func (e *Engine) Snapshot() *Snapshot
|
||||
func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
|
||||
func RecoveryView(s *Snapshot) *Snapshot // every action force-activated; recovery walker only
|
||||
func ReplayContentHash(s *Snapshot) [32]byte // content over replay-eligible rules; partition-independent
|
||||
func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte // hash of replay-eligible rules currently in walk (scan_only)
|
||||
func MaxEffectiveTTL(s *Snapshot) time.Duration // max over active replay actions
|
||||
func RecoveryView(s *Snapshot) *Snapshot
|
||||
func ReplayContentHash(s *Snapshot) [32]byte
|
||||
func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte
|
||||
func MaxEffectiveTTL(s *Snapshot) time.Duration
|
||||
```
|
||||
|
||||
`RulesForShard` and `RecoveryView` both return new `*Snapshot` instances with cloned `*CompiledAction` objects and shared (by pointer) rule definitions. The fields that differ from the base are:
|
||||
`RulesForShard` and `RecoveryView` return new `*Snapshot` instances with cloned `*CompiledAction` objects and shared (by pointer) rule definitions. Fields that differ from the base:
|
||||
|
||||
- **`active`** (per clone) — set per view.
|
||||
- **`Mode`** (per clone) — rewritten to `ModeEventDriven` on `replay` clones; preserved as-is on `walk` and `recovery` clones. The rewrite is required because `router.Route` gates on `Mode == ModeEventDriven` (`router.go:134, 253, 413, 479, 566, 623, 844`), and today's compile preserves a persistent `prior.Mode = ModeScanOnly` that would otherwise lock the action out of replay even after retention rehabilitates it.
|
||||
- **Action-map membership** — `replay`'s map contains only replay-eligible clones; `walk`'s contains only walker-bound clones; `recovery`'s contains every action.
|
||||
- **`active`** — per clone, set per view.
|
||||
- **`Mode`** — rewritten to `ModeEventDriven` on `replay` clones; preserved on `walk` and `recovery` clones. The rewrite is required because `router.Route` gates on `Mode == ModeEventDriven`, and today's compile preserves a persistent `prior.Mode = ModeScanOnly` that would otherwise lock a rule out of replay even after retention rehabilitates it.
|
||||
- **Action-map membership** — `replay` contains only replay-eligible clones; `walk` contains only walker-bound clones; `recovery` contains every action.
|
||||
|
||||
Per-view settings are summarized in the table under "`router.Route` integration."
|
||||
Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table.
|
||||
|
||||
`router.Route` is the existing entry point — it computes `ComputeDueAt`, applies predicate filters, expands one event into 0+ `Match`es (e.g., a demoting PUT produces one for the new version's lifecycle and another for the displaced version's `NoncurrentDays`). The replay path delegates due-time math to the engine so all replay-eligible action kinds (`ExpirationDays`, `NoncurrentDays`, `AbortMPU`) are handled uniformly.
|
||||
## `router.Route` integration
|
||||
|
||||
### `router.Route` integration
|
||||
|
||||
Today's `router.Route(ctx, snap *engine.Snapshot, ev, now, lister)` iterates every action with `IsActive() == true` in the snapshot. `IsActive` is **instance-level state on `*CompiledAction`** (`engine/compile.go:73-85`, `ca.markActive()`), so two snapshots can't disagree on activation if they share the same `*CompiledAction` pointers.
|
||||
|
||||
**Approach: views are independent `*Snapshot` instances with cloned `CompiledAction` objects.**
|
||||
|
||||
Two gating fields are instance-level on `*CompiledAction` and must both match what the downstream call site expects:
|
||||
|
||||
- `active` (`engine/compile.go:73-85`) — gates routing and walker iteration.
|
||||
- `Mode` — `router.Route` requires `Mode == engine.ModeEventDriven` at every emit site (`router.go:134, 253, 413, 479, 566, 623, 844`); the walker rejects only `Mode == engine.ModeDisabled` (`walker.go:147`). Today's compile preserves `prior.Mode`, so a rule previously promoted to `ModeScanOnly` will still be skipped by `router.Route` even if `active=true`.
|
||||
|
||||
`engine.Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)` returns two new `*Snapshot` instances. Each has its own `actions map[ActionKey]*CompiledAction` containing *clones* of the originals with **both `active` and `Mode` rewritten** per view. The partition predicate is "TTL ≤ retentionWindow" for replay membership; the engine does no clock arithmetic of its own — `retentionWindow` is the only window input it receives:
|
||||
`router.Route(ctx, snap, ev, now, lister)` iterates every action with `IsActive() == true` in the snapshot. Two snapshots can't disagree on activation if they share the same `*CompiledAction` pointers, which is why `RulesForShard` clones.
|
||||
|
||||
| View | Clone settings | Why |
|
||||
|---|---|---|
|
||||
| `replay` | `active = true`, `Mode = ModeEventDriven` | `router.Route` requires `ModeEventDriven`. Forced regardless of `prior.Mode`, so a rule previously stuck in `ModeScanOnly` is rehabilitated when retention now fits its TTL. |
|
||||
| `walk` | `active = true`, `Mode` preserved from original (any non-`ModeDisabled` is accepted by the walker) | Walker doesn't gate on a specific Mode value, so no rewrite needed beyond ensuring `active`. |
|
||||
| `recovery` (see below) | `active = true`, `Mode` preserved | Recovery walker iterates all action clones; preserving `Mode` lets each rule's evaluator do its normal predicate work. |
|
||||
| `replay` | `active = true`, `Mode = ModeEventDriven` | `router.Route` requires `ModeEventDriven`. Forced regardless of `prior.Mode`. |
|
||||
| `walk` | `active = true`, `Mode` preserved | Walker accepts any non-`ModeDisabled` Mode. |
|
||||
| `recovery` | `active = true`, `Mode` preserved | Walker iterates all action clones. |
|
||||
|
||||
Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table, and other immutable data — memory overhead is one `CompiledAction` struct per action per view (tens of bytes × dozens of actions, not megabytes).
|
||||
## Subscription model
|
||||
|
||||
Membership:
|
||||
One filer `SubscribeMetadata` stream per `dailyrun.Run()` call, covering every shard in `cfg.Shards`. The `Reader` carries a `ShardPredicate func(int) bool` that accepts the shard set; a fan-out goroutine routes events to per-shard channels by `ev.ShardID`. This replaces the earlier model (16 separate per-shard subscriptions per pass).
|
||||
|
||||
- `replay`: clones of `ExpirationDays`, `NoncurrentDays`, `AbortMPU` actions whose TTL ≤ `retentionWindow`. The Mode rewrite is what makes "rehabilitation" work — retention recovers, a previously `ModeScanOnly`-stuck rule clone now reads as `ModeEventDriven` in the replay view.
|
||||
- `walk`: clones of `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, plus `scan_only`-promoted clones whose TTL exceeds `retentionWindow`.
|
||||
- Either return value may be `nil` if its partition is empty.
|
||||
`globalStartTsNs = min(per-shard cursor, runNow - maxTTL)`. Pre-loaded once at pass start so the subscription's `StartTsNs` covers every shard's needed range; per-shard drains then filter `ev.TsNs <= shard.startTsNs` locally.
|
||||
|
||||
The daily replay loop calls `router.Route(ctx, replay, ev, now, lister)`. Steady-state walker calls `walker.RunForShard(ctx, shardID, walk, limiter)`. Both call sites are drop-in — they see the correct, per-partition set with the correct `(active, Mode)` for their gating logic.
|
||||
Fan-out cancels the reader on the first `ev.TsNs > runNow` (meta-log events arrive in TsNs order; everything after is past the pass boundary). Per-shard channels are buffered to 256 events — large enough to absorb bursts without back-pressuring the fan-out.
|
||||
|
||||
**Recovery view (separate).** The recovery branch needs the walker to see **all** replay-eligible + walker-eligible actions as active, including event-driven actions that the base snapshot may have marked inactive (`compile.go:73-74`: event-driven actions require `BootstrapComplete`). Add:
|
||||
## Action kinds and dispatch paths
|
||||
|
||||
Table uses the S3-spec rule names (what operators type in lifecycle XML). The corresponding engine constants in `weed/s3api/s3lifecycle/action_kind.go` are `ActionKindExpirationDays`, `ActionKindNoncurrentDays`, `ActionKindAbortMPU`, `ActionKindExpirationDate`, `ActionKindExpiredDeleteMarker`, `ActionKindNewerNoncurrent` — same one-to-one mapping, shorter spelling.
|
||||
|
||||
| ActionKind | Trigger | Due time | Path | Early-stop in replay? |
|
||||
|---|---|---|---|---|
|
||||
| `ExpirationDays` | Latest-version PUT | `ev.TsNs + r.ExpirationDays` | Replay | Yes |
|
||||
| `NoncurrentDays` | Demotion (next PUT for same key) | `entry.NoncurrentSince + r.NoncurrentDays` | Replay | Yes |
|
||||
| `AbortIncompleteMultipartUpload` | MPU init | `mpu_init.TsNs + r.AbortMPUDaysAfterInitiation` | Replay | Yes |
|
||||
| `ExpirationDate` | Latest-version PUT, fires on `now >= r.ExpirationDate` | `r.ExpirationDate` (constant) | Walker | n/a |
|
||||
| `ExpiredObjectDeleteMarker` | Delete marker with `NumVersions == 1` | "now if orphaned, else never" | Walker | n/a |
|
||||
| `NewerNoncurrentVersions` | Version becomes noncurrent AND total noncurrents > `r.NewerNoncurrentVersions` | "now if over the cap, else never" | Walker | n/a |
|
||||
|
||||
`ExpiredObjectDeleteMarker` and `NewerNoncurrentVersions` are walker-only because their due-time depends on current sibling state, not on any event's TsNs. The `done` early-stop in replay can't engage — there's nothing event-time-monotonic to early-stop on.
|
||||
|
||||
## Cursor
|
||||
|
||||
Persisted per shard at `/etc/s3/lifecycle/daily-cursors/shard-NN.json`. Shape (`weed/s3api/s3lifecycle/dailyrun/cursor.go`):
|
||||
|
||||
```go
|
||||
func RecoveryView(s *Snapshot) *Snapshot
|
||||
type Cursor struct {
|
||||
TsNs int64 // last meta-log event whose matches all dispatched
|
||||
RuleSetHash [32]byte // ReplayContentHash of the rule set that wrote this
|
||||
PromotedHash [32]byte // PromotedHash with retentionWindow at write time
|
||||
LastWalkedNs int64 // wall-clock of the last successful walker fire
|
||||
}
|
||||
```
|
||||
|
||||
`RecoveryView` is a third `*Snapshot` instance whose action clones are all `markActive()`'d, with `Mode` preserved from the original. The walker accepts any non-`ModeDisabled` Mode, so no rewrite is needed. The recovery branch calls `walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)`. Without this, recovery would silently skip any rule the base snapshot considers inactive — defeating the recovery branch's purpose of catching already-due objects across the full rule set.
|
||||
`LastWalkedNs` is JSON-omitempty, so cursor files written before that field existed decode cleanly as zero (treated as "never walked steady-state" → next pass seeds the anchor).
|
||||
|
||||
`RecoveryView` does not participate in steady-state. It exists only for the recovery walker invocation.
|
||||
Cursor save uses a fresh `context.Background()` with a 5s timeout because the steady-state drain exits via passCtx cancellation (the only way an idle subscription ends). Saving with the canceled passCtx would silently drop the cursor and the next pass would re-replay from the same floor.
|
||||
|
||||
**`scan_only` input path.** The retention boundary used for promotion is supplied explicitly via `retentionWindow time.Duration`, not derived inside the engine. `daily_run` computes it once at run start (`retention_window = duration(now - metalog.EarliestAvailableTsNs(shardID))`) and passes it to `RulesForShard` and `PromotedHash`. The promotion decision is therefore stable across one `daily_run` invocation — a rule that flips between `replay` and `walk` mid-run does not re-route; it gets its new bucket on the next day's `RulesForShard` call. `RecoveryView` takes only the base snapshot — it activates everything, partition-independent.
|
||||
In steady state the start position honors the cursor verbatim — the floor `runNow - maxTTL` is applied only on cold start (`!found`). The drain freezes the cursor at the last pre-skip event so pending matches with `DueTime == TsNs + maxTTL` stay in scope across passes; bumping forward in steady state would orphan exactly those events.
|
||||
|
||||
This approach keeps `router.Route` and `walker.Run` signatures unchanged. The trade-off is cloning `CompiledAction` per view (and rewriting `Mode` in the replay view); the alternative (adding a filter parameter to `Route` or restructuring `IsActive`/`Mode` to be view-scoped) would force every call site — including the streaming dispatcher during Phase 2/3 — to thread the filter, while clone-per-view localizes the partition decision to the engine. The Mode rewrite also bypasses today's persistent `prior.Mode = ModeScanOnly` lock-in, which is now obsoleted by the new partition logic.
|
||||
## Cursor hashes
|
||||
|
||||
Why correct:
|
||||
The cursor stores two hashes that together detect every situation invalidating "everything before persisted.TsNs has been processed under the same rules":
|
||||
|
||||
- Meta-log is TsNs-ordered. For any rule `R`, `due_time` is also monotonic in TsNs (see "Due-time source"), so the first not-yet-due event for `R` proves every later event is also not-yet-due. Hence the `done` flag.
|
||||
- Multiple rules fan out on the same scan: one meta-log read, N predicate checks per event.
|
||||
- Identity drift (object overwritten between event and delete) is caught at the RPC by the existing `LifecycleDelete` identity-CAS, which returns `NOOP_RESOLVED` for stale events. The algorithm dispatches optimistically and lets the server filter.
|
||||
**`RuleSetHash = engine.ReplayContentHash(snap)`** — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds. Partition-independent.
|
||||
|
||||
### Action kinds and dispatch paths
|
||||
|
||||
The engine defines six action kinds (`weed/s3api/s3lifecycle/action_kind.go`). Each has a distinct trigger and due-time semantics, and they split into two dispatch paths. Replay path uses meta-log iteration with the `done` early-stop; walker path lists current bucket state.
|
||||
|
||||
| ActionKind | Trigger | Due time | Path | Early-stop? |
|
||||
|---|---|---|---|---|
|
||||
| `ExpirationDays` | Latest-version PUT | `ev.TsNs + r.ExpirationDays` | Replay | Yes — `due` strictly increasing in iteration order |
|
||||
| `NoncurrentDays` | Demotion (next PUT for same key) | `entry.noncurrent_since + r.NoncurrentDays` | Replay | Yes — `noncurrent_since` is the demoting event's TsNs (see "`noncurrent_since`" below), monotonic |
|
||||
| `AbortMPU` | MPU init | `mpu_init.TsNs + r.AbortMPUDaysAfterInitiation` | Replay | Yes — same monotonicity as `ExpirationDays` |
|
||||
| `ExpirationDate` | Latest-version PUT, fired on `now >= r.ExpirationDate` | `r.ExpirationDate` (constant) | Walker | n/a — fixed wall-clock date, fires once per object |
|
||||
| `ExpiredDeleteMarker` | Delete marker with `NumVersions == 1` (no remaining siblings) | "now if orphaned, else never" — depends on current bucket state, not event age | Walker | n/a — requires sibling lookup, not derivable from event TsNs |
|
||||
| `NewerNoncurrent` | Version becomes noncurrent AND total noncurrents > `r.NewerNoncurrentVersions` | "now if over the count cap, else never" — count is current state | Walker | n/a — requires version-list lookup |
|
||||
|
||||
Why `ExpiredDeleteMarker` and `NewerNoncurrent` are walker-only:
|
||||
|
||||
Both rules' "due time" depends on the *current* state of sibling versions, not on any single event's timestamp. Modeling them as replay would force a sibling-list lookup on every meta-log event for the matching keys (today's `router.Route` does exactly this), and the `done` early-stop would never engage because the trigger condition isn't event-time-monotonic. They fit the walker far better: list each key's versions once per day, evaluate the rule against current state, emit deletes. Today's `bootstrap/walker.go` already handles them via `EvaluateAction`.
|
||||
|
||||
`expected_mtime` passed to `LifecycleDelete` is always the entry's original mtime (used for identity CAS), regardless of action kind. The TTL clock and the CAS identity are separate concerns.
|
||||
|
||||
### Rule changes
|
||||
|
||||
Rule changes invalidate the persisted cursor's assumption that "everything older has already been processed under the same rules." The algorithm tracks `rule_set_hash` alongside the cursor and forces a bootstrap walk on mismatch. Cases this covers:
|
||||
|
||||
- New rule added → walker discovers existing objects under the new rule.
|
||||
- TTL increased (e.g. 30d → 60d) → walker discovers objects in `[now-60d, now-30d]` that were skipped while the old rule ran.
|
||||
- Predicate widened (prefix/tag filter change) → walker re-evaluates everything.
|
||||
- TTL decreased / predicate narrowed → walker still runs (harmless re-evaluation), but the next daily run picks up steady-state quickly.
|
||||
|
||||
Bootstrap walker on rule change is a one-shot full bucket list. Cost is proportional to bucket size, not history; acceptable as an infrequent event.
|
||||
|
||||
### Bootstrap retention window
|
||||
|
||||
The walker only deletes objects whose age already exceeds their TTL. Not-yet-due objects are intentionally left for the meta-log path. If the post-walker cursor were set to `now`, the next daily replay would scan from `now` forward and never see the PUT event for, say, a 20-day-old object under a 30-day rule — that event sits at `TsNs ≈ now - 20d`, before the cursor.
|
||||
|
||||
The cursor is therefore reset to `now - max_ttl` (excluding `ScanAtDate` rules from `max_ttl`, since those don't use the replay path). The clamp `max(persisted, now - max_ttl)` in the steady-state algorithm preserves the sliding window: each day the effective scan range is `[now - max_ttl, now]`, and not-yet-due events surface naturally as they age in.
|
||||
|
||||
Invariant this depends on: **meta-log retention ≥ max_ttl**. If retention is shorter, events for some still-pending objects age out before they become due. Detected at run start by comparing the meta-log's earliest available TsNs against `now - max_ttl`; if the gap is non-empty, the affected rules are flagged `scan_only` for that run and routed to the walker (see "Scan-at-date and scan-only paths").
|
||||
|
||||
### Replay vs walker: single source of truth
|
||||
|
||||
`Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)` is the **only** place that decides which path each compiled action takes during steady state. Both return values are new `*Snapshot` instances containing cloned `*CompiledAction` values with per-partition `active` flags (see "`router.Route` integration" for why cloning is required); either may be `nil` when its partition is empty. Every other component (`daily_run`, the walker invocation, the cursor's `RuleSetHash`, the `max_ttl` calculation) reads through these views. A given compiled action's clone has `active=true` in exactly one of `replay` or `walk`:
|
||||
|
||||
| Reason for walker | Engine signal |
|
||||
|---|---|
|
||||
| `ExpirationDate` (`ModeScanAtDate`) | `compile.go:73` sets `mode == ModeScanAtDate` |
|
||||
| `ExpiredDeleteMarker` | Action kind requires sibling count |
|
||||
| `NewerNoncurrent` | Action kind requires version count |
|
||||
| `scan_only` (TTL > meta-log retention) | `event_log_horizon.go` check against earliest available TsNs at run start |
|
||||
|
||||
Everything else (`ExpirationDays`, `NoncurrentDays`, `AbortMPU` when TTL fits in retention) is on the replay list.
|
||||
|
||||
The `effective_ttl` per active replay action — used by `engine.MaxEffectiveTTL(replay)` — is:
|
||||
|
||||
- `ExpirationDays.ExpirationDays`
|
||||
- `NoncurrentDays.NoncurrentDays`
|
||||
- `AbortMPU.AbortMPUDaysAfterInitiation`
|
||||
|
||||
Walker actions don't appear as active in `replay`, so they naturally contribute nothing to `MaxEffectiveTTL` — the cursor sliding window only governs replay.
|
||||
|
||||
If `snap.replay` is empty (e.g., a bucket whose only rule is `ExpirationDate`), the cursor's `TsNs` is recorded as `0` (or any sentinel) and never read; only `RuleSetHash` matters, so a future addition of a replay rule is detected and triggers the rewind branch. The walker still runs every day for the walker rules.
|
||||
|
||||
### Cursor hashes: `RuleSetHash` and `PromotedHash`
|
||||
|
||||
The cursor stores two hashes. Together they detect every situation that invalidates the cursor's "everything before persisted.TsNs has been processed under the same rules" assumption.
|
||||
|
||||
**`RuleSetHash = engine.ReplayContentHash(snap)`** — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds (`ExpirationDays`, `NoncurrentDays`, `AbortMPU`) in the base snapshot. Partition-independent.
|
||||
|
||||
**`PromotedHash = engine.PromotedHash(snap, retentionWindow)`** — hash of the set of replay-eligible rules that are currently classified as `walk` due to `scan_only` (their TTL exceeds `retentionWindow`). Changes when retention shifts move rules between partitions, even if XML rules and their content are unchanged.
|
||||
|
||||
Why two hashes: a replay-cursor rewind is needed when either the rule definitions change *or* when a rule transitions between partitions in either direction.
|
||||
|
||||
- Promotion (replay → walk, retention dropped): the promoted rule's meta-log events for in-flight objects are no longer reachable. Walker must take over for that rule. Cursor for other replay rules is fine, but the recovery branch's full-rule-set walk is still the safest catch-up — `PromotedHash` change triggers it.
|
||||
- Demotion (walk → replay, retention recovered): the rule's events are now reachable again in meta-log. **But persisted.TsNs may be forward of where the rule's relevant events live**, because other replay rules kept the cursor advancing while this rule was in walk. The `persisted.TsNs < earliest_available` check does not catch this — that check fires only on retention regression. Demotion is detected by `PromotedHash` change.
|
||||
**`PromotedHash = engine.PromotedHash(snap, retentionWindow)`** — hash of replay-eligible rules currently classified as `walk` due to `scan_only` promotion (their TTL exceeds `retentionWindow`).
|
||||
|
||||
Recovery triggers, complete list:
|
||||
|
||||
| Trigger | Detection | Why |
|
||||
|---|---|---|
|
||||
| Cold start | No persisted cursor | First run |
|
||||
| Replay-rule edit | `RuleSetHash` mismatch | Content of replay-eligible rules changed |
|
||||
| Retention loss | `replay != nil && persisted.TsNs < earliest_available` | Worker was down or retention shrank past cursor. Gated on `replay != nil` so the TsNs sentinel (`0`) used by pure walker buckets doesn't perpetually trigger recovery. |
|
||||
| Cold start | No persisted cursor | First run for this shard |
|
||||
| Replay-rule edit | `RuleSetHash` mismatch | Replay-eligible rule content changed |
|
||||
| Partition flip | `PromotedHash` mismatch | A replay-eligible rule moved between `replay` and `walk` |
|
||||
|
||||
Walker-only rule edits (`ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule with no replay-eligible action kind) do not change either hash. They are picked up by the steady-state `walk` invocation; replay cursor is untouched.
|
||||
Retention loss as a recovery trigger is a known gap: in stock SeaweedFS the filer's meta-log is effectively never GC'd (no on-disk retention policy for `/topics/.system/log`), so `cfg.RetentionWindow` defaults to `maxTTL` and PromotedHash stays empty. When operators add explicit meta-log retention, the cursor-vs-earliest-available check becomes load-bearing again.
|
||||
|
||||
Practical consequences:
|
||||
## Walker throttle
|
||||
|
||||
- Adding/removing/modifying a walker-only rule: both hashes unchanged, replay cursor untouched.
|
||||
- Retention drops (replay rule promotes to walk): `RuleSetHash` unchanged, `PromotedHash` changes → recovery fires. Walker over base catches already-due objects; cursor rewinds for replay continuity.
|
||||
- Retention recovers (rule demotes back to replay): `RuleSetHash` unchanged, `PromotedHash` changes → recovery fires. The demoted rule's events get a fresh scan window via the rewind.
|
||||
- Editing a replay-eligible XML rule: `RuleSetHash` changes → recovery fires. Walker covers the full rule set; already-due objects under the new rule are caught.
|
||||
`cfg.WalkerInterval` decouples the walker's cadence from `Run()` invocation cadence. The s3tests CI workflow invokes the worker every 2s; a single daily admin schedule invokes it once per day. The walker fire rate should be set by walk cost, not by the invocation interval — so:
|
||||
|
||||
### Shard-aware walker
|
||||
- Steady-state and empty-replay walker fires gate on `walkerDue(persisted.LastWalkedNs, runNow, WalkerInterval)`.
|
||||
- `0` keeps the prior "fire every pass" behavior (back-compat for tests).
|
||||
- Production: pick the walk cost budget per shard per cluster. Small cluster: 1h. Large cluster: 6h+.
|
||||
- Recovery walker fires (cold-start, hash mismatch) are unconditional — these are bounded events that must run once.
|
||||
|
||||
Today's walker (`bootstrap/walker.go`) is bucket-wide: it lists every entry under a bucket and emits matches regardless of `ShardID`. Shards are derived as `sha256(bucket||"/"||key) >> 252` (`weed/s3api/s3lifecycle/shard.go:23`), so a single bucket's objects spread across all 16 shards.
|
||||
Walker fires update `Cursor.LastWalkedNs` so the next pass's throttle has a fresh anchor. The recovery walker also updates it, so a steady-state branch in the same pass doesn't double-walk over the same superset.
|
||||
|
||||
Daily replay is per-shard, so a per-shard `daily_run(shardID)` invoking the bucket-wide walker would have all 16 shards re-walk every bucket — 16× the filer listing cost, 16× the duplicate delete RPCs (idempotent but wasteful and noisy in metrics).
|
||||
## Delete failure handling
|
||||
|
||||
Two paths are workable; the design picks the simpler one and notes the optimization:
|
||||
Cursor advance is gated on success. The cursor only moves past events whose matches all returned `DONE`, `NOOP_RESOLVED`, or `SKIPPED_OBJECT_LOCK`. Any other outcome (`RETRY_LATER`, `BLOCKED`, transport error after in-run retries) halts the run and persists the cursor at the last fully-processed event.
|
||||
|
||||
1. **Shard-filtering walker (chosen).** Add `walker.RunForShard(shardID int, ...)`. The walker still does one full bucket listing, but for each entry it computes `ShardID(bucket, key)` and skips entries not in the target shard. Cost: still 1 listing per shard per bucket = 16× listing total. Simpler; no coordination across workers. Acceptable for buckets up to ~100M entries where listing is cheap relative to deletes.
|
||||
2. **Bucket-coordinated walker (future).** A separate coordinator (e.g., the worker that owns shard 0 for that bucket) runs the walker once, routing emitted matches to each shard's local injection point. 1 listing per bucket total. Requires a coordination layer that today's plugin/worker model doesn't have. Tracked as an optimization, not blocking.
|
||||
|
||||
Path #1 is wire-compatible with today's walker — only the iteration filter changes. `RunForShard` is a thin wrapper that takes a shard predicate; the existing `Run(bucket)` becomes `RunForShard(shard, bucket)` with `shard == any`.
|
||||
|
||||
### Delete failure handling
|
||||
|
||||
Cursor advance is gated on success. The cursor only moves past events whose deletes returned `DONE`, `NOOP_RESOLVED`, or `SKIPPED_OBJECT_LOCK` (lock-protected, but rule semantics are satisfied). Any other outcome — `RETRY_LATER`, `BLOCKED`, or transport error after in-run retries — halts the run and persists the cursor at the last fully-processed event.
|
||||
|
||||
Semantics:
|
||||
|
||||
- **Head-of-line blocking is intentional.** A transient filer error stalls the rest of today's pass; tomorrow's run resumes at the same cursor and retries from the failing event. This is loud — operator sees a stuck cursor in metrics — and idempotent (identity-CAS makes any redundant delete a no-op).
|
||||
- **In-run retry with backoff** for transport errors only (default 3 attempts, exponential backoff). Server-side outcomes (`RETRY_LATER`, `BLOCKED`) are not retried in-run; the server already classified them as "wait until next run."
|
||||
- **No separate retry queue.** Removing the per-key freeze state was the whole point — adding it back would re-introduce the state machine this design replaces.
|
||||
|
||||
## Data model changes
|
||||
|
||||
### `noncurrent_since` on version entries
|
||||
|
||||
A non-current version's TTL clock starts when the next version was written, not at its own mtime. Today there's no field for that timestamp; the engine has to derive it by joining successive PUTs for the same key.
|
||||
|
||||
Change: at the S3 PUT handler that demotes a prior version (current → noncurrent), write `noncurrent_since` on the demoted entry **to the TsNs of the demoting event**, not local wall-clock. The demoting event is itself a meta-log PUT with a well-defined TsNs; using that value keeps `noncurrent_since` strictly monotonic in meta-log order across all replicas and immune to wall-clock skew between S3 handler nodes.
|
||||
|
||||
The lifecycle evaluator uses `ev.TsNs` for current-version rules and `entry.noncurrent_since` for noncurrent rules — both monotonic in the iteration order, so the algorithm's `done` flag still holds. `expected_mtime` (passed to `LifecycleDelete` for identity CAS) remains the entry's own mtime; CAS identity and TTL clock are kept separate.
|
||||
|
||||
### Cursor
|
||||
|
||||
Per-shard cursor is `{ TsNs int64, RuleSetHash [32]byte, PromotedHash [32]byte }`. No frozen-cursor set, no per-key freeze state — failed deletes halt the run and leave the cursor at the last successful event (see "Delete failure handling"). `RuleSetHash` and `PromotedHash` together gate recovery — mismatch on either triggers the recovery branch on the next run. See "Cursor hashes."
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| `engine/` (rule compilation, snapshot) | Keep |
|
||||
| `evaluate.go`, `due_at.go`, `rule_hash.go`, `tags.go` | Keep |
|
||||
| `reader/` (meta-log subscribe) | Keep, used once per daily pass instead of continuously |
|
||||
| `router/router.go` (rule evaluation per event) | Keep, callable in the per-event loop |
|
||||
| `router/schedule.go` (heap) | **Remove** |
|
||||
| `dispatcher/dispatcher.go` (tick, retries, freeze) | **Remove**; per-call `LifecycleDelete` moves into the daily loop |
|
||||
| `dispatcher/pipeline.go` | **Replace** with `daily_run` per shard |
|
||||
| `scheduler/scheduler.go` (long-running goroutines + refresh ticker) | **Replace** with one-shot per-shard fan-out |
|
||||
| `bootstrap/walker.go` | Keep, with shard filter added (`RunForShard`). Two invocation modes: (a) **steady-state daily** against the `walk` view (`ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, `scan_only` rules); (b) **recovery** against `engine.RecoveryView(snap)` (cold start, replay-rule edit, retention loss, partition flip). No longer feeds an in-memory schedule. |
|
||||
- **Head-of-line blocking is intentional.** A transient filer error stalls today's pass; tomorrow's run resumes at the same cursor. Loud (operator sees stuck cursor in metrics) and idempotent (identity-CAS makes redundant deletes no-ops).
|
||||
- **In-run retry with backoff** for transport errors only — default 3 attempts, exponential backoff capped at 5s. Server-side outcomes are not retried in-run.
|
||||
- **No retry queue.** Removing the per-key freeze state was the whole point; adding it back would re-introduce the state machine the redesign replaces.
|
||||
|
||||
## Rate limiting
|
||||
|
||||
Cluster-wide deletes-per-second cap, set as admin config. Admin computes per-worker share at job-dispatch time:
|
||||
Cluster-wide deletes-per-second cap, set in admin config. The admin allocator:
|
||||
|
||||
- Admin counts workers capable of `s3_lifecycle` from the registry, divides `cluster_deletes_per_second` by the count, writes the share into `ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"]`.
|
||||
- Worker reads the share and constructs one `golang.org/x/time/rate.Limiter` shared across all shard goroutines. `dispatch_delete` calls `limiter.Wait(ctx)` before each RPC.
|
||||
- Brief over/undershoot during worker join/leave is acceptable for a bulk workload.
|
||||
1. Counts workers capable of `s3_lifecycle` from the registry.
|
||||
2. Divides `cluster_deletes_per_second` by the count.
|
||||
3. Writes the per-worker share into `ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"]`.
|
||||
|
||||
No proto changes: `ClusterContext.Metadata` already exists.
|
||||
The worker reads the share and constructs one `golang.org/x/time/rate.Limiter` shared across all shard goroutines. `dispatchWithRetry` calls `limiter.Wait(ctx)` before each `LifecycleDelete` RPC.
|
||||
|
||||
## Observability
|
||||
|
||||
Per-shard Prometheus gauges (`weed/stats/metrics.go`):
|
||||
|
||||
| Metric | What it tells you |
|
||||
|---|---|
|
||||
| `s3_lifecycle_cursor_min_ts_ns{shard}` | `now - this` is the per-shard replay lag |
|
||||
| `s3_lifecycle_daily_run_last_walked_ns{shard}` | `now - this` is walker freshness; stuck = throttle misconfigured or walker failing |
|
||||
| `s3_lifecycle_daily_run_shard_duration_seconds{shard}` | Wall-clock per shard pass |
|
||||
| `s3_lifecycle_daily_run_events_scanned_total{shard}` | Counter of meta-log events drainShardEvents processed |
|
||||
| `s3_lifecycle_dispatch_limiter_wait_seconds` | Per-dispatch wait time on the cluster rate limiter |
|
||||
| `s3_lifecycle_dispatch_total{bucket,kind,outcome}` | Per-bucket dispatch counter |
|
||||
|
||||
Heartbeat log line, emitted once per `Run()`:
|
||||
|
||||
```text
|
||||
daily_run: status=ok shards=16 errors=0 duration=7s cursor_lag_max=2h walked_max_age=3m
|
||||
```
|
||||
|
||||
Tokens `status`, `shards`, `errors`, `duration` are stable for grep. `cursor_lag_max=cold` and `walked_max_age=cold` distinguish "not started yet" from "0s caught up".
|
||||
|
||||
## Data model
|
||||
|
||||
### `noncurrent_since` on version entries
|
||||
|
||||
A non-current version's TTL clock starts when the next version was written, not at its own mtime. The demoting PUT writes `NoncurrentSinceNs` on the demoted entry, set to the TsNs of the demoting meta-log event. Using the meta-log TsNs keeps `noncurrent_since` strictly monotonic in meta-log order across all replicas, immune to wall-clock skew.
|
||||
|
||||
The lifecycle evaluator uses `ev.TsNs` for current-version rules and `entry.NoncurrentSinceNs` for noncurrent rules — both monotonic in iteration order. Legacy entries with `NoncurrentSinceNs == 0` fall back to entry mtime.
|
||||
|
||||
`expected_mtime` passed to `LifecycleDelete` (for identity CAS) is always the entry's own mtime. CAS identity and TTL clock are separate concerns.
|
||||
|
||||
## Components
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `engine/` | Rule compilation, snapshot, partition views |
|
||||
| `evaluate.go`, `due_at.go`, `rule_hash.go`, `tags.go` | Engine-side rule evaluation |
|
||||
| `reader/` | Meta-log subscribe; one subscription per dailyrun.Run pass |
|
||||
| `router/router.go` | Per-event rule evaluation |
|
||||
| `bootstrap/walker.go` | Bucket walker with `RunForShard(view, shardID)` filter |
|
||||
| `dispatcher/filer_persister.go` | Filer-backed cursor I/O |
|
||||
| `dailyrun/run.go` | Main pass orchestrator: subscription, fan-out, per-shard `runShard` |
|
||||
| `dailyrun/cursor.go` | Cursor type + filer JSON serializer |
|
||||
| `dailyrun/walker_dispatcher.go` | Adapter from walker to `LifecycleDelete` RPC |
|
||||
|
||||
## Configuration
|
||||
|
||||
Admin config (`weed/worker/tasks/s3_lifecycle/`):
|
||||
|
||||
| Key | Type | Default | What |
|
||||
|---|---|---|---|
|
||||
| `cluster_deletes_per_second` | int64 | 0 (unlimited) | Cluster-wide ceiling on lifecycle delete RPCs/s. Allocated per worker. |
|
||||
| `cluster_deletes_burst` | int64 | 0 (= 2× rate) | Token-bucket burst across cluster. |
|
||||
| `meta_log_retention_days` | int64 | 0 (unbounded) | How far back the filer's meta-log can reach. Rules with TTL > retention promote to walker. |
|
||||
| `walker_interval_minutes` | int64 | 0 (fire every pass) | Minimum time between steady-state walker fires per shard. Set positive when worker runs at tighter cadence than the desired walk frequency. |
|
||||
|
||||
Worker config:
|
||||
|
||||
| Key | Default | What |
|
||||
|---|---|---|
|
||||
| `max_runtime_minutes` | 60 | Wall-clock cap per `dailyrun.Run` call |
|
||||
|
||||
## Failure & recovery
|
||||
|
||||
- **Worker crashes mid-run.** Cursor advances only past successfully-deleted events (see "Delete failure handling"); on restart the next run resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
|
||||
- **Transient delete failure.** Run halts at the failing event, cursor stays. Tomorrow's run retries from the same point. Stuck cursor is visible in metrics — operators see head-of-line blocking and address root cause.
|
||||
Four triggers route into the same **recovery branch** (cold-start, replay-rule edit, retention loss, partition flip). All four run the walker against `engine.RecoveryView(snap)` (every action force-`markActive()`'d, regardless of compile-time `IsActive`/`BootstrapComplete`) to catch already-due objects across the full rule set, then rewind the cursor to `now - MaxEffectiveTTL(replay)` (or sentinel `0` if replay is empty). The merge is by design: distinguishing the four would require independent state that isn't worth the complexity, and the walker is idempotent.
|
||||
- **Worker crashes mid-run.** Cursor advances only past successfully-deleted events. On restart, the next pass resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
|
||||
- **Transient delete failure.** Pass halts at the failing event, cursor stays. Tomorrow's pass retries from the same point. Stuck cursor is visible in `s3_lifecycle_cursor_min_ts_ns`; operators see head-of-line blocking and address the root cause.
|
||||
- **Identity drift** (object overwritten between event and delete). Handled by `LifecycleDelete` RPC's identity-CAS, which returns `NOOP_RESOLVED` for stale events. The algorithm dispatches optimistically and lets the server filter.
|
||||
- **Cold start, rule edit, partition flip.** All route into the recovery branch. The walker over `engine.RecoveryView(snap)` catches already-due objects across the full rule set, then the cursor rewinds (rule edit) or stays at the cold-start floor.
|
||||
|
||||
- **Cold start (no persisted cursor).** Recovery branch.
|
||||
- **Replay-rule edit** (new/removed/changed `ExpirationDays`, `NoncurrentDays`, or `AbortMPU` rule). Detected by `RuleSetHash` mismatch. Recovery branch.
|
||||
- **Retention loss** (`replay != nil && persisted.TsNs < earliest_available`). Worker was down longer than meta-log retention, or retention itself shrunk. Gated on `replay != nil` so the sentinel `TsNs=0` used by pure walker buckets doesn't perpetually re-trigger recovery. Recovery branch.
|
||||
- **Partition flip** (`PromotedHash` mismatch — a replay-eligible rule moved between `replay` and `walk` since last run). Covers both directions:
|
||||
- Promotion (replay → walk, retention dropped): walker takes over for that rule. Recovery fires to refresh state.
|
||||
- Demotion (walk → replay, retention recovered): persisted.TsNs is likely forward of the rule's relevant events (other replay rules kept advancing during the walk era). Recovery rewinds the cursor so the demoted rule's events get a fresh scan window.
|
||||
- **Walker-only rule edit** (added/removed/modified `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule with no replay-eligible action kind). Not a recovery trigger — both hashes unchanged, cursor untouched. Walker re-reads `walk` on every steady-state run and applies the new rule automatically.
|
||||
- **`ExpirationDate` rules.** Always routed to the walker — fixed wall-clock due time can't be derived from event TsNs, and objects created before the date may have aged out of meta-log. See "Replay vs walker."
|
||||
- **Identity drift.** Handled by `LifecycleDelete` RPC as today. No worker-side state needed.
|
||||
## Future work
|
||||
|
||||
## Comparison
|
||||
Tracked as optimizations rather than blockers:
|
||||
|
||||
| | Streaming (today) | Daily replay (this design) |
|
||||
|---|---|---|
|
||||
| Worker lifetime | Long-running, capped by `MaxRuntime` | Starts, completes pass, exits |
|
||||
| Memory | O(events with DueTime in `[now, now+TTL]`) | O(in-flight delete RPCs) |
|
||||
| Disk reads | Continuous meta-log subscribe + periodic bootstrap | One meta-log scan per day, length ≤ max_ttl |
|
||||
| Latency | DispatchTick (~1 min) | Up to 24h |
|
||||
| Future-buffering | Yes, in heap | No |
|
||||
| State persisted | Cursor + frozen set + heap (rebuilt from meta-log on restart) | Cursor only |
|
||||
| Dispatch tick / checkpoint tick / refresh tick | Required | Removed |
|
||||
| Bootstrap walker | Hot path (every run, feeds heap) | Daily for `walk` rules; recovery for full rule set |
|
||||
| Failure recovery | Heap rebuild from meta-log replay | Cursor advance |
|
||||
|
||||
## Configuration changes
|
||||
|
||||
The admin and worker forms collapse significantly. New worker config:
|
||||
|
||||
```text
|
||||
deletes_per_run_concurrency // optional, in-pass parallelism (default 1)
|
||||
max_runtime_minutes // safety cap; default 1440 (24h)
|
||||
```
|
||||
|
||||
Removed worker config (no longer meaningful):
|
||||
|
||||
```text
|
||||
dispatch_tick_minutes
|
||||
checkpoint_tick_seconds
|
||||
refresh_interval_minutes
|
||||
bootstrap_interval_minutes
|
||||
```
|
||||
|
||||
New admin config (added to the existing form):
|
||||
|
||||
```text
|
||||
cluster_deletes_per_second // 0 = unlimited
|
||||
cluster_deletes_burst // 0 = 2× rps
|
||||
```
|
||||
|
||||
Admin runtime stays at `DetectionIntervalMinutes=1440` (daily detection produces one execution per worker).
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
Sequenced so each phase compiles and ships independently. Phase 1 lands first because it's a prerequisite the algorithm relies on; phases 2–4 can be authored in parallel but should merge in order.
|
||||
|
||||
### Phase 1 — `noncurrent_since` on version entries
|
||||
|
||||
Smallest, isolated, prerequisite.
|
||||
|
||||
- Add `NoncurrentSinceNs int64` to the filer entry extended attributes (or as a top-level field if the proto allows non-breaking add).
|
||||
- At the S3 PUT handler in `weed/s3api/s3api_object_handlers_put.go` (or whichever site demotes the prior current version), set `NoncurrentSinceNs` to the **TsNs of the demoting meta-log event**, not `time.Now().UnixNano()`. Read the value from the in-flight write's TsNs (the filer assigns this) — coordinate with the filer write API to make that value available to the demotion path. If the API can't surface it cleanly, fall back to a single `time.Now().UnixNano()` captured at the start of the demoting RPC, accepting that this introduces sub-millisecond skew relative to meta-log TsNs (still monotonic per node, weakly monotonic across nodes within clock-sync tolerance).
|
||||
- Backfill: nil/zero on existing entries — the lifecycle evaluator treats `NoncurrentSinceNs == 0` as "fall back to entry mtime" so legacy data still expires, just with the old (slightly looser) semantics.
|
||||
- Unit tests in the s3api package covering PUT-over-existing and PUT-with-versioning-suspended.
|
||||
- Verify with a clock-skew test: two S3 nodes with ±5s clock drift produce a sequence of PUT/demotion events; assert `noncurrent_since` is non-decreasing in meta-log iteration order.
|
||||
|
||||
### Phase 2 — Daily-replay worker (parallel to today's, behind a flag)
|
||||
|
||||
Add the new code path without disturbing the existing one. Allows side-by-side smoke testing.
|
||||
|
||||
- New file `weed/s3api/s3lifecycle/dailyrun/dailyrun.go` with `func Run(ctx, shardID, ...) error` implementing the algorithm.
|
||||
- Reuses `engine.Snapshot`, `evaluate.go`, `due_at.go`, the existing `reader.Reader`, and the existing `LifecycleDelete` RPC. No new RPCs.
|
||||
- Per-shard cursor type `dailyrun.Cursor { TsNs int64; RuleSetHash [32]byte; PromotedHash [32]byte }`, JSON-persisted under the same filer path as today's cursor, keyed by `daily/` prefix to avoid collision. Cursor load returns `(cursor, isFresh bool)` so the algorithm can distinguish first-run from a rule-change rewrite. In Phase 2 (no walker plumbing), `PromotedHash` is always the empty hash; Phase 4 wires `engine.PromotedHash(snap, retentionWindow)` and the partition-flip recovery trigger.
|
||||
- **Replay/walker partition**: full `RulesForShard` / `RecoveryView` plumbing lands in Phase 4. For Phase 2 alone, since walker rules are rejected (next bullet), Phase 2 is **replay-only**: pass the **base snapshot** directly to `router.Route` (no view, no clone, no Mode rewrite). This is acceptable for Phase 2 because the unsupported-rule check guarantees no walker-bound actions are present, and the absence of walker rules means there's no risk of router.Route emitting walker-bound matches. Mode rewrite is also unnecessary in Phase 2: any rule that compiled to `ModeScanOnly` (legacy persistent state) falls into the unsupported-rule path along with `scan_only`-promoted rules. Phase 4 then introduces clone-per-view (including Mode rewrite) and the recovery branch.
|
||||
- **Refuse to run on unsupported rule sets.** Before invoking the replay loop, `dailyrun.Run` checks the compiled rules for any action of kind `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule promoted to `scan_only`. If any are present, the handler:
|
||||
- Logs at error level which rules are unsupported.
|
||||
- Returns a typed error from `Execute` so admin marks the run failed (visible in the activity log).
|
||||
- Does **not** advance the cursor.
|
||||
This makes flipping `algorithm: daily_replay` before Phase 4 a loud failure, not a silent dropped rule. Operators see the error and either revert to `streaming` or wait for Phase 4.
|
||||
- **Rule-change handling**: compute `rsh = engine.ReplayContentHash(snap)` at run start (partition-independent — see "Cursor hashes"). If `rsh != cursor.RuleSetHash` and `replay` is non-`nil`, persist `{ TsNs: now - MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted }` and return. If `replay` is `nil`, persist `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted }` and return. In Phase 2, `promoted` is the empty hash (no walker plumbing); Phase 4 wires real values. All cursor writes carry all three fields so Phase 4 doesn't have to migrate existing Phase 2 cursors. Tested explicitly: add a replay-eligible rule mid-test, assert cursor TsNs rewinds to `now - max_ttl` and `RuleSetHash` updates.
|
||||
- **Delete failure handling**: track `last_ok TsNs` as the algorithm progresses; only set `last_ok = ev.TsNs` after a successful or `NOOP_RESOLVED` outcome for *all* matches produced from that event. On any unresolved outcome (`RETRY_LATER`, `BLOCKED`, transport error after in-run retries), break the loop and persist `{ TsNs: last_ok, RuleSetHash: rsh, PromotedHash: promoted }`. In-run retry policy for transport errors: 3 attempts with exponential backoff capped at 5s.
|
||||
- **Match expansion**: invoke `router.Route(ctx, snap, ev, now, lister)` per event, where `snap` is the **base snapshot** directly (Phase 2 is replay-only — see "Replay/walker partition" bullet above; the view-based call site `router.Route(ctx, replay, ev, ...)` lands in Phase 4 when the partition is wired). Handles `ExpirationDays`, `NoncurrentDays`, `AbortMPU` correctly out of the box (already in `due_at.go`).
|
||||
- Rate limiter accepted as a parameter; constructed once per `Execute()` and shared across all shard goroutines.
|
||||
- New admin config flag `algorithm: streaming | daily_replay` in `weed/worker/tasks/s3_lifecycle/handler.go`. When `daily_replay`, the handler invokes `dailyrun.Run` and exits. When `streaming` (default), today's code path runs unchanged.
|
||||
- Unit tests:
|
||||
- Per-rule `done` flag correctness (single rule, multi-rule).
|
||||
- Multiple-match fan-out from one event (PUT-over-existing emits both `ExpirationDays` for the new version and `NoncurrentDays` for the displaced one).
|
||||
- Cursor advance halts at the first unresolved failure; subsequent run resumes there.
|
||||
- Identity-CAS skip path (event mtime stale → `NOOP_RESOLVED` still advances cursor).
|
||||
- Rule-set-hash mismatch rewinds cursor to `now - max_ttl` (Phase 4 adds the walker call on this path; Phase 2 just rewinds).
|
||||
- `replay` empty case: cursor persists `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: <empty in Phase 2> }`, no meta-log read attempted.
|
||||
- `NoncurrentDays` rule uses `entry.noncurrent_since`, `ExpirationDays` rule uses `ev.TsNs`; both monotonic in iteration order.
|
||||
- `AbortMPU` test: MPU init event ages past `AbortMPUDaysAfterInitiation`, delete emitted.
|
||||
- **Unsupported-rule rejection**: bucket has an `ExpirationDate` (or `ExpiredDeleteMarker` or `NewerNoncurrent`) rule and the flag is set to `daily_replay`. Assert `Execute` returns the typed unsupported-rule error, the cursor is not advanced, and the streaming code path was not invoked. Repeat for a rule promoted to `scan_only`.
|
||||
|
||||
### Phase 3 — Cluster rate limit allocation
|
||||
|
||||
Admin-side share computation. Lands once Phase 2 has a working `Limiter` parameter to receive it.
|
||||
|
||||
- `weed/worker/tasks/s3_lifecycle/handler.go`: add `cluster_deletes_per_second`, `cluster_deletes_burst` to `AdminConfigForm.Sections["scope"]`.
|
||||
- `weed/admin/plugin/plugin.go` near `RunDetectionRequest`/`ExecuteJobRequest` build sites: count capable workers, divide rps, write `s3_lifecycle.deletes_per_second` and `s3_lifecycle.deletes_burst` into `ClusterContext.Metadata`.
|
||||
- Worker `Execute`: parse metadata keys, build `rate.Limiter`, pass to `dailyrun.Run`.
|
||||
- New histogram `S3LifecycleDispatchLimiterWaitSeconds` for operator visibility.
|
||||
- Tests: admin allocation math under join/leave, worker correctly slows under low share.
|
||||
|
||||
### Phase 4 — Shard-aware walker + scan-at-date / scan-only routing
|
||||
|
||||
Reposition the walker and add shard-filtering + per-rule routing.
|
||||
|
||||
- **Shard filter on the walker.** Add `walker.RunForShard(ctx context.Context, shardID int, view *engine.Snapshot, limiter *rate.Limiter) error`. `view` is one of `walk`/`replay`/`recovery` views — its action set already filters to the right partition (via per-clone `active` flags), so the walker iterates the bucket and evaluates each entry against `view` exactly as today's `Run` does, with the only added step being `if ShardID(bucket, entry.Name) != shardID { continue }`. The existing bucket-wide `Run` becomes `RunForShard` with `shardID = -1` (any) for backward compatibility, or is deleted outright in Phase 5.
|
||||
- **Engine surface.** Four new exports on the engine package, all consumed by `daily_run`:
|
||||
- `func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)`. Returns two new `*Snapshot` instances with **cloned `*CompiledAction`** values; per-view `active` flags (and `Mode` rewrite on `replay`) determine partition membership. Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table. Either return value may be `nil` when its partition is empty. The retention window is an explicit input — `daily_run` computes it once at run start as `duration(now - metalog.EarliestAvailableTsNs(shardID))` and passes it in. The engine does no clock arithmetic of its own. The partition is stable across one `daily_run` invocation. Membership:
|
||||
- **`replay`**: cloned `ExpirationDays`, `NoncurrentDays`, `AbortMPU` actions, `active=true`, where the action's TTL ≤ `retentionWindow`.
|
||||
- **`walk`**: cloned `ExpirationDate` (`ModeScanAtDate`), `ExpiredDeleteMarker`, `NewerNoncurrent` actions, plus any of the above promoted to `scan_only` because their TTL exceeds the retention boundary. All `active=true`.
|
||||
- A single XML rule that compiles to multiple action kinds (e.g., `ExpirationDays` + `NewerNoncurrent` on the same rule) is split: each compiled action lands in `replay` or `walk` independently. `ReplayContentHash` only hashes replay-eligible rule definitions from the base snapshot, so neither the `NewerNoncurrent` action nor any walker-only edit affects the hash.
|
||||
- `func RecoveryView(s *Snapshot) *Snapshot`. Returns a third `*Snapshot` instance with every action cloned and `markActive()`'d — regardless of `Mode` or prior `BootstrapComplete` state. The recovery walker uses this view to see all replay-eligible + walker-eligible rules. Without it, recovery would skip event-driven actions the base snapshot marks inactive (`compile.go:73-74`).
|
||||
- `func ReplayContentHash(s *Snapshot) [32]byte`. Stable across reorderings, partition-independent. Used as `RuleSetHash = ReplayContentHash(snap)`. Returns the empty hash when no replay-eligible rules exist. See "Cursor hashes."
|
||||
- `func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte`. Hash of the set of replay-eligible rules currently classified as `walk` due to `scan_only` (TTL > `retentionWindow`). Detects partition flips in either direction. Takes the same `retentionWindow` value as `RulesForShard` so the two cannot disagree about partition membership. See "Cursor hashes."
|
||||
- `func MaxEffectiveTTL(s *Snapshot) time.Duration`. Max over active replay actions; returns `0` for `nil` or empty replay (caller must already be in the empty-replay branch).
|
||||
- `func CurrentSnapshot() *Snapshot`. The base snapshot before partitioning. Recovery branch passes this to `RecoveryView`.
|
||||
- **`dailyrun.Run` flow.** Cold-start, replay-rule edit (`RuleSetHash` mismatch), retention loss, and partition flip (`PromotedHash` mismatch) all route into the **recovery branch**: walker over `engine.RecoveryView(snap)` so already-due objects across all rules are caught (including event-driven actions the base snapshot may mark inactive), then cursor rewinds. Steady-state path runs walker over `walk` only, then replay over `replay`. Cursor stores `{ TsNs, RuleSetHash, PromotedHash }`; steady-state writes both hashes through unchanged.
|
||||
- **Recovery branch.** Triggered by any of: no persisted cursor, `RuleSetHash` mismatch, `PromotedHash` mismatch, or `replay != nil && persisted.TsNs < earliest_available`. The retention-loss trigger uses the **absolute** `earliest_available` TsNs (the oldest available meta-log TsNs from `metalog.EarliestAvailableTsNs(shardID)`), not the `retentionWindow` duration that `RulesForShard`/`PromotedHash` consume — they're different shapes for different purposes: the retention check on the cursor compares two absolute timestamps, while the engine partition check compares a per-action TTL to a duration. Gated on `replay != nil` because pure walker buckets persist a sentinel `TsNs=0` that would otherwise re-fire recovery every run. Invokes `walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)`. Persists cursor at:
|
||||
- `{ TsNs: now - engine.MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted }` if `replay` is non-`nil`.
|
||||
- `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted }` if `replay` is `nil` (pure walker bucket).
|
||||
`MaxEffectiveTTL` is undefined over an empty set, so the sentinel form is required when `replay` is `nil`. **Both hashes are written on every recovery persistence** — failing to update `PromotedHash` here would cause the partition-flip trigger to re-fire on every subsequent run. See "Bootstrap retention window."
|
||||
- **Steady-state walker invocation.** Distinct from recovery: `walker.RunForShard(ctx, shardID, walk, limiter)`. `walk`'s cloned actions are masked to walker-bound kinds only, so the walker doesn't re-process replay rules that the meta-log scan already handles on the same run.
|
||||
- Drop the bootstrap walker's call from `scheduler.go` (removed entirely in Phase 5).
|
||||
- Tests:
|
||||
- Cold start with mixed walker-only + replay rules in same bucket: recovery branch fires once (walker over `engine.RecoveryView(snap)`, then cursor rewind). Next day's run is steady-state (walker over `walk` only, replay over `replay`).
|
||||
- `ExpirationDate` rule fires only on/after the date (walker path).
|
||||
- `ExpiredDeleteMarker` rule deletes orphaned markers (walker path, sibling-aware).
|
||||
- `NewerNoncurrent` count rule deletes oldest noncurrent versions beyond the cap (walker path, version-list-aware).
|
||||
- Bucket with only walker rules (e.g., pure `ExpirationDate`): `replay` is `nil`, cursor persisted with sentinel `TsNs=0`, no meta-log subscribe attempted. Critically — assert that across 5 consecutive daily runs, recovery fires exactly once (on the first run); subsequent runs see no recovery despite `persisted.TsNs (0) < earliest_available`, because the retention-loss check is gated on `replay != nil`.
|
||||
- Replay-rule edit triggers recovery: walker runs over `RecoveryView(snap)` (catches an object already 60d old under a brand-new 30d rule), then cursor rewinds. Object aged 20d at edit time is later deleted at age 30 via the replay path.
|
||||
- `RecoveryView` activates event-driven actions: simulate a base snapshot where an `ExpirationDays` action has `IsActive=false` (e.g., `BootstrapComplete=false`); assert the recovery walker still sees and evaluates the rule via the recovery view, while the steady-state walker over `walk` does not.
|
||||
- **Replay view rewrites Mode**: simulate a base snapshot with a `ModeScanOnly`-locked `ExpirationDays` action (carried over from `prior.Mode`); assert that `replay`'s clone of the action has `Mode == ModeEventDriven`, and that `router.Route(ctx, replay, ev, ...)` emits a Match for it. The test should fail if Mode rewrite is skipped — confirming the rehabilitation path is wired.
|
||||
- Cursor persistence after recovery: assert `PromotedHash` is written through on every recovery branch persistence; simulate a partition flip across two runs and verify the second run does not re-fire recovery because the prior `promoted_hash` was correctly stored.
|
||||
- Adding a walker-only rule to an existing replay-only bucket: `ReplayContentHash(snap)` is unchanged, replay cursor TsNs unchanged, walker picks up the new rule on its next steady-state pass.
|
||||
- `scan_only` promotion (retention drops, replay-eligible rule moves to `walk`): `ReplayContentHash` unchanged, **`PromotedHash` changes** → recovery branch fires. Walker over base catches in-flight objects under the promoted rule; cursor rewinds; `PromotedHash` is persisted.
|
||||
- Retention recovery (`scan_only` rule demotes back to `replay` while other replay rules' cursor has advanced past the rule's relevant events): `ReplayContentHash` unchanged, `persisted.TsNs > earliest_available` so retention-loss check does NOT fire, **`PromotedHash` changes** → recovery branch fires. Asserts the demoted rule's older events get processed (this is the test that would have failed before adding `PromotedHash`).
|
||||
- `PromotedHash` persistence: simulate a partition flip in either direction across two runs, assert cursor's `promoted_hash` field carries through and gates the next run's recovery decision correctly.
|
||||
- Shard filter: 2 shards × 1 bucket walker run leaves no double-deletes (assert via RPC counter).
|
||||
- `scan_only` promotion: rule with TTL > meta-log retention moves from `replay` to `walk` at run start.
|
||||
|
||||
### Phase 5 — Switch default and delete streaming path
|
||||
|
||||
Cut over and clean up. Should be a single PR that's mostly deletions.
|
||||
|
||||
- Default `algorithm: daily_replay` and remove the streaming alternative from the descriptor.
|
||||
- Delete:
|
||||
- `weed/s3api/s3lifecycle/router/schedule.go` (+ tests)
|
||||
- `weed/s3api/s3lifecycle/dispatcher/dispatcher.go` (+ tests), `freeze`-related code, retry state
|
||||
- `weed/s3api/s3lifecycle/dispatcher/pipeline.go`
|
||||
- `weed/s3api/s3lifecycle/scheduler/scheduler.go` (+ tests), refresh ticker
|
||||
- Removed worker config fields and their `ParseConfig` handling
|
||||
- Rewrite `weed/s3api/s3lifecycle/DESIGN.md` to remove "streaming" sections.
|
||||
- One full end-to-end test on a bucket with 1M objects, mixed-TTL rules, with cluster rate cap engaged.
|
||||
|
||||
### Phase 6 — Observability polish
|
||||
|
||||
After cutover, before announcing.
|
||||
|
||||
- Daily-run summary log line: shard, rules-touched, events-scanned, deletes-issued, walker-fallback (y/n), duration.
|
||||
- Prometheus: `S3LifecycleDailyRunDurationSeconds`, `S3LifecycleDailyRunEventsScanned`, `S3LifecycleDailyRunDeletesIssued`, all labelled by `shard_id` and `outcome`.
|
||||
- Admin UI: a single "Last Run" card per bucket showing the above summary line. Removes the now-meaningless "Next Run / Detection Results" widgets we already hid for `s3_lifecycle`.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **In-pass concurrency.** Today's worker has 1–16 shard goroutines via `workers` config. Daily replay can either (a) keep that knob, fanning out shard scans across goroutines that share the limiter, or (b) drop it since the limiter is now the throughput governor. Lean toward (a) — shards are independent meta-log subscriptions and parallel reads cut wall-clock significantly.
|
||||
2. **Cursor format compatibility.** Worth deciding whether the new cursor coexists at a separate filer path (clean cutover, no migration) or overwrites the old one (single source of truth, requires a one-time read-then-rewrite during cutover). Lean toward separate paths during phases 2–4, unified at Phase 5.
|
||||
3. **Per-shard rule snapshots vs. global.** Engine snapshots are bucket-scoped today; Phase 4 adds `Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)`. Open: should `RulesForShard` also gate by which buckets a shard "owns" (every bucket is on every shard via `ShardID(bucket, key)`), or just return the global rule set partitioned by mode? Current lean: return global partition, since every shard worker runs against every bucket and the actual shard filter happens at the entry-iteration site (meta-log subscription and walker entry loop).
|
||||
6. **Bucket-coordinated walker.** Phase 4 has each shard walk the full bucket and filter by `ShardID` — simple, but 16× the listing cost. A future optimization is a per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards via the existing injection point). Worth doing if listing cost becomes the bottleneck for very large buckets. Not blocking initial cutover.
|
||||
4. **`noncurrent_since` source from filer write API.** Phase 1 prefers the demoting event's TsNs over wall-clock. Whether the existing filer write path surfaces that TsNs to the S3 handler without a proto change is a Phase 1 design decision; the fallback (handler-local `time.Now()`) is documented but loses cross-node monotonicity guarantees and should be a temporary measure.
|
||||
5. **Retry budget for `RETRY_LATER`.** The current dispatcher tracks per-key retry counts; this design removes that state and lets head-of-line blocking surface the issue. Worth confirming that operators are OK with a stuck cursor as the failure signal vs. silent retry. If not, add a bounded retry counter on the cursor (one per stuck event) — small state, but inches back toward the model we're replacing.
|
||||
1. **Long-lived subscription across passes.** Today the subscription is rebuilt per `Run()`. Keeping it alive across passes would eliminate the 7s ctx-timeout per pass and the start/teardown overhead. Requires per-shard pending heap (events whose `DueTime > runNow` would be parked in-memory instead of replayed) and a hot-swappable snapshot for mid-pass config changes. Multi-day refactor; current model works.
|
||||
2. **Bucket-coordinated walker.** Phase 4 has each shard walk the full bucket and filter by `ShardID(bucket, key)` — simple but 16× the listing cost. A per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards) would cut listing cost. Worth doing if listing becomes the bottleneck for very large buckets.
|
||||
3. **Per-bucket dispatch lag metric.** Currently only per-shard lag is exposed. Per-bucket would require a per-bucket cursor or a derived metric from `s3_lifecycle_dispatch_total{bucket,kind,outcome}`. Punted on cardinality concerns; revisit when an operator asks for it.
|
||||
4. **Meta-log retention plumbing.** If the filer adds GC for `/topics/.system/log`, the `PromotedHash` partition flip needs to consume the filer's actual retention horizon (currently dormant because retention is effectively infinite).
|
||||
|
||||
Loading…
Reference in New Issue
Block a user
