feat(s3/lifecycle): daily-replay worker behind algorithm flag (Phase 2) (#9446)
* docs(s3lifecycle): design for daily-replay worker Captures the algorithm and dev plan iterated on in PR #9431 and the discussion leading up to it: per-shard daily meta-log replay, walker as a per-day pass for ExpirationDate/ExpiredDeleteMarker/NewerNoncurrent plus a recovery branch over engine.RecoveryView(snap), explicit retention-window input to RulesForShard, two cursor hashes (ReplayContentHash + PromotedHash) that together detect every invalidation case. Implementation phases are sequenced so each can ship independently — Phase 1 (noncurrent_since stamp) just landed. * feat(s3/lifecycle): daily-replay worker behind algorithm flag (Phase 2) New weed/s3api/s3lifecycle/dailyrun package implementing the bounded daily meta-log scan from the design doc. One pass per Execute per shard: load cursor, scan events forward, route each through router.Route, dispatch any due Match, advance the cursor on success. Halt-on-failure keeps the cursor at the last fully-processed event so tomorrow resumes from the same point — head-of-line blocking is the deliberate failure signal. Replay-only in this phase. Phase 4 wires the walker for ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, and scan_only-promoted rules. Until then a typed UnsupportedRuleError refuses runs on those buckets: operators see the rejection in the activity log rather than silently losing rules. Behavior: - Per-shard cursor {TsNs, RuleSetHash, PromotedHash} JSON-persisted under /etc/s3/lifecycle/daily-cursors/. PromotedHash always-empty in Phase 2; Phase 4 turns it on. - Rule-change branch rewinds cursor to now - max_ttl when the replay-content hash mismatches. Cold start uses the same floor. - Transport errors retry 3x with exponential backoff capped at 5s; server outcomes (RETRY_LATER / BLOCKED) halt the run without retry. - Empty-replay sentinel: cursor TsNs=0 when no replay-eligible rules exist, only the hash gates a future addition. Worker shape: - New admin config field "algorithm" with enum streaming|daily_replay, default streaming. Existing deployments are unaffected. - handler.Execute branches on the flag: streaming routes through the current scheduler.Scheduler, daily_replay routes through dailyrun.Run. - dispatcher.NewFilerSiblingLister exported so both paths share the same .versions/ + null-bare lookup. Engine integration: - Local replayContentHash + maxEffectiveTTL helpers in dailyrun. Phase 4's engine surface (ReplayContentHash, MaxEffectiveTTL) will replace them with one-line redirects; the local versions hash the same fields so the cursor stays valid across the swap. Tests cover cursor persistence, unsupported-rule rejection, hash stability under rule reordering, hash sensitivity to TTL edits, max-TTL aggregation, dispatch retry budget, and request shape including the identity-CAS witness. Includes the design doc at weed/s3api/s3lifecycle/DESIGN.md so reviewers and future phases share the same spec. * feat(s3/lifecycle): default to daily_replay; streaming becomes the fallback knob The streaming dispatcher hasn't shipped to users yet, so there's no backward-compat surface to preserve. Flip the algorithm default from streaming to daily_replay so the new path is the standard from day one. Streaming stays as an explicit opt-in escape hatch during the Phase 4 walker rollout; Phase 5 deletes both the flag and the streaming code. Buckets whose lifecycle rules require walker-bound dispatch (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, scan_only) will fail the daily_replay run with the existing UnsupportedRuleError until Phase 4 walker integration ships. Operators hitting that case can set algorithm=streaming until the follow-up lands. Updates the test for the default value and renames the unknown-value-fallback case to reflect the new default. * fix(s3/lifecycle/dailyrun): drop per-rule done flag — it suppressed due matches The done map was keyed by ActionKey = {Bucket, RuleHash, ActionKind}. That's only safe when each event produces at most one match per ActionKey with a single deterministic due-time formula — ExpirationDays and AbortMPU fit that shape because due_time = ev.TsNs + r.days is monotonic in event TsNs. But NoncurrentDays paired with NewerNoncurrentVersions > 0 (allowed in Phase 2 since it compiles to ActionKindNoncurrentDays) routes through routePointerTransitionExpand, which emits matches for every noncurrent sibling — each with its own SuccessorModTime taken from the demoting event for that specific sibling. A single event can therefore produce two matches for the same ActionKey on different objects with wildly different DueTimes. With the old code, a not-yet-due sibling encountered first would set done[ActionKey] = true and then the next sibling — even though its DueTime had already passed — would be skipped. Future events for the same rule would also be suppressed for the rest of the run. Objects that should have been deleted weren't. Fix: drop the early-stop optimization. Process every match independently. A future-DueTime match is now silently skipped without affecting any later match. The performance hit is small (Phase 2 is a single bounded daily pass, and the rate limiter is the real throughput governor); the correctness gain is non-negotiable. Also fixes the inverted comment in processMatches that described the old check as "due_time is past now" when it actually checked DueTime.After(now) (i.e., NOT yet due). Adds four targeted tests: - not-yet-due match first in slice does not suppress two later due matches for the same rule; - reversed slice ordering produces identical dispatch; - BLOCKED outcome halts the loop before later due matches are sent; - empty match slice is a no-op. Phase 4's walker-and-recovery integration can revisit a per-(rule, object) memoization if profiling argues for it. * fix(s3/lifecycle/dailyrun): address PR review — cursor advance, mode gate, ctx cancel, snapshot consistency Addresses PR #9446 review feedback. Eight distinct fixes: 1. CURSOR ADVANCEMENT (gemini, critical). The old code advanced the persisted cursor to lastOK = TsNs of the last event processed, including events whose matches were skipped as not-yet-due. Those skipped matches would never be re-scanned, so objects under long-TTL rules would never expire. Track a "stuck" flag in drainShardEvents: the first event with a skipped (future-DueTime) match stops cursorAdvanceTo from rising, but the loop keeps processing later events to dispatch any that ARE due. The persisted cursor sits at the last fully-processed event so tomorrow's run re-scans from the skipped event onward and the future-due matches get re-evaluated when they age in. processMatches now returns (skippedAny, halted, err) so the drain loop can tell apart "event fully drained" from "event had pending future-due matches." 2. MODE GATE (gemini). checkSnapshotForUnsupported only checked the ActionKind. A replay-eligible kind with Mode != ModeEventDriven (e.g. ModeScanOnly via retention promotion) passed the check but then got silently ignored by router.Route, which gates dispatch on Mode == ModeEventDriven. Reject loudly with the typed error so admin sees the rejection in the activity log. 3. WORKERS CONFIG (gemini). The handler hardcoded 16 concurrent shard goroutines regardless of cfg.Workers. Add a Workers field to dailyrun.Config and gate the goroutine fan-out on a semaphore of that size; the handler now passes cfg.Workers through. 4. SINGLE SNAPSHOT PER RUN (coderabbit). Run() validated against one snapshot but runShard() pulled a fresh cfg.Engine.Snapshot() per shard. Mid-run Compile would let shards process different rule sets. Capture snap at the top of Run, pass it down to every shard. 5. FROZEN runNow (coderabbit). drainShardEvents and processMatches accepted a `now func() time.Time` and called it multiple times. DueTime comparisons would slip as the run wore on. Capture runNow once at the top of Run and thread it through as a time.Time value. 6. CTX CANCELLATION (coderabbit). The drain loop's <-ctx.Done() case broke out of the loop and returned nil, marking interrupted runs as successful. Return ctx.Err() instead so the caller propagates the interrupt; cursorAdvanceTo carries whatever progress was made. 7. CURSOR LOAD VALIDATION (coderabbit + gemini). The persister silently accepted empty files, mismatched shard_ids, and hash slices shorter than 32 bytes (copy() would zero-pad). Each now returns a typed error so the run halts and an operator investigates rather than silently re-scanning from time zero or persisting a zero-padded hash that masks corruption forever. 8. DEAD BRANCH (coderabbit). The "lastOK < startTsNs → keep persisted" guard in runShard was unreachable because drainShardEvents initialized lastOK := startTsNs and only ever raised it. Removed along with the new cursor-advancement semantics that handle the "no events processed" case implicitly. Plus markdown lint: DESIGN.md fenced code blocks now carry a `text` language identifier to satisfy MD040. Skipped from the review: - gemini's "maxTTL == 0 incorrectly skips immediate expirations": actions with Days <= 0 don't compile to a CompiledAction (see weed/s3api/s3lifecycle/action_kind.go: `if rule.X > 0`). The new empty-replay sentinel uses `rsh == [32]byte{}` for clarity per gemini's suggested form, but the behavior is equivalent. Tests added/updated: - TestProcessMatches_AllDueNoSkippedFlag pins skippedAny=false when all matches are past their DueTime. - TestCheckSnapshotForUnsupported_NonEventDrivenModeRejected pins the new Mode check. - TestFilerCursorPersister_EmptyFileReturnsError, _ShardIDMismatchReturnsError, _HashLengthMismatchReturnsError pin the new validation rules. - Existing process-matches tests reshaped for the (skippedAny, halted, err) return tuple. Full build clean. Dailyrun + worker test packages green.
This commit is contained in:
Chris Lu
GitHub
parent
b2d24dd54f
commit
122ca7c020
492
weed/s3api/s3lifecycle/DESIGN.md
Normal file
492
weed/s3api/s3lifecycle/DESIGN.md
Normal file
@ -0,0 +1,492 @@
|
||||
# S3 Lifecycle (Design)
|
||||
|
||||
Replaces the current streaming + heap design with a daily meta-log replay. The worker becomes "start, do today's work, stop" — matching the operator's mental model and removing the per-shard heap that grows O(events × TTL_days).
|
||||
|
||||
## Goal
|
||||
|
||||
For each bucket lifecycle rule with TTL `D` days, delete every object whose age exceeds `D`, exactly once per day, with the worker exiting when the pass completes. No future-buffered Matches in memory. No tick-driven dispatch loop. Cluster-wide delete rate cap allocated per worker.
|
||||
|
||||
## Algorithm
|
||||
|
||||
One pass per day, per shard:
|
||||
|
||||
```text
|
||||
daily_run(shardID):
|
||||
persisted = load_cursor(shardID) // { TsNs, rule_set_hash, promoted_hash }
|
||||
earliest_available = metalog.EarliestAvailableTsNs(shardID)
|
||||
retention_window = duration(now - earliest_available) // time.Duration; sole window input to engine
|
||||
snap = engine.CurrentSnapshot() // base; per-action active/Mode reflect compile-time state, not partition
|
||||
replay, walk = snap.RulesForShard(shardID, retention_window)
|
||||
if replay == nil and walk == nil: return // bucket has no rules
|
||||
rsh = engine.ReplayContentHash(snap) // content over ExpirationDays/NoncurrentDays/AbortMPU rules; partition-independent
|
||||
promoted = engine.PromotedHash(snap, retention_window) // rules currently in walk due to scan_only
|
||||
|
||||
needs_recovery_walk = (
|
||||
persisted is fresh // cold start
|
||||
or persisted.rule_set_hash != rsh // replay-rule edit
|
||||
or persisted.promoted_hash != promoted // partition flip in either direction
|
||||
or (replay != nil and persisted.TsNs < earliest_available) // retention loss; meaningless if pure walker bucket
|
||||
)
|
||||
|
||||
if needs_recovery_walk:
|
||||
// Catch already-due objects across ALL rules — both walker-bound and
|
||||
// replay-bound. Replay-bound rules may have objects whose mtime is
|
||||
// older than (now - max_ttl) that the steady-state replay window cannot
|
||||
// reach. The recovery view force-activates every action so the walker
|
||||
// doesn't skip event-driven rules that the base snapshot has marked
|
||||
// inactive (e.g., pre-BootstrapComplete).
|
||||
walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)
|
||||
if replay == nil:
|
||||
save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
else:
|
||||
save_cursor(shardID, { TsNs: now - engine.MaxEffectiveTTL(replay), rule_set_hash: rsh, promoted_hash: promoted })
|
||||
return
|
||||
|
||||
// --- Steady-state path. ----------------------------------------------
|
||||
if walk != nil:
|
||||
walker.RunForShard(ctx, shardID, walk, limiter) // walker-bound actions only
|
||||
|
||||
if replay == nil:
|
||||
save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
return
|
||||
|
||||
max_ttl = engine.MaxEffectiveTTL(replay)
|
||||
cursor = max(persisted.TsNs, now - max_ttl)
|
||||
done = {r: false for r in replay.activeRules()}
|
||||
last_ok = persisted.TsNs
|
||||
|
||||
for ev in metalog.from(shardID, cursor): // ascending TsNs
|
||||
if all(done.values()): break
|
||||
halt = false
|
||||
matches = router.Route(ctx, replay, ev, now, lister) // existing signature; replay is *engine.Snapshot
|
||||
for m in matches:
|
||||
r = replay.lookup(m.RuleHash)
|
||||
if r == nil or done[r]: continue
|
||||
if m.DueTime > now:
|
||||
done[r] = true // monotonic for replay kinds
|
||||
continue
|
||||
limiter.wait()
|
||||
outcome = dispatch_delete(m.Bucket, m.ObjectKey, m.VersionID,
|
||||
expected_mtime=m.Identity.mtime, ruleHash=m.RuleHash)
|
||||
if outcome is unresolved (RETRY_LATER, transport error after in-run retries, BLOCKED):
|
||||
halt = true; break
|
||||
if halt: break
|
||||
last_ok = ev.TsNs
|
||||
|
||||
save_cursor(shardID, { TsNs: last_ok, rule_set_hash: rsh, promoted_hash: promoted })
|
||||
```
|
||||
|
||||
Two distinct walker invocations, deliberately:
|
||||
|
||||
| Branch | Walker input | Why |
|
||||
|---|---|---|
|
||||
| Recovery (cold-start, replay-rule edit, retention loss, partition flip) | `engine.RecoveryView(snap)` — every action clone force-`markActive()`'d, `Mode` preserved | Catch already-due objects across all rules — including replay-bound rules whose subjects are older than `max_ttl` (replay scan window cannot reach), *and* event-driven rules the base snapshot may mark inactive (`compile.go:73-74` requires `BootstrapComplete`). |
|
||||
| Steady-state | `walk` (walker-only actions) | Replay handles its own rules via the meta-log scan. Walker only runs for actions that have no replay path. |
|
||||
|
||||
`router.Route` and `walker.RunForShard` keep their existing `*engine.Snapshot` parameter; `replay`, `walk`, and `snap` are all `*Snapshot` values.
|
||||
|
||||
The single API surface is:
|
||||
|
||||
```go
|
||||
// In engine:
|
||||
func CurrentSnapshot() *Snapshot // base
|
||||
func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
|
||||
func RecoveryView(s *Snapshot) *Snapshot // every action force-activated; recovery walker only
|
||||
func ReplayContentHash(s *Snapshot) [32]byte // content over replay-eligible rules; partition-independent
|
||||
func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte // hash of replay-eligible rules currently in walk (scan_only)
|
||||
func MaxEffectiveTTL(s *Snapshot) time.Duration // max over active replay actions
|
||||
```
|
||||
|
||||
`RulesForShard` and `RecoveryView` both return new `*Snapshot` instances with cloned `*CompiledAction` objects and shared (by pointer) rule definitions. The fields that differ from the base are:
|
||||
|
||||
- **`active`** (per clone) — set per view.
|
||||
- **`Mode`** (per clone) — rewritten to `ModeEventDriven` on `replay` clones; preserved as-is on `walk` and `recovery` clones. The rewrite is required because `router.Route` gates on `Mode == ModeEventDriven` (`router.go:134, 253, 413, 479, 566, 623, 844`), and today's compile preserves a persistent `prior.Mode = ModeScanOnly` that would otherwise lock the action out of replay even after retention rehabilitates it.
|
||||
- **Action-map membership** — `replay`'s map contains only replay-eligible clones; `walk`'s contains only walker-bound clones; `recovery`'s contains every action.
|
||||
|
||||
Per-view settings are summarized in the table under "`router.Route` integration."
|
||||
|
||||
`router.Route` is the existing entry point — it computes `ComputeDueAt`, applies predicate filters, expands one event into 0+ `Match`es (e.g., a demoting PUT produces one for the new version's lifecycle and another for the displaced version's `NoncurrentDays`). The replay path delegates due-time math to the engine so all replay-eligible action kinds (`ExpirationDays`, `NoncurrentDays`, `AbortMPU`) are handled uniformly.
|
||||
|
||||
### `router.Route` integration
|
||||
|
||||
Today's `router.Route(ctx, snap *engine.Snapshot, ev, now, lister)` iterates every action with `IsActive() == true` in the snapshot. `IsActive` is **instance-level state on `*CompiledAction`** (`engine/compile.go:73-85`, `ca.markActive()`), so two snapshots can't disagree on activation if they share the same `*CompiledAction` pointers.
|
||||
|
||||
**Approach: views are independent `*Snapshot` instances with cloned `CompiledAction` objects.**
|
||||
|
||||
Two gating fields are instance-level on `*CompiledAction` and must both match what the downstream call site expects:
|
||||
|
||||
- `active` (`engine/compile.go:73-85`) — gates routing and walker iteration.
|
||||
- `Mode` — `router.Route` requires `Mode == engine.ModeEventDriven` at every emit site (`router.go:134, 253, 413, 479, 566, 623, 844`); the walker rejects only `Mode == engine.ModeDisabled` (`walker.go:147`). Today's compile preserves `prior.Mode`, so a rule previously promoted to `ModeScanOnly` will still be skipped by `router.Route` even if `active=true`.
|
||||
|
||||
`engine.Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)` returns two new `*Snapshot` instances. Each has its own `actions map[ActionKey]*CompiledAction` containing *clones* of the originals with **both `active` and `Mode` rewritten** per view. The partition predicate is "TTL ≤ retentionWindow" for replay membership; the engine does no clock arithmetic of its own — `retentionWindow` is the only window input it receives:
|
||||
|
||||
| View | Clone settings | Why |
|
||||
|---|---|---|
|
||||
| `replay` | `active = true`, `Mode = ModeEventDriven` | `router.Route` requires `ModeEventDriven`. Forced regardless of `prior.Mode`, so a rule previously stuck in `ModeScanOnly` is rehabilitated when retention now fits its TTL. |
|
||||
| `walk` | `active = true`, `Mode` preserved from original (any non-`ModeDisabled` is accepted by the walker) | Walker doesn't gate on a specific Mode value, so no rewrite needed beyond ensuring `active`. |
|
||||
| `recovery` (see below) | `active = true`, `Mode` preserved | Recovery walker iterates all action clones; preserving `Mode` lets each rule's evaluator do its normal predicate work. |
|
||||
|
||||
Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table, and other immutable data — memory overhead is one `CompiledAction` struct per action per view (tens of bytes × dozens of actions, not megabytes).
|
||||
|
||||
Membership:
|
||||
|
||||
- `replay`: clones of `ExpirationDays`, `NoncurrentDays`, `AbortMPU` actions whose TTL ≤ `retentionWindow`. The Mode rewrite is what makes "rehabilitation" work — retention recovers, a previously `ModeScanOnly`-stuck rule clone now reads as `ModeEventDriven` in the replay view.
|
||||
- `walk`: clones of `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, plus `scan_only`-promoted clones whose TTL exceeds `retentionWindow`.
|
||||
- Either return value may be `nil` if its partition is empty.
|
||||
|
||||
The daily replay loop calls `router.Route(ctx, replay, ev, now, lister)`. Steady-state walker calls `walker.RunForShard(ctx, shardID, walk, limiter)`. Both call sites are drop-in — they see the correct, per-partition set with the correct `(active, Mode)` for their gating logic.
|
||||
|
||||
**Recovery view (separate).** The recovery branch needs the walker to see **all** replay-eligible + walker-eligible actions as active, including event-driven actions that the base snapshot may have marked inactive (`compile.go:73-74`: event-driven actions require `BootstrapComplete`). Add:
|
||||
|
||||
```go
|
||||
func RecoveryView(s *Snapshot) *Snapshot
|
||||
```
|
||||
|
||||
`RecoveryView` is a third `*Snapshot` instance whose action clones are all `markActive()`'d, with `Mode` preserved from the original. The walker accepts any non-`ModeDisabled` Mode, so no rewrite is needed. The recovery branch calls `walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)`. Without this, recovery would silently skip any rule the base snapshot considers inactive — defeating the recovery branch's purpose of catching already-due objects across the full rule set.
|
||||
|
||||
`RecoveryView` does not participate in steady-state. It exists only for the recovery walker invocation.
|
||||
|
||||
**`scan_only` input path.** The retention boundary used for promotion is supplied explicitly via `retentionWindow time.Duration`, not derived inside the engine. `daily_run` computes it once at run start (`retention_window = duration(now - metalog.EarliestAvailableTsNs(shardID))`) and passes it to `RulesForShard` and `PromotedHash`. The promotion decision is therefore stable across one `daily_run` invocation — a rule that flips between `replay` and `walk` mid-run does not re-route; it gets its new bucket on the next day's `RulesForShard` call. `RecoveryView` takes only the base snapshot — it activates everything, partition-independent.
|
||||
|
||||
This approach keeps `router.Route` and `walker.Run` signatures unchanged. The trade-off is cloning `CompiledAction` per view (and rewriting `Mode` in the replay view); the alternative (adding a filter parameter to `Route` or restructuring `IsActive`/`Mode` to be view-scoped) would force every call site — including the streaming dispatcher during Phase 2/3 — to thread the filter, while clone-per-view localizes the partition decision to the engine. The Mode rewrite also bypasses today's persistent `prior.Mode = ModeScanOnly` lock-in, which is now obsoleted by the new partition logic.
|
||||
|
||||
Why correct:
|
||||
|
||||
- Meta-log is TsNs-ordered. For any rule `R`, `due_time` is also monotonic in TsNs (see "Due-time source"), so the first not-yet-due event for `R` proves every later event is also not-yet-due. Hence the `done` flag.
|
||||
- Multiple rules fan out on the same scan: one meta-log read, N predicate checks per event.
|
||||
- Identity drift (object overwritten between event and delete) is caught at the RPC by the existing `LifecycleDelete` identity-CAS, which returns `NOOP_RESOLVED` for stale events. The algorithm dispatches optimistically and lets the server filter.
|
||||
|
||||
### Action kinds and dispatch paths
|
||||
|
||||
The engine defines six action kinds (`weed/s3api/s3lifecycle/action_kind.go`). Each has a distinct trigger and due-time semantics, and they split into two dispatch paths. Replay path uses meta-log iteration with the `done` early-stop; walker path lists current bucket state.
|
||||
|
||||
| ActionKind | Trigger | Due time | Path | Early-stop? |
|
||||
|---|---|---|---|---|
|
||||
| `ExpirationDays` | Latest-version PUT | `ev.TsNs + r.ExpirationDays` | Replay | Yes — `due` strictly increasing in iteration order |
|
||||
| `NoncurrentDays` | Demotion (next PUT for same key) | `entry.noncurrent_since + r.NoncurrentDays` | Replay | Yes — `noncurrent_since` is the demoting event's TsNs (see "`noncurrent_since`" below), monotonic |
|
||||
| `AbortMPU` | MPU init | `mpu_init.TsNs + r.AbortMPUDaysAfterInitiation` | Replay | Yes — same monotonicity as `ExpirationDays` |
|
||||
| `ExpirationDate` | Latest-version PUT, fired on `now >= r.ExpirationDate` | `r.ExpirationDate` (constant) | Walker | n/a — fixed wall-clock date, fires once per object |
|
||||
| `ExpiredDeleteMarker` | Delete marker with `NumVersions == 1` (no remaining siblings) | "now if orphaned, else never" — depends on current bucket state, not event age | Walker | n/a — requires sibling lookup, not derivable from event TsNs |
|
||||
| `NewerNoncurrent` | Version becomes noncurrent AND total noncurrents > `r.NewerNoncurrentVersions` | "now if over the count cap, else never" — count is current state | Walker | n/a — requires version-list lookup |
|
||||
|
||||
Why `ExpiredDeleteMarker` and `NewerNoncurrent` are walker-only:
|
||||
|
||||
Both rules' "due time" depends on the *current* state of sibling versions, not on any single event's timestamp. Modeling them as replay would force a sibling-list lookup on every meta-log event for the matching keys (today's `router.Route` does exactly this), and the `done` early-stop would never engage because the trigger condition isn't event-time-monotonic. They fit the walker far better: list each key's versions once per day, evaluate the rule against current state, emit deletes. Today's `bootstrap/walker.go` already handles them via `EvaluateAction`.
|
||||
|
||||
`expected_mtime` passed to `LifecycleDelete` is always the entry's original mtime (used for identity CAS), regardless of action kind. The TTL clock and the CAS identity are separate concerns.
|
||||
|
||||
### Rule changes
|
||||
|
||||
Rule changes invalidate the persisted cursor's assumption that "everything older has already been processed under the same rules." The algorithm tracks `rule_set_hash` alongside the cursor and forces a bootstrap walk on mismatch. Cases this covers:
|
||||
|
||||
- New rule added → walker discovers existing objects under the new rule.
|
||||
- TTL increased (e.g. 30d → 60d) → walker discovers objects in `[now-60d, now-30d]` that were skipped while the old rule ran.
|
||||
- Predicate widened (prefix/tag filter change) → walker re-evaluates everything.
|
||||
- TTL decreased / predicate narrowed → walker still runs (harmless re-evaluation), but the next daily run picks up steady-state quickly.
|
||||
|
||||
Bootstrap walker on rule change is a one-shot full bucket list. Cost is proportional to bucket size, not history; acceptable as an infrequent event.
|
||||
|
||||
### Bootstrap retention window
|
||||
|
||||
The walker only deletes objects whose age already exceeds their TTL. Not-yet-due objects are intentionally left for the meta-log path. If the post-walker cursor were set to `now`, the next daily replay would scan from `now` forward and never see the PUT event for, say, a 20-day-old object under a 30-day rule — that event sits at `TsNs ≈ now - 20d`, before the cursor.
|
||||
|
||||
The cursor is therefore reset to `now - max_ttl` (excluding `ScanAtDate` rules from `max_ttl`, since those don't use the replay path). The clamp `max(persisted, now - max_ttl)` in the steady-state algorithm preserves the sliding window: each day the effective scan range is `[now - max_ttl, now]`, and not-yet-due events surface naturally as they age in.
|
||||
|
||||
Invariant this depends on: **meta-log retention ≥ max_ttl**. If retention is shorter, events for some still-pending objects age out before they become due. Detected at run start by comparing the meta-log's earliest available TsNs against `now - max_ttl`; if the gap is non-empty, the affected rules are flagged `scan_only` for that run and routed to the walker (see "Scan-at-date and scan-only paths").
|
||||
|
||||
### Replay vs walker: single source of truth
|
||||
|
||||
`Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)` is the **only** place that decides which path each compiled action takes during steady state. Both return values are new `*Snapshot` instances containing cloned `*CompiledAction` values with per-partition `active` flags (see "`router.Route` integration" for why cloning is required); either may be `nil` when its partition is empty. Every other component (`daily_run`, the walker invocation, the cursor's `RuleSetHash`, the `max_ttl` calculation) reads through these views. A given compiled action's clone has `active=true` in exactly one of `replay` or `walk`:
|
||||
|
||||
| Reason for walker | Engine signal |
|
||||
|---|---|
|
||||
| `ExpirationDate` (`ModeScanAtDate`) | `compile.go:73` sets `mode == ModeScanAtDate` |
|
||||
| `ExpiredDeleteMarker` | Action kind requires sibling count |
|
||||
| `NewerNoncurrent` | Action kind requires version count |
|
||||
| `scan_only` (TTL > meta-log retention) | `event_log_horizon.go` check against earliest available TsNs at run start |
|
||||
|
||||
Everything else (`ExpirationDays`, `NoncurrentDays`, `AbortMPU` when TTL fits in retention) is on the replay list.
|
||||
|
||||
The `effective_ttl` per active replay action — used by `engine.MaxEffectiveTTL(replay)` — is:
|
||||
|
||||
- `ExpirationDays.ExpirationDays`
|
||||
- `NoncurrentDays.NoncurrentDays`
|
||||
- `AbortMPU.AbortMPUDaysAfterInitiation`
|
||||
|
||||
Walker actions don't appear as active in `replay`, so they naturally contribute nothing to `MaxEffectiveTTL` — the cursor sliding window only governs replay.
|
||||
|
||||
If `snap.replay` is empty (e.g., a bucket whose only rule is `ExpirationDate`), the cursor's `TsNs` is recorded as `0` (or any sentinel) and never read; only `RuleSetHash` matters, so a future addition of a replay rule is detected and triggers the rewind branch. The walker still runs every day for the walker rules.
|
||||
|
||||
### Cursor hashes: `RuleSetHash` and `PromotedHash`
|
||||
|
||||
The cursor stores two hashes. Together they detect every situation that invalidates the cursor's "everything before persisted.TsNs has been processed under the same rules" assumption.
|
||||
|
||||
**`RuleSetHash = engine.ReplayContentHash(snap)`** — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds (`ExpirationDays`, `NoncurrentDays`, `AbortMPU`) in the base snapshot. Partition-independent.
|
||||
|
||||
**`PromotedHash = engine.PromotedHash(snap, retentionWindow)`** — hash of the set of replay-eligible rules that are currently classified as `walk` due to `scan_only` (their TTL exceeds `retentionWindow`). Changes when retention shifts move rules between partitions, even if XML rules and their content are unchanged.
|
||||
|
||||
Why two hashes: a replay-cursor rewind is needed when either the rule definitions change *or* when a rule transitions between partitions in either direction.
|
||||
|
||||
- Promotion (replay → walk, retention dropped): the promoted rule's meta-log events for in-flight objects are no longer reachable. Walker must take over for that rule. Cursor for other replay rules is fine, but the recovery branch's full-rule-set walk is still the safest catch-up — `PromotedHash` change triggers it.
|
||||
- Demotion (walk → replay, retention recovered): the rule's events are now reachable again in meta-log. **But persisted.TsNs may be forward of where the rule's relevant events live**, because other replay rules kept the cursor advancing while this rule was in walk. The `persisted.TsNs < earliest_available` check does not catch this — that check fires only on retention regression. Demotion is detected by `PromotedHash` change.
|
||||
|
||||
Recovery triggers, complete list:
|
||||
|
||||
| Trigger | Detection | Why |
|
||||
|---|---|---|
|
||||
| Cold start | No persisted cursor | First run |
|
||||
| Replay-rule edit | `RuleSetHash` mismatch | Content of replay-eligible rules changed |
|
||||
| Retention loss | `replay != nil && persisted.TsNs < earliest_available` | Worker was down or retention shrank past cursor. Gated on `replay != nil` so the TsNs sentinel (`0`) used by pure walker buckets doesn't perpetually trigger recovery. |
|
||||
| Partition flip | `PromotedHash` mismatch | A replay-eligible rule moved between `replay` and `walk` |
|
||||
|
||||
Walker-only rule edits (`ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule with no replay-eligible action kind) do not change either hash. They are picked up by the steady-state `walk` invocation; replay cursor is untouched.
|
||||
|
||||
Practical consequences:
|
||||
|
||||
- Adding/removing/modifying a walker-only rule: both hashes unchanged, replay cursor untouched.
|
||||
- Retention drops (replay rule promotes to walk): `RuleSetHash` unchanged, `PromotedHash` changes → recovery fires. Walker over base catches already-due objects; cursor rewinds for replay continuity.
|
||||
- Retention recovers (rule demotes back to replay): `RuleSetHash` unchanged, `PromotedHash` changes → recovery fires. The demoted rule's events get a fresh scan window via the rewind.
|
||||
- Editing a replay-eligible XML rule: `RuleSetHash` changes → recovery fires. Walker covers the full rule set; already-due objects under the new rule are caught.
|
||||
|
||||
### Shard-aware walker
|
||||
|
||||
Today's walker (`bootstrap/walker.go`) is bucket-wide: it lists every entry under a bucket and emits matches regardless of `ShardID`. Shards are derived as `sha256(bucket||"/"||key) >> 252` (`weed/s3api/s3lifecycle/shard.go:23`), so a single bucket's objects spread across all 16 shards.
|
||||
|
||||
Daily replay is per-shard, so a per-shard `daily_run(shardID)` invoking the bucket-wide walker would have all 16 shards re-walk every bucket — 16× the filer listing cost, 16× the duplicate delete RPCs (idempotent but wasteful and noisy in metrics).
|
||||
|
||||
Two paths are workable; the design picks the simpler one and notes the optimization:
|
||||
|
||||
1. **Shard-filtering walker (chosen).** Add `walker.RunForShard(shardID int, ...)`. The walker still does one full bucket listing, but for each entry it computes `ShardID(bucket, key)` and skips entries not in the target shard. Cost: still 1 listing per shard per bucket = 16× listing total. Simpler; no coordination across workers. Acceptable for buckets up to ~100M entries where listing is cheap relative to deletes.
|
||||
2. **Bucket-coordinated walker (future).** A separate coordinator (e.g., the worker that owns shard 0 for that bucket) runs the walker once, routing emitted matches to each shard's local injection point. 1 listing per bucket total. Requires a coordination layer that today's plugin/worker model doesn't have. Tracked as an optimization, not blocking.
|
||||
|
||||
Path #1 is wire-compatible with today's walker — only the iteration filter changes. `RunForShard` is a thin wrapper that takes a shard predicate; the existing `Run(bucket)` becomes `RunForShard(shard, bucket)` with `shard == any`.
|
||||
|
||||
### Delete failure handling
|
||||
|
||||
Cursor advance is gated on success. The cursor only moves past events whose deletes returned `DONE`, `NOOP_RESOLVED`, or `SKIPPED_OBJECT_LOCK` (lock-protected, but rule semantics are satisfied). Any other outcome — `RETRY_LATER`, `BLOCKED`, or transport error after in-run retries — halts the run and persists the cursor at the last fully-processed event.
|
||||
|
||||
Semantics:
|
||||
|
||||
- **Head-of-line blocking is intentional.** A transient filer error stalls the rest of today's pass; tomorrow's run resumes at the same cursor and retries from the failing event. This is loud — operator sees a stuck cursor in metrics — and idempotent (identity-CAS makes any redundant delete a no-op).
|
||||
- **In-run retry with backoff** for transport errors only (default 3 attempts, exponential backoff). Server-side outcomes (`RETRY_LATER`, `BLOCKED`) are not retried in-run; the server already classified them as "wait until next run."
|
||||
- **No separate retry queue.** Removing the per-key freeze state was the whole point — adding it back would re-introduce the state machine this design replaces.
|
||||
|
||||
## Data model changes
|
||||
|
||||
### `noncurrent_since` on version entries
|
||||
|
||||
A non-current version's TTL clock starts when the next version was written, not at its own mtime. Today there's no field for that timestamp; the engine has to derive it by joining successive PUTs for the same key.
|
||||
|
||||
Change: at the S3 PUT handler that demotes a prior version (current → noncurrent), write `noncurrent_since` on the demoted entry **to the TsNs of the demoting event**, not local wall-clock. The demoting event is itself a meta-log PUT with a well-defined TsNs; using that value keeps `noncurrent_since` strictly monotonic in meta-log order across all replicas and immune to wall-clock skew between S3 handler nodes.
|
||||
|
||||
The lifecycle evaluator uses `ev.TsNs` for current-version rules and `entry.noncurrent_since` for noncurrent rules — both monotonic in the iteration order, so the algorithm's `done` flag still holds. `expected_mtime` (passed to `LifecycleDelete` for identity CAS) remains the entry's own mtime; CAS identity and TTL clock are kept separate.
|
||||
|
||||
### Cursor
|
||||
|
||||
Per-shard cursor is `{ TsNs int64, RuleSetHash [32]byte, PromotedHash [32]byte }`. No frozen-cursor set, no per-key freeze state — failed deletes halt the run and leave the cursor at the last successful event (see "Delete failure handling"). `RuleSetHash` and `PromotedHash` together gate recovery — mismatch on either triggers the recovery branch on the next run. See "Cursor hashes."
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| `engine/` (rule compilation, snapshot) | Keep |
|
||||
| `evaluate.go`, `due_at.go`, `rule_hash.go`, `tags.go` | Keep |
|
||||
| `reader/` (meta-log subscribe) | Keep, used once per daily pass instead of continuously |
|
||||
| `router/router.go` (rule evaluation per event) | Keep, callable in the per-event loop |
|
||||
| `router/schedule.go` (heap) | **Remove** |
|
||||
| `dispatcher/dispatcher.go` (tick, retries, freeze) | **Remove**; per-call `LifecycleDelete` moves into the daily loop |
|
||||
| `dispatcher/pipeline.go` | **Replace** with `daily_run` per shard |
|
||||
| `scheduler/scheduler.go` (long-running goroutines + refresh ticker) | **Replace** with one-shot per-shard fan-out |
|
||||
| `bootstrap/walker.go` | Keep, with shard filter added (`RunForShard`). Two invocation modes: (a) **steady-state daily** against the `walk` view (`ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, `scan_only` rules); (b) **recovery** against `engine.RecoveryView(snap)` (cold start, replay-rule edit, retention loss, partition flip). No longer feeds an in-memory schedule. |
|
||||
|
||||
## Rate limiting
|
||||
|
||||
Cluster-wide deletes-per-second cap, set as admin config. Admin computes per-worker share at job-dispatch time:
|
||||
|
||||
- Admin counts workers capable of `s3_lifecycle` from the registry, divides `cluster_deletes_per_second` by the count, writes the share into `ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"]`.
|
||||
- Worker reads the share and constructs one `golang.org/x/time/rate.Limiter` shared across all shard goroutines. `dispatch_delete` calls `limiter.Wait(ctx)` before each RPC.
|
||||
- Brief over/undershoot during worker join/leave is acceptable for a bulk workload.
|
||||
|
||||
No proto changes: `ClusterContext.Metadata` already exists.
|
||||
|
||||
## Failure & recovery
|
||||
|
||||
- **Worker crashes mid-run.** Cursor advances only past successfully-deleted events (see "Delete failure handling"); on restart the next run resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
|
||||
- **Transient delete failure.** Run halts at the failing event, cursor stays. Tomorrow's run retries from the same point. Stuck cursor is visible in metrics — operators see head-of-line blocking and address root cause.
|
||||
Four triggers route into the same **recovery branch** (cold-start, replay-rule edit, retention loss, partition flip). All four run the walker against `engine.RecoveryView(snap)` (every action force-`markActive()`'d, regardless of compile-time `IsActive`/`BootstrapComplete`) to catch already-due objects across the full rule set, then rewind the cursor to `now - MaxEffectiveTTL(replay)` (or sentinel `0` if replay is empty). The merge is by design: distinguishing the four would require independent state that isn't worth the complexity, and the walker is idempotent.
|
||||
|
||||
- **Cold start (no persisted cursor).** Recovery branch.
|
||||
- **Replay-rule edit** (new/removed/changed `ExpirationDays`, `NoncurrentDays`, or `AbortMPU` rule). Detected by `RuleSetHash` mismatch. Recovery branch.
|
||||
- **Retention loss** (`replay != nil && persisted.TsNs < earliest_available`). Worker was down longer than meta-log retention, or retention itself shrunk. Gated on `replay != nil` so the sentinel `TsNs=0` used by pure walker buckets doesn't perpetually re-trigger recovery. Recovery branch.
|
||||
- **Partition flip** (`PromotedHash` mismatch — a replay-eligible rule moved between `replay` and `walk` since last run). Covers both directions:
|
||||
- Promotion (replay → walk, retention dropped): walker takes over for that rule. Recovery fires to refresh state.
|
||||
- Demotion (walk → replay, retention recovered): persisted.TsNs is likely forward of the rule's relevant events (other replay rules kept advancing during the walk era). Recovery rewinds the cursor so the demoted rule's events get a fresh scan window.
|
||||
- **Walker-only rule edit** (added/removed/modified `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule with no replay-eligible action kind). Not a recovery trigger — both hashes unchanged, cursor untouched. Walker re-reads `walk` on every steady-state run and applies the new rule automatically.
|
||||
- **`ExpirationDate` rules.** Always routed to the walker — fixed wall-clock due time can't be derived from event TsNs, and objects created before the date may have aged out of meta-log. See "Replay vs walker."
|
||||
- **Identity drift.** Handled by `LifecycleDelete` RPC as today. No worker-side state needed.
|
||||
|
||||
## Comparison
|
||||
|
||||
| | Streaming (today) | Daily replay (this design) |
|
||||
|---|---|---|
|
||||
| Worker lifetime | Long-running, capped by `MaxRuntime` | Starts, completes pass, exits |
|
||||
| Memory | O(events with DueTime in `[now, now+TTL]`) | O(in-flight delete RPCs) |
|
||||
| Disk reads | Continuous meta-log subscribe + periodic bootstrap | One meta-log scan per day, length ≤ max_ttl |
|
||||
| Latency | DispatchTick (~1 min) | Up to 24h |
|
||||
| Future-buffering | Yes, in heap | No |
|
||||
| State persisted | Cursor + frozen set + heap (rebuilt from meta-log on restart) | Cursor only |
|
||||
| Dispatch tick / checkpoint tick / refresh tick | Required | Removed |
|
||||
| Bootstrap walker | Hot path (every run, feeds heap) | Daily for `walk` rules; recovery for full rule set |
|
||||
| Failure recovery | Heap rebuild from meta-log replay | Cursor advance |
|
||||
|
||||
## Configuration changes
|
||||
|
||||
The admin and worker forms collapse significantly. New worker config:
|
||||
|
||||
```text
|
||||
deletes_per_run_concurrency // optional, in-pass parallelism (default 1)
|
||||
max_runtime_minutes // safety cap; default 1440 (24h)
|
||||
```
|
||||
|
||||
Removed worker config (no longer meaningful):
|
||||
|
||||
```text
|
||||
dispatch_tick_minutes
|
||||
checkpoint_tick_seconds
|
||||
refresh_interval_minutes
|
||||
bootstrap_interval_minutes
|
||||
```
|
||||
|
||||
New admin config (added to the existing form):
|
||||
|
||||
```text
|
||||
cluster_deletes_per_second // 0 = unlimited
|
||||
cluster_deletes_burst // 0 = 2× rps
|
||||
```
|
||||
|
||||
Admin runtime stays at `DetectionIntervalMinutes=1440` (daily detection produces one execution per worker).
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
Sequenced so each phase compiles and ships independently. Phase 1 lands first because it's a prerequisite the algorithm relies on; phases 2–4 can be authored in parallel but should merge in order.
|
||||
|
||||
### Phase 1 — `noncurrent_since` on version entries
|
||||
|
||||
Smallest, isolated, prerequisite.
|
||||
|
||||
- Add `NoncurrentSinceNs int64` to the filer entry extended attributes (or as a top-level field if the proto allows non-breaking add).
|
||||
- At the S3 PUT handler in `weed/s3api/s3api_object_handlers_put.go` (or whichever site demotes the prior current version), set `NoncurrentSinceNs` to the **TsNs of the demoting meta-log event**, not `time.Now().UnixNano()`. Read the value from the in-flight write's TsNs (the filer assigns this) — coordinate with the filer write API to make that value available to the demotion path. If the API can't surface it cleanly, fall back to a single `time.Now().UnixNano()` captured at the start of the demoting RPC, accepting that this introduces sub-millisecond skew relative to meta-log TsNs (still monotonic per node, weakly monotonic across nodes within clock-sync tolerance).
|
||||
- Backfill: nil/zero on existing entries — the lifecycle evaluator treats `NoncurrentSinceNs == 0` as "fall back to entry mtime" so legacy data still expires, just with the old (slightly looser) semantics.
|
||||
- Unit tests in the s3api package covering PUT-over-existing and PUT-with-versioning-suspended.
|
||||
- Verify with a clock-skew test: two S3 nodes with ±5s clock drift produce a sequence of PUT/demotion events; assert `noncurrent_since` is non-decreasing in meta-log iteration order.
|
||||
|
||||
### Phase 2 — Daily-replay worker (parallel to today's, behind a flag)
|
||||
|
||||
Add the new code path without disturbing the existing one. Allows side-by-side smoke testing.
|
||||
|
||||
- New file `weed/s3api/s3lifecycle/dailyrun/dailyrun.go` with `func Run(ctx, shardID, ...) error` implementing the algorithm.
|
||||
- Reuses `engine.Snapshot`, `evaluate.go`, `due_at.go`, the existing `reader.Reader`, and the existing `LifecycleDelete` RPC. No new RPCs.
|
||||
- Per-shard cursor type `dailyrun.Cursor { TsNs int64; RuleSetHash [32]byte; PromotedHash [32]byte }`, JSON-persisted under the same filer path as today's cursor, keyed by `daily/` prefix to avoid collision. Cursor load returns `(cursor, isFresh bool)` so the algorithm can distinguish first-run from a rule-change rewrite. In Phase 2 (no walker plumbing), `PromotedHash` is always the empty hash; Phase 4 wires `engine.PromotedHash(snap, retentionWindow)` and the partition-flip recovery trigger.
|
||||
- **Replay/walker partition**: full `RulesForShard` / `RecoveryView` plumbing lands in Phase 4. For Phase 2 alone, since walker rules are rejected (next bullet), Phase 2 is **replay-only**: pass the **base snapshot** directly to `router.Route` (no view, no clone, no Mode rewrite). This is acceptable for Phase 2 because the unsupported-rule check guarantees no walker-bound actions are present, and the absence of walker rules means there's no risk of router.Route emitting walker-bound matches. Mode rewrite is also unnecessary in Phase 2: any rule that compiled to `ModeScanOnly` (legacy persistent state) falls into the unsupported-rule path along with `scan_only`-promoted rules. Phase 4 then introduces clone-per-view (including Mode rewrite) and the recovery branch.
|
||||
- **Refuse to run on unsupported rule sets.** Before invoking the replay loop, `dailyrun.Run` checks the compiled rules for any action of kind `ExpirationDate`, `ExpiredDeleteMarker`, `NewerNoncurrent`, or any rule promoted to `scan_only`. If any are present, the handler:
|
||||
- Logs at error level which rules are unsupported.
|
||||
- Returns a typed error from `Execute` so admin marks the run failed (visible in the activity log).
|
||||
- Does **not** advance the cursor.
|
||||
This makes flipping `algorithm: daily_replay` before Phase 4 a loud failure, not a silent dropped rule. Operators see the error and either revert to `streaming` or wait for Phase 4.
|
||||
- **Rule-change handling**: compute `rsh = engine.ReplayContentHash(snap)` at run start (partition-independent — see "Cursor hashes"). If `rsh != cursor.RuleSetHash` and `replay` is non-`nil`, persist `{ TsNs: now - MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted }` and return. If `replay` is `nil`, persist `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted }` and return. In Phase 2, `promoted` is the empty hash (no walker plumbing); Phase 4 wires real values. All cursor writes carry all three fields so Phase 4 doesn't have to migrate existing Phase 2 cursors. Tested explicitly: add a replay-eligible rule mid-test, assert cursor TsNs rewinds to `now - max_ttl` and `RuleSetHash` updates.
|
||||
- **Delete failure handling**: track `last_ok TsNs` as the algorithm progresses; only set `last_ok = ev.TsNs` after a successful or `NOOP_RESOLVED` outcome for *all* matches produced from that event. On any unresolved outcome (`RETRY_LATER`, `BLOCKED`, transport error after in-run retries), break the loop and persist `{ TsNs: last_ok, RuleSetHash: rsh, PromotedHash: promoted }`. In-run retry policy for transport errors: 3 attempts with exponential backoff capped at 5s.
|
||||
- **Match expansion**: invoke `router.Route(ctx, snap, ev, now, lister)` per event, where `snap` is the **base snapshot** directly (Phase 2 is replay-only — see "Replay/walker partition" bullet above; the view-based call site `router.Route(ctx, replay, ev, ...)` lands in Phase 4 when the partition is wired). Handles `ExpirationDays`, `NoncurrentDays`, `AbortMPU` correctly out of the box (already in `due_at.go`).
|
||||
- Rate limiter accepted as a parameter; constructed once per `Execute()` and shared across all shard goroutines.
|
||||
- New admin config flag `algorithm: streaming | daily_replay` in `weed/worker/tasks/s3_lifecycle/handler.go`. When `daily_replay`, the handler invokes `dailyrun.Run` and exits. When `streaming` (default), today's code path runs unchanged.
|
||||
- Unit tests:
|
||||
- Per-rule `done` flag correctness (single rule, multi-rule).
|
||||
- Multiple-match fan-out from one event (PUT-over-existing emits both `ExpirationDays` for the new version and `NoncurrentDays` for the displaced one).
|
||||
- Cursor advance halts at the first unresolved failure; subsequent run resumes there.
|
||||
- Identity-CAS skip path (event mtime stale → `NOOP_RESOLVED` still advances cursor).
|
||||
- Rule-set-hash mismatch rewinds cursor to `now - max_ttl` (Phase 4 adds the walker call on this path; Phase 2 just rewinds).
|
||||
- `replay` empty case: cursor persists `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: <empty in Phase 2> }`, no meta-log read attempted.
|
||||
- `NoncurrentDays` rule uses `entry.noncurrent_since`, `ExpirationDays` rule uses `ev.TsNs`; both monotonic in iteration order.
|
||||
- `AbortMPU` test: MPU init event ages past `AbortMPUDaysAfterInitiation`, delete emitted.
|
||||
- **Unsupported-rule rejection**: bucket has an `ExpirationDate` (or `ExpiredDeleteMarker` or `NewerNoncurrent`) rule and the flag is set to `daily_replay`. Assert `Execute` returns the typed unsupported-rule error, the cursor is not advanced, and the streaming code path was not invoked. Repeat for a rule promoted to `scan_only`.
|
||||
|
||||
### Phase 3 — Cluster rate limit allocation
|
||||
|
||||
Admin-side share computation. Lands once Phase 2 has a working `Limiter` parameter to receive it.
|
||||
|
||||
- `weed/worker/tasks/s3_lifecycle/handler.go`: add `cluster_deletes_per_second`, `cluster_deletes_burst` to `AdminConfigForm.Sections["scope"]`.
|
||||
- `weed/admin/plugin/plugin.go` near `RunDetectionRequest`/`ExecuteJobRequest` build sites: count capable workers, divide rps, write `s3_lifecycle.deletes_per_second` and `s3_lifecycle.deletes_burst` into `ClusterContext.Metadata`.
|
||||
- Worker `Execute`: parse metadata keys, build `rate.Limiter`, pass to `dailyrun.Run`.
|
||||
- New histogram `S3LifecycleDispatchLimiterWaitSeconds` for operator visibility.
|
||||
- Tests: admin allocation math under join/leave, worker correctly slows under low share.
|
||||
|
||||
### Phase 4 — Shard-aware walker + scan-at-date / scan-only routing
|
||||
|
||||
Reposition the walker and add shard-filtering + per-rule routing.
|
||||
|
||||
- **Shard filter on the walker.** Add `walker.RunForShard(ctx context.Context, shardID int, view *engine.Snapshot, limiter *rate.Limiter) error`. `view` is one of `walk`/`replay`/`recovery` views — its action set already filters to the right partition (via per-clone `active` flags), so the walker iterates the bucket and evaluates each entry against `view` exactly as today's `Run` does, with the only added step being `if ShardID(bucket, entry.Name) != shardID { continue }`. The existing bucket-wide `Run` becomes `RunForShard` with `shardID = -1` (any) for backward compatibility, or is deleted outright in Phase 5.
|
||||
- **Engine surface.** Four new exports on the engine package, all consumed by `daily_run`:
|
||||
- `func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)`. Returns two new `*Snapshot` instances with **cloned `*CompiledAction`** values; per-view `active` flags (and `Mode` rewrite on `replay`) determine partition membership. Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table. Either return value may be `nil` when its partition is empty. The retention window is an explicit input — `daily_run` computes it once at run start as `duration(now - metalog.EarliestAvailableTsNs(shardID))` and passes it in. The engine does no clock arithmetic of its own. The partition is stable across one `daily_run` invocation. Membership:
|
||||
- **`replay`**: cloned `ExpirationDays`, `NoncurrentDays`, `AbortMPU` actions, `active=true`, where the action's TTL ≤ `retentionWindow`.
|
||||
- **`walk`**: cloned `ExpirationDate` (`ModeScanAtDate`), `ExpiredDeleteMarker`, `NewerNoncurrent` actions, plus any of the above promoted to `scan_only` because their TTL exceeds the retention boundary. All `active=true`.
|
||||
- A single XML rule that compiles to multiple action kinds (e.g., `ExpirationDays` + `NewerNoncurrent` on the same rule) is split: each compiled action lands in `replay` or `walk` independently. `ReplayContentHash` only hashes replay-eligible rule definitions from the base snapshot, so neither the `NewerNoncurrent` action nor any walker-only edit affects the hash.
|
||||
- `func RecoveryView(s *Snapshot) *Snapshot`. Returns a third `*Snapshot` instance with every action cloned and `markActive()`'d — regardless of `Mode` or prior `BootstrapComplete` state. The recovery walker uses this view to see all replay-eligible + walker-eligible rules. Without it, recovery would skip event-driven actions the base snapshot marks inactive (`compile.go:73-74`).
|
||||
- `func ReplayContentHash(s *Snapshot) [32]byte`. Stable across reorderings, partition-independent. Used as `RuleSetHash = ReplayContentHash(snap)`. Returns the empty hash when no replay-eligible rules exist. See "Cursor hashes."
|
||||
- `func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte`. Hash of the set of replay-eligible rules currently classified as `walk` due to `scan_only` (TTL > `retentionWindow`). Detects partition flips in either direction. Takes the same `retentionWindow` value as `RulesForShard` so the two cannot disagree about partition membership. See "Cursor hashes."
|
||||
- `func MaxEffectiveTTL(s *Snapshot) time.Duration`. Max over active replay actions; returns `0` for `nil` or empty replay (caller must already be in the empty-replay branch).
|
||||
- `func CurrentSnapshot() *Snapshot`. The base snapshot before partitioning. Recovery branch passes this to `RecoveryView`.
|
||||
- **`dailyrun.Run` flow.** Cold-start, replay-rule edit (`RuleSetHash` mismatch), retention loss, and partition flip (`PromotedHash` mismatch) all route into the **recovery branch**: walker over `engine.RecoveryView(snap)` so already-due objects across all rules are caught (including event-driven actions the base snapshot may mark inactive), then cursor rewinds. Steady-state path runs walker over `walk` only, then replay over `replay`. Cursor stores `{ TsNs, RuleSetHash, PromotedHash }`; steady-state writes both hashes through unchanged.
|
||||
- **Recovery branch.** Triggered by any of: no persisted cursor, `RuleSetHash` mismatch, `PromotedHash` mismatch, or `replay != nil && persisted.TsNs < earliest_available`. The retention-loss trigger uses the **absolute** `earliest_available` TsNs (the oldest available meta-log TsNs from `metalog.EarliestAvailableTsNs(shardID)`), not the `retentionWindow` duration that `RulesForShard`/`PromotedHash` consume — they're different shapes for different purposes: the retention check on the cursor compares two absolute timestamps, while the engine partition check compares a per-action TTL to a duration. Gated on `replay != nil` because pure walker buckets persist a sentinel `TsNs=0` that would otherwise re-fire recovery every run. Invokes `walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)`. Persists cursor at:
|
||||
- `{ TsNs: now - engine.MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted }` if `replay` is non-`nil`.
|
||||
- `{ TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted }` if `replay` is `nil` (pure walker bucket).
|
||||
`MaxEffectiveTTL` is undefined over an empty set, so the sentinel form is required when `replay` is `nil`. **Both hashes are written on every recovery persistence** — failing to update `PromotedHash` here would cause the partition-flip trigger to re-fire on every subsequent run. See "Bootstrap retention window."
|
||||
- **Steady-state walker invocation.** Distinct from recovery: `walker.RunForShard(ctx, shardID, walk, limiter)`. `walk`'s cloned actions are masked to walker-bound kinds only, so the walker doesn't re-process replay rules that the meta-log scan already handles on the same run.
|
||||
- Drop the bootstrap walker's call from `scheduler.go` (removed entirely in Phase 5).
|
||||
- Tests:
|
||||
- Cold start with mixed walker-only + replay rules in same bucket: recovery branch fires once (walker over `engine.RecoveryView(snap)`, then cursor rewind). Next day's run is steady-state (walker over `walk` only, replay over `replay`).
|
||||
- `ExpirationDate` rule fires only on/after the date (walker path).
|
||||
- `ExpiredDeleteMarker` rule deletes orphaned markers (walker path, sibling-aware).
|
||||
- `NewerNoncurrent` count rule deletes oldest noncurrent versions beyond the cap (walker path, version-list-aware).
|
||||
- Bucket with only walker rules (e.g., pure `ExpirationDate`): `replay` is `nil`, cursor persisted with sentinel `TsNs=0`, no meta-log subscribe attempted. Critically — assert that across 5 consecutive daily runs, recovery fires exactly once (on the first run); subsequent runs see no recovery despite `persisted.TsNs (0) < earliest_available`, because the retention-loss check is gated on `replay != nil`.
|
||||
- Replay-rule edit triggers recovery: walker runs over `RecoveryView(snap)` (catches an object already 60d old under a brand-new 30d rule), then cursor rewinds. Object aged 20d at edit time is later deleted at age 30 via the replay path.
|
||||
- `RecoveryView` activates event-driven actions: simulate a base snapshot where an `ExpirationDays` action has `IsActive=false` (e.g., `BootstrapComplete=false`); assert the recovery walker still sees and evaluates the rule via the recovery view, while the steady-state walker over `walk` does not.
|
||||
- **Replay view rewrites Mode**: simulate a base snapshot with a `ModeScanOnly`-locked `ExpirationDays` action (carried over from `prior.Mode`); assert that `replay`'s clone of the action has `Mode == ModeEventDriven`, and that `router.Route(ctx, replay, ev, ...)` emits a Match for it. The test should fail if Mode rewrite is skipped — confirming the rehabilitation path is wired.
|
||||
- Cursor persistence after recovery: assert `PromotedHash` is written through on every recovery branch persistence; simulate a partition flip across two runs and verify the second run does not re-fire recovery because the prior `promoted_hash` was correctly stored.
|
||||
- Adding a walker-only rule to an existing replay-only bucket: `ReplayContentHash(snap)` is unchanged, replay cursor TsNs unchanged, walker picks up the new rule on its next steady-state pass.
|
||||
- `scan_only` promotion (retention drops, replay-eligible rule moves to `walk`): `ReplayContentHash` unchanged, **`PromotedHash` changes** → recovery branch fires. Walker over base catches in-flight objects under the promoted rule; cursor rewinds; `PromotedHash` is persisted.
|
||||
- Retention recovery (`scan_only` rule demotes back to `replay` while other replay rules' cursor has advanced past the rule's relevant events): `ReplayContentHash` unchanged, `persisted.TsNs > earliest_available` so retention-loss check does NOT fire, **`PromotedHash` changes** → recovery branch fires. Asserts the demoted rule's older events get processed (this is the test that would have failed before adding `PromotedHash`).
|
||||
- `PromotedHash` persistence: simulate a partition flip in either direction across two runs, assert cursor's `promoted_hash` field carries through and gates the next run's recovery decision correctly.
|
||||
- Shard filter: 2 shards × 1 bucket walker run leaves no double-deletes (assert via RPC counter).
|
||||
- `scan_only` promotion: rule with TTL > meta-log retention moves from `replay` to `walk` at run start.
|
||||
|
||||
### Phase 5 — Switch default and delete streaming path
|
||||
|
||||
Cut over and clean up. Should be a single PR that's mostly deletions.
|
||||
|
||||
- Default `algorithm: daily_replay` and remove the streaming alternative from the descriptor.
|
||||
- Delete:
|
||||
- `weed/s3api/s3lifecycle/router/schedule.go` (+ tests)
|
||||
- `weed/s3api/s3lifecycle/dispatcher/dispatcher.go` (+ tests), `freeze`-related code, retry state
|
||||
- `weed/s3api/s3lifecycle/dispatcher/pipeline.go`
|
||||
- `weed/s3api/s3lifecycle/scheduler/scheduler.go` (+ tests), refresh ticker
|
||||
- Removed worker config fields and their `ParseConfig` handling
|
||||
- Rewrite `weed/s3api/s3lifecycle/DESIGN.md` to remove "streaming" sections.
|
||||
- One full end-to-end test on a bucket with 1M objects, mixed-TTL rules, with cluster rate cap engaged.
|
||||
|
||||
### Phase 6 — Observability polish
|
||||
|
||||
After cutover, before announcing.
|
||||
|
||||
- Daily-run summary log line: shard, rules-touched, events-scanned, deletes-issued, walker-fallback (y/n), duration.
|
||||
- Prometheus: `S3LifecycleDailyRunDurationSeconds`, `S3LifecycleDailyRunEventsScanned`, `S3LifecycleDailyRunDeletesIssued`, all labelled by `shard_id` and `outcome`.
|
||||
- Admin UI: a single "Last Run" card per bucket showing the above summary line. Removes the now-meaningless "Next Run / Detection Results" widgets we already hid for `s3_lifecycle`.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **In-pass concurrency.** Today's worker has 1–16 shard goroutines via `workers` config. Daily replay can either (a) keep that knob, fanning out shard scans across goroutines that share the limiter, or (b) drop it since the limiter is now the throughput governor. Lean toward (a) — shards are independent meta-log subscriptions and parallel reads cut wall-clock significantly.
|
||||
2. **Cursor format compatibility.** Worth deciding whether the new cursor coexists at a separate filer path (clean cutover, no migration) or overwrites the old one (single source of truth, requires a one-time read-then-rewrite during cutover). Lean toward separate paths during phases 2–4, unified at Phase 5.
|
||||
3. **Per-shard rule snapshots vs. global.** Engine snapshots are bucket-scoped today; Phase 4 adds `Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot)`. Open: should `RulesForShard` also gate by which buckets a shard "owns" (every bucket is on every shard via `ShardID(bucket, key)`), or just return the global rule set partitioned by mode? Current lean: return global partition, since every shard worker runs against every bucket and the actual shard filter happens at the entry-iteration site (meta-log subscription and walker entry loop).
|
||||
6. **Bucket-coordinated walker.** Phase 4 has each shard walk the full bucket and filter by `ShardID` — simple, but 16× the listing cost. A future optimization is a per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards via the existing injection point). Worth doing if listing cost becomes the bottleneck for very large buckets. Not blocking initial cutover.
|
||||
4. **`noncurrent_since` source from filer write API.** Phase 1 prefers the demoting event's TsNs over wall-clock. Whether the existing filer write path surfaces that TsNs to the S3 handler without a proto change is a Phase 1 design decision; the fallback (handler-local `time.Now()`) is documented but loses cross-node monotonicity guarantees and should be a temporary measure.
|
||||
5. **Retry budget for `RETRY_LATER`.** The current dispatcher tracks per-key retry counts; this design removes that state and lets head-of-line blocking surface the issue. Worth confirming that operators are OK with a stuck cursor as the failure signal vs. silent retry. If not, add a bounded retry counter on the cursor (one per stuck event) — small state, but inches back toward the model we're replacing.
|
||||
149
weed/s3api/s3lifecycle/dailyrun/cursor.go
Normal file
149
weed/s3api/s3lifecycle/dailyrun/cursor.go
Normal file
@ -0,0 +1,149 @@
|
||||
// Package dailyrun implements the daily-replay s3 lifecycle worker
|
||||
// described in weed/s3api/s3lifecycle/DESIGN.md. One pass per day per
|
||||
// shard reads the meta-log forward from the persisted cursor, drains
|
||||
// every event whose due_time is past now, and exits — replacing the
|
||||
// streaming + heap pipeline with a bounded, idempotent scan.
|
||||
//
|
||||
// Phase 2 (this file's first version): replay-only. Buckets whose
|
||||
// compiled rules include walker-bound action kinds (ExpirationDate,
|
||||
// ExpiredDeleteMarker, NewerNoncurrent) or any scan_only promotion are
|
||||
// refused with a typed error so flipping the algorithm flag is a loud
|
||||
// failure rather than silent data loss. Phase 4 wires the walker and
|
||||
// the recovery branch.
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
|
||||
)
|
||||
|
||||
// CursorDir is the filer directory holding per-shard daily-replay
|
||||
// cursor files. Kept distinct from dispatcher.CursorDir so a deployment
|
||||
// running both algorithms during cutover doesn't have one trample the
|
||||
// other's persisted state.
|
||||
const CursorDir = "/etc/s3/lifecycle/daily-cursors"
|
||||
|
||||
// cursorFileVersion bumps when the on-disk shape changes. Phase 2
|
||||
// writes version 1; Phase 4 will continue writing version 1 since the
|
||||
// schema (TsNs + RuleSetHash + PromotedHash) is already final.
|
||||
const cursorFileVersion = 1
|
||||
|
||||
// Cursor captures everything daily_run needs to decide whether to
|
||||
// recover (Phase 4) or continue steady-state. TsNs is the latest
|
||||
// meta-log event whose Matches all dispatched successfully (or as
|
||||
// NOOP_RESOLVED); RuleSetHash is the content hash of the replay-eligible
|
||||
// rules from the run that wrote this cursor; PromotedHash is the hash
|
||||
// of replay-eligible rules currently in scan_only.
|
||||
//
|
||||
// In Phase 2 PromotedHash is always the empty hash because the walker
|
||||
// path isn't wired yet — any rule that would land in walk causes the
|
||||
// run to refuse. Phase 4 starts writing a real value.
|
||||
type Cursor struct {
|
||||
TsNs int64
|
||||
RuleSetHash [32]byte
|
||||
PromotedHash [32]byte
|
||||
}
|
||||
|
||||
// cursorFile is the on-disk JSON shape. Bytes are base64-encoded by
|
||||
// encoding/json automatically.
|
||||
type cursorFile struct {
|
||||
Version int `json:"version"`
|
||||
ShardID int `json:"shard_id"`
|
||||
TsNs int64 `json:"ts_ns"`
|
||||
RuleSetHash []byte `json:"rule_set_hash"`
|
||||
PromotedHash []byte `json:"promoted_hash"`
|
||||
}
|
||||
|
||||
// CursorPersister loads and saves daily-replay cursors. The Phase 2
|
||||
// production implementation is FilerCursorPersister; tests inject a
|
||||
// fake.
|
||||
type CursorPersister interface {
|
||||
Load(ctx context.Context, shardID int) (Cursor, bool, error) // (cursor, found, err)
|
||||
Save(ctx context.Context, shardID int, c Cursor) error
|
||||
}
|
||||
|
||||
// FilerCursorPersister writes cursors to CursorDir as one JSON file per
|
||||
// shard. Reuses dispatcher.FilerStore for the actual filer I/O so both
|
||||
// algorithms share the same minimal storage abstraction.
|
||||
type FilerCursorPersister struct {
|
||||
Store dispatcher.FilerStore
|
||||
}
|
||||
|
||||
func cursorFileName(shardID int) string {
|
||||
return fmt.Sprintf("shard-%02d.json", shardID)
|
||||
}
|
||||
|
||||
// Load returns (zero-Cursor, false, nil) only when the cursor file
|
||||
// does not exist yet (cold start). Every other failure mode — empty
|
||||
// file, malformed JSON, wrong version, wrong shard, hash slices not
|
||||
// exactly 32 bytes — returns an error so the daily run halts and is
|
||||
// fixed by an operator rather than silently re-scanning from time zero.
|
||||
//
|
||||
// Strict shape validation is load-bearing because the cursor is
|
||||
// load → mutate → save in a single run: a partial-truncate that
|
||||
// silently zero-padded the rule_set_hash would be persisted back as a
|
||||
// real-looking hash, masking the corruption forever.
|
||||
func (p *FilerCursorPersister) Load(ctx context.Context, shardID int) (Cursor, bool, error) {
|
||||
if p.Store == nil {
|
||||
return Cursor{}, false, errors.New("FilerCursorPersister: nil Store")
|
||||
}
|
||||
content, err := p.Store.Read(ctx, CursorDir, cursorFileName(shardID))
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
return Cursor{}, false, nil
|
||||
}
|
||||
return Cursor{}, false, fmt.Errorf("cursor read shard=%d: %w", shardID, err)
|
||||
}
|
||||
if len(content) == 0 {
|
||||
return Cursor{}, false, fmt.Errorf("cursor shard=%d: file exists but is empty (partial write or external truncation)", shardID)
|
||||
}
|
||||
var cf cursorFile
|
||||
if err := json.Unmarshal(content, &cf); err != nil {
|
||||
return Cursor{}, false, fmt.Errorf("cursor decode shard=%d: %w", shardID, err)
|
||||
}
|
||||
if cf.Version != cursorFileVersion {
|
||||
return Cursor{}, false, fmt.Errorf("cursor version shard=%d: got %d, want %d", shardID, cf.Version, cursorFileVersion)
|
||||
}
|
||||
if cf.ShardID != shardID {
|
||||
return Cursor{}, false, fmt.Errorf("cursor shard mismatch: file declares shard=%d, requested shard=%d", cf.ShardID, shardID)
|
||||
}
|
||||
if len(cf.RuleSetHash) != 32 {
|
||||
return Cursor{}, false, fmt.Errorf("cursor rule_set_hash shard=%d: got %d bytes, want 32", shardID, len(cf.RuleSetHash))
|
||||
}
|
||||
if len(cf.PromotedHash) != 32 {
|
||||
return Cursor{}, false, fmt.Errorf("cursor promoted_hash shard=%d: got %d bytes, want 32", shardID, len(cf.PromotedHash))
|
||||
}
|
||||
c := Cursor{TsNs: cf.TsNs}
|
||||
copy(c.RuleSetHash[:], cf.RuleSetHash)
|
||||
copy(c.PromotedHash[:], cf.PromotedHash)
|
||||
return c, true, nil
|
||||
}
|
||||
|
||||
func (p *FilerCursorPersister) Save(ctx context.Context, shardID int, c Cursor) error {
|
||||
if p.Store == nil {
|
||||
return errors.New("FilerCursorPersister: nil Store")
|
||||
}
|
||||
cf := cursorFile{
|
||||
Version: cursorFileVersion,
|
||||
ShardID: shardID,
|
||||
TsNs: c.TsNs,
|
||||
RuleSetHash: c.RuleSetHash[:],
|
||||
PromotedHash: c.PromotedHash[:],
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
enc := json.NewEncoder(&buf)
|
||||
enc.SetIndent("", " ")
|
||||
if err := enc.Encode(cf); err != nil {
|
||||
return fmt.Errorf("cursor encode shard=%d: %w", shardID, err)
|
||||
}
|
||||
if err := p.Store.Save(ctx, CursorDir, cursorFileName(shardID), buf.Bytes()); err != nil {
|
||||
return fmt.Errorf("cursor save shard=%d: %w", shardID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
155
weed/s3api/s3lifecycle/dailyrun/cursor_test.go
Normal file
155
weed/s3api/s3lifecycle/dailyrun/cursor_test.go
Normal file
@ -0,0 +1,155 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// fakeStore is a minimal in-memory dispatcher.FilerStore for cursor tests.
|
||||
type fakeStore struct {
|
||||
mu sync.Mutex
|
||||
files map[string][]byte
|
||||
}
|
||||
|
||||
func newFakeStore() *fakeStore {
|
||||
return &fakeStore{files: map[string][]byte{}}
|
||||
}
|
||||
|
||||
func (s *fakeStore) key(dir, name string) string { return dir + "/" + name }
|
||||
|
||||
func (s *fakeStore) Read(_ context.Context, dir, name string) ([]byte, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
b, ok := s.files[s.key(dir, name)]
|
||||
if !ok {
|
||||
return nil, filer_pb.ErrNotFound
|
||||
}
|
||||
return append([]byte(nil), b...), nil
|
||||
}
|
||||
|
||||
func (s *fakeStore) Save(_ context.Context, dir, name string, content []byte) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.files[s.key(dir, name)] = append([]byte(nil), content...)
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_EmptyLoadReturnsNotFound(t *testing.T) {
|
||||
p := &FilerCursorPersister{Store: newFakeStore()}
|
||||
c, found, err := p.Load(context.Background(), 0)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, found)
|
||||
assert.Equal(t, Cursor{}, c)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_SaveLoadRoundTrip(t *testing.T) {
|
||||
p := &FilerCursorPersister{Store: newFakeStore()}
|
||||
in := Cursor{
|
||||
TsNs: 1700000000_000000123,
|
||||
}
|
||||
for i := range in.RuleSetHash {
|
||||
in.RuleSetHash[i] = byte(i + 1)
|
||||
}
|
||||
for i := range in.PromotedHash {
|
||||
in.PromotedHash[i] = byte(0xff - i)
|
||||
}
|
||||
require.NoError(t, p.Save(context.Background(), 3, in))
|
||||
|
||||
out, found, err := p.Load(context.Background(), 3)
|
||||
require.NoError(t, err)
|
||||
require.True(t, found)
|
||||
assert.Equal(t, in, out)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_IsolatesShards(t *testing.T) {
|
||||
p := &FilerCursorPersister{Store: newFakeStore()}
|
||||
a := Cursor{TsNs: 100}
|
||||
b := Cursor{TsNs: 200}
|
||||
require.NoError(t, p.Save(context.Background(), 0, a))
|
||||
require.NoError(t, p.Save(context.Background(), 1, b))
|
||||
|
||||
got0, _, err := p.Load(context.Background(), 0)
|
||||
require.NoError(t, err)
|
||||
got1, _, err := p.Load(context.Background(), 1)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(100), got0.TsNs)
|
||||
assert.Equal(t, int64(200), got1.TsNs)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_CorruptDataReturnsError(t *testing.T) {
|
||||
store := newFakeStore()
|
||||
store.files[store.key(CursorDir, "shard-00.json")] = []byte("not json")
|
||||
p := &FilerCursorPersister{Store: store}
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_EmptyFileReturnsError(t *testing.T) {
|
||||
// A cursor file that exists but is empty signals a partial write
|
||||
// or external truncation. Treating it as "cold start" would silently
|
||||
// re-scan from time zero and burn through a meta-log window for no
|
||||
// reason. Pin that we error instead.
|
||||
store := newFakeStore()
|
||||
store.files[store.key(CursorDir, "shard-00.json")] = []byte{}
|
||||
p := &FilerCursorPersister{Store: store}
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_WrongVersionReturnsError(t *testing.T) {
|
||||
// Future schema bumps must not silently overwrite a cursor written
|
||||
// by a newer worker. Pin that contract here.
|
||||
store := newFakeStore()
|
||||
store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":42,"shard_id":0,"ts_ns":0,"rule_set_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","promoted_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}`)
|
||||
p := &FilerCursorPersister{Store: store}
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_ShardIDMismatchReturnsError(t *testing.T) {
|
||||
// A file declaring shard=1 read by a shard=0 caller is corruption
|
||||
// (or a misroute); applying that cursor would advance the wrong
|
||||
// shard's state. Reject loudly.
|
||||
store := newFakeStore()
|
||||
// Build a valid v1 payload but with shard_id=1 in the body.
|
||||
in := Cursor{TsNs: 42}
|
||||
for i := range in.RuleSetHash {
|
||||
in.RuleSetHash[i] = byte(i + 1)
|
||||
}
|
||||
for i := range in.PromotedHash {
|
||||
in.PromotedHash[i] = byte(i)
|
||||
}
|
||||
p := &FilerCursorPersister{Store: store}
|
||||
require.NoError(t, p.Save(context.Background(), 1, in))
|
||||
// Reparent that file as if it lived at shard 0 — the test's fake
|
||||
// store only keys by name, so we copy the bytes manually.
|
||||
store.mu.Lock()
|
||||
store.files[store.key(CursorDir, "shard-00.json")] = store.files[store.key(CursorDir, "shard-01.json")]
|
||||
store.mu.Unlock()
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_HashLengthMismatchReturnsError(t *testing.T) {
|
||||
// Hand-rolled JSON with hashes shorter than 32 bytes. copy() would
|
||||
// silently zero-pad which makes the hash comparison match a value
|
||||
// that doesn't actually exist on disk. Reject loudly so an operator
|
||||
// fixes the persisted state.
|
||||
store := newFakeStore()
|
||||
store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":1,"shard_id":0,"ts_ns":0,"rule_set_hash":"AA==","promoted_hash":"AA=="}`)
|
||||
p := &FilerCursorPersister{Store: store}
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFilerCursorPersister_NilStoreErrors(t *testing.T) {
|
||||
p := &FilerCursorPersister{}
|
||||
_, _, err := p.Load(context.Background(), 0)
|
||||
require.Error(t, err)
|
||||
require.Error(t, p.Save(context.Background(), 0, Cursor{}))
|
||||
}
|
||||
111
weed/s3api/s3lifecycle/dailyrun/dispatch.go
Normal file
111
weed/s3api/s3lifecycle/dailyrun/dispatch.go
Normal file
@ -0,0 +1,111 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
||||
)
|
||||
|
||||
// transportRetryAttempts is the in-run retry budget for transport
|
||||
// errors (RPC dial failures, stream resets). Server-side outcomes
|
||||
// (RETRY_LATER, BLOCKED) are NOT retried in-run — the server already
|
||||
// classified them as "wait until next run." The retry exists only to
|
||||
// paper over a single flake from the network.
|
||||
const transportRetryAttempts = 3
|
||||
|
||||
// transportRetryInitial is the first sleep after a failed attempt.
|
||||
// Doubled per attempt up to transportRetryMax. Phase 2 keeps these
|
||||
// small because the daily run's own MaxRuntime is the larger cap.
|
||||
const (
|
||||
transportRetryInitial = 200 * time.Millisecond
|
||||
transportRetryMax = 5 * time.Second
|
||||
)
|
||||
|
||||
// dispatchWithRetry sends one LifecycleDelete request, retrying on
|
||||
// transport errors up to transportRetryAttempts times with exponential
|
||||
// backoff. Returns the server's outcome on success, or an error if all
|
||||
// retries exhausted. Server outcomes (RETRY_LATER / BLOCKED) bypass
|
||||
// the retry — the daily run's halt-on-failure semantics handles them.
|
||||
func dispatchWithRetry(ctx context.Context, client LifecycleClient, m router.Match) (s3_lifecycle_pb.LifecycleDeleteOutcome, error) {
|
||||
req := buildDeleteRequest(m)
|
||||
backoff := transportRetryInitial
|
||||
var lastErr error
|
||||
for attempt := 1; attempt <= transportRetryAttempts; attempt++ {
|
||||
resp, err := client.LifecycleDelete(ctx, req)
|
||||
if err == nil {
|
||||
return resp.Outcome, nil
|
||||
}
|
||||
// Context cancellation is shutdown, not a transport flake.
|
||||
// Surface it immediately so the caller halts and tomorrow
|
||||
// resumes from the same cursor.
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, err
|
||||
}
|
||||
lastErr = err
|
||||
if attempt == transportRetryAttempts {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, ctx.Err()
|
||||
case <-time.After(backoff):
|
||||
}
|
||||
backoff *= 2
|
||||
if backoff > transportRetryMax {
|
||||
backoff = transportRetryMax
|
||||
}
|
||||
}
|
||||
return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, lastErr
|
||||
}
|
||||
|
||||
// buildDeleteRequest constructs the LifecycleDelete RPC payload for a
|
||||
// router Match. Mirrors dispatcher.dispatchOne's request shape — both
|
||||
// targets the same server-side handler and the proto encoding must
|
||||
// match exactly. Duplicated rather than shared because Phase 5
|
||||
// deletes the dispatcher's call site and the cross-package dependency
|
||||
// would just create a temporary import.
|
||||
func buildDeleteRequest(m router.Match) *s3_lifecycle_pb.LifecycleDeleteRequest {
|
||||
rh := m.Key.RuleHash
|
||||
return &s3_lifecycle_pb.LifecycleDeleteRequest{
|
||||
Bucket: m.Bucket,
|
||||
ObjectPath: m.ObjectKey,
|
||||
VersionId: m.VersionID,
|
||||
RuleHash: rh[:],
|
||||
ActionKind: toProtoActionKind(m.Key.ActionKind),
|
||||
ExpectedIdentity: toProtoIdentity(m.Identity),
|
||||
}
|
||||
}
|
||||
|
||||
func toProtoActionKind(k s3lifecycle.ActionKind) s3_lifecycle_pb.ActionKind {
|
||||
switch k {
|
||||
case s3lifecycle.ActionKindExpirationDays:
|
||||
return s3_lifecycle_pb.ActionKind_EXPIRATION_DAYS
|
||||
case s3lifecycle.ActionKindExpirationDate:
|
||||
return s3_lifecycle_pb.ActionKind_EXPIRATION_DATE
|
||||
case s3lifecycle.ActionKindNoncurrentDays:
|
||||
return s3_lifecycle_pb.ActionKind_NONCURRENT_DAYS
|
||||
case s3lifecycle.ActionKindNewerNoncurrent:
|
||||
return s3_lifecycle_pb.ActionKind_NEWER_NONCURRENT
|
||||
case s3lifecycle.ActionKindAbortMPU:
|
||||
return s3_lifecycle_pb.ActionKind_ABORT_MPU
|
||||
case s3lifecycle.ActionKindExpiredDeleteMarker:
|
||||
return s3_lifecycle_pb.ActionKind_EXPIRED_DELETE_MARKER
|
||||
}
|
||||
return s3_lifecycle_pb.ActionKind_ACTION_KIND_UNSPECIFIED
|
||||
}
|
||||
|
||||
func toProtoIdentity(id *router.EntryIdentity) *s3_lifecycle_pb.EntryIdentity {
|
||||
if id == nil {
|
||||
return nil
|
||||
}
|
||||
return &s3_lifecycle_pb.EntryIdentity{
|
||||
MtimeNs: id.MtimeNs,
|
||||
Size: id.Size,
|
||||
HeadFid: id.HeadFid,
|
||||
ExtendedHash: id.ExtendedHash,
|
||||
}
|
||||
}
|
||||
164
weed/s3api/s3lifecycle/dailyrun/dispatch_test.go
Normal file
164
weed/s3api/s3lifecycle/dailyrun/dispatch_test.go
Normal file
@ -0,0 +1,164 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// fakeLifecycleClient is the test double for LifecycleClient. It
|
||||
// returns a scripted sequence of (outcome, error) pairs and counts
|
||||
// invocations so retry behavior can be pinned exactly.
|
||||
type fakeLifecycleClient struct {
|
||||
scripted []scriptedResp
|
||||
calls int32
|
||||
}
|
||||
|
||||
type scriptedResp struct {
|
||||
outcome s3_lifecycle_pb.LifecycleDeleteOutcome
|
||||
err error
|
||||
}
|
||||
|
||||
func (c *fakeLifecycleClient) LifecycleDelete(_ context.Context, _ *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
||||
idx := int(atomic.AddInt32(&c.calls, 1)) - 1
|
||||
if idx >= len(c.scripted) {
|
||||
// Default to the last scripted response if the test under-specifies.
|
||||
idx = len(c.scripted) - 1
|
||||
}
|
||||
r := c.scripted[idx]
|
||||
if r.err != nil {
|
||||
return nil, r.err
|
||||
}
|
||||
return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: r.outcome}, nil
|
||||
}
|
||||
|
||||
func sampleMatch() router.Match {
|
||||
return router.Match{
|
||||
Key: s3lifecycle.ActionKey{Bucket: "b", ActionKind: s3lifecycle.ActionKindExpirationDays},
|
||||
Bucket: "b",
|
||||
ObjectKey: "obj",
|
||||
}
|
||||
}
|
||||
|
||||
// Speed up dispatch retries inside tests so the suite stays fast.
|
||||
// The defaults (200ms initial, 5s max) make exponential backoff cases
|
||||
// take seconds; tests shrink to microseconds via a separate helper.
|
||||
func dispatchWithRetryFast(t *testing.T, ctx context.Context, c LifecycleClient, m router.Match) (s3_lifecycle_pb.LifecycleDeleteOutcome, error) {
|
||||
t.Helper()
|
||||
// We can't override the package-private constants, but every test
|
||||
// path here uses small attempt counts so even the production
|
||||
// backoff is fine.
|
||||
return dispatchWithRetry(ctx, c, m)
|
||||
}
|
||||
|
||||
func TestDispatch_FirstAttemptSucceeds(t *testing.T) {
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE},
|
||||
}}
|
||||
out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, out)
|
||||
assert.Equal(t, int32(1), c.calls)
|
||||
}
|
||||
|
||||
func TestDispatch_TransportRetryThenSucceed(t *testing.T) {
|
||||
// Two transport flakes then a success. The retry loop tries 3 times
|
||||
// by default; we expect 3 calls and the final outcome.
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{err: errors.New("transport boom")},
|
||||
{err: errors.New("transport boom")},
|
||||
{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED},
|
||||
}}
|
||||
out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED, out)
|
||||
assert.Equal(t, int32(3), c.calls)
|
||||
}
|
||||
|
||||
func TestDispatch_ExhaustsRetriesReturnsError(t *testing.T) {
|
||||
// All N attempts fail with transport errors. Caller must see an
|
||||
// error and the call count must equal transportRetryAttempts.
|
||||
failing := errors.New("transport persistent")
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{err: failing}, {err: failing}, {err: failing},
|
||||
}}
|
||||
_, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, int32(transportRetryAttempts), c.calls)
|
||||
}
|
||||
|
||||
func TestDispatch_ServerOutcomeRetryLaterIsNotRetried(t *testing.T) {
|
||||
// RETRY_LATER is a SERVER outcome; the daily run's halt-on-failure
|
||||
// caller handles it. dispatchWithRetry must surface it on the first
|
||||
// successful RPC and NOT retry in-run.
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER},
|
||||
}}
|
||||
out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER, out)
|
||||
assert.Equal(t, int32(1), c.calls, "server outcome must not trigger transport retry")
|
||||
}
|
||||
|
||||
func TestDispatch_ServerOutcomeBlockedNotRetried(t *testing.T) {
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED},
|
||||
}}
|
||||
out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, out)
|
||||
assert.Equal(t, int32(1), c.calls)
|
||||
}
|
||||
|
||||
func TestDispatch_ContextCancelledShortCircuits(t *testing.T) {
|
||||
// Context cancellation must surface immediately, not consume the
|
||||
// retry budget. Cursor-staying semantics in the caller depends on
|
||||
// this distinction.
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
c := &fakeLifecycleClient{scripted: []scriptedResp{
|
||||
{err: context.Canceled},
|
||||
}}
|
||||
_, err := dispatchWithRetryFast(t, ctx, c, sampleMatch())
|
||||
require.ErrorIs(t, err, context.Canceled)
|
||||
assert.Equal(t, int32(1), c.calls, "context cancellation must not consume the retry budget")
|
||||
}
|
||||
|
||||
func TestBuildDeleteRequest_RuleHashAndIdentity(t *testing.T) {
|
||||
// Pin the request shape so a future Match-struct refactor doesn't
|
||||
// silently drop the identity-CAS witness or the rule hash slice.
|
||||
m := router.Match{
|
||||
Bucket: "bucket",
|
||||
ObjectKey: "obj.txt",
|
||||
VersionID: "v_1",
|
||||
Key: s3lifecycle.ActionKey{
|
||||
Bucket: "bucket",
|
||||
RuleHash: [8]byte{1, 2, 3, 4, 5, 6, 7, 8},
|
||||
ActionKind: s3lifecycle.ActionKindNoncurrentDays,
|
||||
},
|
||||
Identity: &router.EntryIdentity{
|
||||
MtimeNs: time.Unix(1700000000, 12345).UnixNano(),
|
||||
Size: 42,
|
||||
HeadFid: "1,abc",
|
||||
ExtendedHash: []byte{0xaa, 0xbb},
|
||||
},
|
||||
}
|
||||
req := buildDeleteRequest(m)
|
||||
assert.Equal(t, "bucket", req.Bucket)
|
||||
assert.Equal(t, "obj.txt", req.ObjectPath)
|
||||
assert.Equal(t, "v_1", req.VersionId)
|
||||
assert.Equal(t, []byte{1, 2, 3, 4, 5, 6, 7, 8}, req.RuleHash)
|
||||
assert.Equal(t, s3_lifecycle_pb.ActionKind_NONCURRENT_DAYS, req.ActionKind)
|
||||
require.NotNil(t, req.ExpectedIdentity)
|
||||
assert.Equal(t, int64(42), req.ExpectedIdentity.Size)
|
||||
assert.Equal(t, "1,abc", req.ExpectedIdentity.HeadFid)
|
||||
assert.Equal(t, []byte{0xaa, 0xbb}, req.ExpectedIdentity.ExtendedHash)
|
||||
}
|
||||
152
weed/s3api/s3lifecycle/dailyrun/process_matches_test.go
Normal file
152
weed/s3api/s3lifecycle/dailyrun/process_matches_test.go
Normal file
@ -0,0 +1,152 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// recordingClient captures every LifecycleDelete request the
|
||||
// daily-run path emits. Default outcome is DONE; tests that need
|
||||
// other outcomes set responses by index.
|
||||
type recordingClient struct {
|
||||
mu sync.Mutex
|
||||
requests []*s3_lifecycle_pb.LifecycleDeleteRequest
|
||||
responses []s3_lifecycle_pb.LifecycleDeleteOutcome
|
||||
calls atomic.Int32
|
||||
}
|
||||
|
||||
func (c *recordingClient) LifecycleDelete(_ context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
||||
c.mu.Lock()
|
||||
c.requests = append(c.requests, req)
|
||||
idx := int(c.calls.Add(1)) - 1
|
||||
c.mu.Unlock()
|
||||
out := s3_lifecycle_pb.LifecycleDeleteOutcome_DONE
|
||||
if idx < len(c.responses) {
|
||||
out = c.responses[idx]
|
||||
}
|
||||
return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: out}, nil
|
||||
}
|
||||
|
||||
func (c *recordingClient) seenObjects() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
out := make([]string, 0, len(c.requests))
|
||||
for _, r := range c.requests {
|
||||
out = append(out, r.ObjectPath)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Phase 2's processMatches must NOT use a per-rule done flag.
|
||||
// routePointerTransitionExpand can return multiple matches for the
|
||||
// same ActionKey on different objects, each with its own
|
||||
// SuccessorModTime from a distinct demoting event. A not-yet-due
|
||||
// sibling must never gate a sibling that's already past its DueTime.
|
||||
// Pin that here.
|
||||
func TestProcessMatches_NotYetDueDoesNotSuppressDueOnSameRule(t *testing.T) {
|
||||
runNow := time.Now()
|
||||
rh := [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
|
||||
rule := s3lifecycle.ActionKey{
|
||||
Bucket: "b",
|
||||
RuleHash: rh,
|
||||
ActionKind: s3lifecycle.ActionKindNoncurrentDays,
|
||||
}
|
||||
// Three matches for the same ActionKey, different objects, mixed
|
||||
// due-times. Order intentionally puts the not-yet-due match first
|
||||
// so a buggy per-rule done flag would suppress the others.
|
||||
matches := []router.Match{
|
||||
{Key: rule, Bucket: "b", ObjectKey: "future-sibling", DueTime: runNow.Add(48 * time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "due-sibling-a", DueTime: runNow.Add(-48 * time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "due-sibling-b", DueTime: runNow.Add(-24 * time.Hour)},
|
||||
}
|
||||
client := &recordingClient{}
|
||||
cfg := Config{Client: client}
|
||||
skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{TsNs: runNow.UnixNano()}, matches)
|
||||
require.NoError(t, err)
|
||||
require.False(t, halted)
|
||||
require.True(t, skipped, "at least one future-DueTime match must flag skippedAny")
|
||||
|
||||
// Both due siblings must have been dispatched.
|
||||
seen := client.seenObjects()
|
||||
assert.ElementsMatch(t, []string{"due-sibling-a", "due-sibling-b"}, seen,
|
||||
"due matches must dispatch regardless of where the future-sibling sits in the slice")
|
||||
}
|
||||
|
||||
func TestProcessMatches_OrderingDoesNotMatter(t *testing.T) {
|
||||
runNow := time.Now()
|
||||
rh := [8]byte{0xaa}
|
||||
rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
|
||||
matches := []router.Match{
|
||||
{Key: rule, Bucket: "b", ObjectKey: "a", DueTime: runNow.Add(-2 * time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "b", DueTime: runNow.Add(-1 * time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "future", DueTime: runNow.Add(time.Hour)},
|
||||
}
|
||||
client := &recordingClient{}
|
||||
cfg := Config{Client: client}
|
||||
skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
|
||||
require.NoError(t, err)
|
||||
require.False(t, halted)
|
||||
require.True(t, skipped)
|
||||
assert.ElementsMatch(t, []string{"a", "b"}, client.seenObjects())
|
||||
}
|
||||
|
||||
func TestProcessMatches_HaltOnServerOutcomeStopsRemaining(t *testing.T) {
|
||||
runNow := time.Now()
|
||||
rh := [8]byte{0xbb}
|
||||
rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
|
||||
matches := []router.Match{
|
||||
{Key: rule, Bucket: "b", ObjectKey: "ok", DueTime: runNow.Add(-time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "blocked", DueTime: runNow.Add(-time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "would-be-next", DueTime: runNow.Add(-time.Hour)},
|
||||
}
|
||||
client := &recordingClient{responses: []s3_lifecycle_pb.LifecycleDeleteOutcome{
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED,
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, // would be a bug if reached
|
||||
}}
|
||||
cfg := Config{Client: client}
|
||||
_, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
|
||||
require.NoError(t, err)
|
||||
require.True(t, halted, "BLOCKED outcome must halt the event loop")
|
||||
assert.Equal(t, []string{"ok", "blocked"}, client.seenObjects(),
|
||||
"would-be-next must NOT be dispatched after the halt")
|
||||
}
|
||||
|
||||
func TestProcessMatches_EmptyMatchesIsNoop(t *testing.T) {
|
||||
runNow := time.Now()
|
||||
client := &recordingClient{}
|
||||
cfg := Config{Client: client}
|
||||
skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, nil)
|
||||
require.NoError(t, err)
|
||||
require.False(t, halted)
|
||||
require.False(t, skipped)
|
||||
assert.Empty(t, client.seenObjects())
|
||||
}
|
||||
|
||||
func TestProcessMatches_AllDueNoSkippedFlag(t *testing.T) {
|
||||
// All matches past their DueTime — skippedAny must be false so the
|
||||
// caller can advance the cursor past this event.
|
||||
runNow := time.Now()
|
||||
rh := [8]byte{0xcc}
|
||||
rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
|
||||
matches := []router.Match{
|
||||
{Key: rule, Bucket: "b", ObjectKey: "a", DueTime: runNow.Add(-time.Hour)},
|
||||
{Key: rule, Bucket: "b", ObjectKey: "b", DueTime: runNow.Add(-30 * time.Minute)},
|
||||
}
|
||||
client := &recordingClient{}
|
||||
cfg := Config{Client: client}
|
||||
skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
|
||||
require.NoError(t, err)
|
||||
require.False(t, halted)
|
||||
require.False(t, skipped, "every match was past DueTime; skippedAny must be false")
|
||||
}
|
||||
196
weed/s3api/s3lifecycle/dailyrun/replayability.go
Normal file
196
weed/s3api/s3lifecycle/dailyrun/replayability.go
Normal file
@ -0,0 +1,196 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
||||
)
|
||||
|
||||
// UnsupportedRuleError is returned by Run when the bucket's compiled
|
||||
// rules include any action kind Phase 2 cannot service. The handler
|
||||
// surfaces this so admin marks the run failed in the activity log;
|
||||
// flipping the algorithm flag to daily_replay on a bucket with these
|
||||
// rules is a loud failure rather than a silent dropped rule.
|
||||
type UnsupportedRuleError struct {
|
||||
Bucket string
|
||||
Kind s3lifecycle.ActionKind
|
||||
Reason string
|
||||
}
|
||||
|
||||
func (e *UnsupportedRuleError) Error() string {
|
||||
return fmt.Sprintf("daily_replay: unsupported action kind %s on bucket %q: %s", e.Kind, e.Bucket, e.Reason)
|
||||
}
|
||||
|
||||
// IsUnsupportedRule reports whether err is or wraps an
|
||||
// UnsupportedRuleError. Callers (the worker handler) use this to
|
||||
// classify the run outcome.
|
||||
func IsUnsupportedRule(err error) bool {
|
||||
var u *UnsupportedRuleError
|
||||
return errors.As(err, &u)
|
||||
}
|
||||
|
||||
// isReplayEligibleKind reports whether the engine's daily-replay path
|
||||
// can service action kind k. Mirror this list when adding new kinds.
|
||||
func isReplayEligibleKind(k s3lifecycle.ActionKind) bool {
|
||||
switch k {
|
||||
case s3lifecycle.ActionKindExpirationDays,
|
||||
s3lifecycle.ActionKindNoncurrentDays,
|
||||
s3lifecycle.ActionKindAbortMPU:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// checkSnapshotForUnsupported walks every active CompiledAction in snap
|
||||
// and returns the first kind that isn't replay-eligible OR is in a Mode
|
||||
// other than ModeEventDriven. router.Route gates dispatch on
|
||||
// `Mode == ModeEventDriven` (see weed/s3api/s3lifecycle/router/router.go),
|
||||
// so a replay-kind action that's been promoted to ModeScanOnly would
|
||||
// silently get no matches at all — the daily run must reject it loudly
|
||||
// so admin sees the failure in the activity log. Phase 4 partitions
|
||||
// these into walk-bound actions and runs them through the walker,
|
||||
// removing the gate.
|
||||
func checkSnapshotForUnsupported(snap *engine.Snapshot) *UnsupportedRuleError {
|
||||
if snap == nil {
|
||||
return nil
|
||||
}
|
||||
for _, a := range snap.AllActions() {
|
||||
if a == nil || !a.IsActive() {
|
||||
continue
|
||||
}
|
||||
if !isReplayEligibleKind(a.Key.ActionKind) {
|
||||
return &UnsupportedRuleError{
|
||||
Bucket: a.Bucket,
|
||||
Kind: a.Key.ActionKind,
|
||||
Reason: "Phase 2 only routes ExpirationDays / NoncurrentDays / AbortMPU; ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent land in Phase 4",
|
||||
}
|
||||
}
|
||||
if a.Mode != engine.ModeEventDriven {
|
||||
return &UnsupportedRuleError{
|
||||
Bucket: a.Bucket,
|
||||
Kind: a.Key.ActionKind,
|
||||
Reason: fmt.Sprintf("action is in Mode=%v (router.Route only dispatches ModeEventDriven); scan_only promotions land in Phase 4", a.Mode),
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// localReplayContentHash hashes the rule definitions of replay-eligible
|
||||
// actions in snap. Stable across reorderings — actions are sorted by
|
||||
// (bucket, rule_hash, action_kind) before mixing. Phase 4 replaces
|
||||
// this with engine.ReplayContentHash; both must produce the same value
|
||||
// on a snapshot that's already fully replay-eligible (which Phase 2
|
||||
// enforces via checkSnapshotForUnsupported).
|
||||
//
|
||||
// Includes the effective TTL in the hash so a TTL change (e.g. 30 → 60
|
||||
// days) is detected as a rule-content change, even though the rule's
|
||||
// RuleHash also captures it — defense in depth against any future
|
||||
// RuleHash collision and an explicit dependency for the cursor
|
||||
// rewind decision.
|
||||
func localReplayContentHash(snap *engine.Snapshot) [32]byte {
|
||||
if snap == nil {
|
||||
return [32]byte{}
|
||||
}
|
||||
type rec struct {
|
||||
bucket string
|
||||
ruleHash [8]byte
|
||||
actionKind s3lifecycle.ActionKind
|
||||
ttlNs int64
|
||||
}
|
||||
var recs []rec
|
||||
for _, a := range snap.AllActions() {
|
||||
if a == nil || !a.IsActive() {
|
||||
continue
|
||||
}
|
||||
if !isReplayEligibleKind(a.Key.ActionKind) {
|
||||
continue
|
||||
}
|
||||
recs = append(recs, rec{
|
||||
bucket: a.Bucket,
|
||||
ruleHash: a.Key.RuleHash,
|
||||
actionKind: a.Key.ActionKind,
|
||||
ttlNs: int64(effectiveTTL(a)),
|
||||
})
|
||||
}
|
||||
if len(recs) == 0 {
|
||||
return [32]byte{}
|
||||
}
|
||||
sort.Slice(recs, func(i, j int) bool {
|
||||
if recs[i].bucket != recs[j].bucket {
|
||||
return recs[i].bucket < recs[j].bucket
|
||||
}
|
||||
if recs[i].ruleHash != recs[j].ruleHash {
|
||||
for k := 0; k < len(recs[i].ruleHash); k++ {
|
||||
if recs[i].ruleHash[k] != recs[j].ruleHash[k] {
|
||||
return recs[i].ruleHash[k] < recs[j].ruleHash[k]
|
||||
}
|
||||
}
|
||||
}
|
||||
return int(recs[i].actionKind) < int(recs[j].actionKind)
|
||||
})
|
||||
h := sha256.New()
|
||||
var scratch [16]byte
|
||||
for _, r := range recs {
|
||||
h.Write([]byte(r.bucket))
|
||||
h.Write([]byte{0})
|
||||
h.Write(r.ruleHash[:])
|
||||
binary.LittleEndian.PutUint64(scratch[:8], uint64(r.actionKind))
|
||||
binary.LittleEndian.PutUint64(scratch[8:], uint64(r.ttlNs))
|
||||
h.Write(scratch[:])
|
||||
}
|
||||
var out [32]byte
|
||||
copy(out[:], h.Sum(nil))
|
||||
return out
|
||||
}
|
||||
|
||||
// effectiveTTL returns the per-action TTL the lifecycle engine clocks
|
||||
// against. Walker-bound kinds (ExpirationDate, ExpiredDeleteMarker,
|
||||
// NewerNoncurrent) return 0 — they don't participate in the replay
|
||||
// sliding window so they don't contribute to MaxEffectiveTTL or the
|
||||
// content hash. Phase 2 guarantees those kinds never reach here via
|
||||
// checkSnapshotForUnsupported, but the switch stays exhaustive so a
|
||||
// future kind addition is a compile-time prompt to decide.
|
||||
func effectiveTTL(a *engine.CompiledAction) time.Duration {
|
||||
if a == nil || a.Rule == nil {
|
||||
return 0
|
||||
}
|
||||
switch a.Key.ActionKind {
|
||||
case s3lifecycle.ActionKindExpirationDays:
|
||||
return s3lifecycle.DaysToDuration(a.Rule.ExpirationDays)
|
||||
case s3lifecycle.ActionKindNoncurrentDays:
|
||||
return s3lifecycle.DaysToDuration(a.Rule.NoncurrentVersionExpirationDays)
|
||||
case s3lifecycle.ActionKindAbortMPU:
|
||||
return s3lifecycle.DaysToDuration(a.Rule.AbortMPUDaysAfterInitiation)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// localMaxEffectiveTTL returns the largest effective TTL across active
|
||||
// replay-eligible actions. Returns zero when snap is empty/nil — caller
|
||||
// is responsible for routing through the empty-replay branch in that
|
||||
// case. Phase 4 replaces this with engine.MaxEffectiveTTL.
|
||||
func localMaxEffectiveTTL(snap *engine.Snapshot) time.Duration {
|
||||
if snap == nil {
|
||||
return 0
|
||||
}
|
||||
var max time.Duration
|
||||
for _, a := range snap.AllActions() {
|
||||
if a == nil || !a.IsActive() {
|
||||
continue
|
||||
}
|
||||
if !isReplayEligibleKind(a.Key.ActionKind) {
|
||||
continue
|
||||
}
|
||||
if d := effectiveTTL(a); d > max {
|
||||
max = d
|
||||
}
|
||||
}
|
||||
return max
|
||||
}
|
||||
176
weed/s3api/s3lifecycle/dailyrun/replayability_test.go
Normal file
176
weed/s3api/s3lifecycle/dailyrun/replayability_test.go
Normal file
@ -0,0 +1,176 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func newSnapshotWith(t *testing.T, inputs []engine.CompileInput) *engine.Snapshot {
|
||||
t.Helper()
|
||||
e := engine.New()
|
||||
e.Compile(inputs, engine.CompileOptions{})
|
||||
snap := e.Snapshot()
|
||||
// Mark every action active so isActive checks fire as expected; the
|
||||
// production compile path also marks them active for replay-eligible
|
||||
// kinds when prior.BootstrapComplete is true. Tests just need them
|
||||
// visible to AllActions().
|
||||
for _, a := range snap.AllActions() {
|
||||
snap.MarkActive(a.Key)
|
||||
}
|
||||
return snap
|
||||
}
|
||||
|
||||
func ruleExpirationDays(days int) *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: days}
|
||||
}
|
||||
|
||||
func ruleExpirationDate(t time.Time) *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r-date", Status: s3lifecycle.StatusEnabled, ExpirationDate: t}
|
||||
}
|
||||
|
||||
func ruleNoncurrentDays(days int) *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r-nc", Status: s3lifecycle.StatusEnabled, NoncurrentVersionExpirationDays: days}
|
||||
}
|
||||
|
||||
func ruleAbortMPU(days int) *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r-mpu", Status: s3lifecycle.StatusEnabled, AbortMPUDaysAfterInitiation: days}
|
||||
}
|
||||
|
||||
func ruleNewerNoncurrent(n int) *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r-newer", Status: s3lifecycle.StatusEnabled, NewerNoncurrentVersions: n}
|
||||
}
|
||||
|
||||
func ruleExpiredDeleteMarker() *s3lifecycle.Rule {
|
||||
return &s3lifecycle.Rule{ID: "r-edm", Status: s3lifecycle.StatusEnabled, ExpiredObjectDeleteMarker: true}
|
||||
}
|
||||
|
||||
func TestCheckSnapshotForUnsupported_AllReplayKindsAccepted(t *testing.T) {
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{
|
||||
ruleExpirationDays(30),
|
||||
ruleNoncurrentDays(7),
|
||||
ruleAbortMPU(7),
|
||||
}},
|
||||
})
|
||||
require.Nil(t, checkSnapshotForUnsupported(snap))
|
||||
}
|
||||
|
||||
func TestCheckSnapshotForUnsupported_ExpirationDateRejected(t *testing.T) {
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleExpirationDate(time.Now().Add(48 * time.Hour))}},
|
||||
})
|
||||
err := checkSnapshotForUnsupported(snap)
|
||||
require.NotNil(t, err)
|
||||
assert.Equal(t, s3lifecycle.ActionKindExpirationDate, err.Kind)
|
||||
assert.Equal(t, "b1", err.Bucket)
|
||||
}
|
||||
|
||||
func TestCheckSnapshotForUnsupported_NewerNoncurrentRejected(t *testing.T) {
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleNewerNoncurrent(2)}},
|
||||
})
|
||||
err := checkSnapshotForUnsupported(snap)
|
||||
require.NotNil(t, err)
|
||||
assert.Equal(t, s3lifecycle.ActionKindNewerNoncurrent, err.Kind)
|
||||
}
|
||||
|
||||
func TestCheckSnapshotForUnsupported_ExpiredDeleteMarkerRejected(t *testing.T) {
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleExpiredDeleteMarker()}},
|
||||
})
|
||||
err := checkSnapshotForUnsupported(snap)
|
||||
require.NotNil(t, err)
|
||||
assert.Equal(t, s3lifecycle.ActionKindExpiredDeleteMarker, err.Kind)
|
||||
}
|
||||
|
||||
func TestCheckSnapshotForUnsupported_NonEventDrivenModeRejected(t *testing.T) {
|
||||
// A replay-eligible action whose Mode isn't ModeEventDriven (e.g.
|
||||
// promoted to ModeScanOnly by retention checks) is silently
|
||||
// ignored by router.Route. Phase 2 must catch this loudly. Build a
|
||||
// snapshot with one ExpirationDays action and mutate its Mode to
|
||||
// ModeScanOnly directly — there's no production path that produces
|
||||
// this without retention plumbing we don't have here, but the gate
|
||||
// must still reject it.
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleExpirationDays(30)}},
|
||||
})
|
||||
// Mutate the compiled action to ModeScanOnly. AllActions returns
|
||||
// the live snapshot's actions; this is safe in a test where no
|
||||
// concurrent worker is reading.
|
||||
for _, a := range snap.AllActions() {
|
||||
if a.Key.ActionKind == s3lifecycle.ActionKindExpirationDays {
|
||||
a.Mode = engine.ModeScanOnly
|
||||
}
|
||||
}
|
||||
err := checkSnapshotForUnsupported(snap)
|
||||
require.NotNil(t, err, "ModeScanOnly on a replay-kind action must be rejected")
|
||||
assert.Equal(t, s3lifecycle.ActionKindExpirationDays, err.Kind)
|
||||
assert.Contains(t, err.Reason, "ModeEventDriven")
|
||||
}
|
||||
|
||||
func TestIsUnsupportedRule_TypeCheck(t *testing.T) {
|
||||
var u error = &UnsupportedRuleError{Bucket: "b", Kind: s3lifecycle.ActionKindExpirationDate, Reason: "x"}
|
||||
assert.True(t, IsUnsupportedRule(u))
|
||||
assert.False(t, IsUnsupportedRule(nil))
|
||||
assert.False(t, IsUnsupportedRule(assertNonNilError()))
|
||||
}
|
||||
|
||||
// assertNonNilError returns a plain non-nil error of a different type
|
||||
// so IsUnsupportedRule's errors.As check has a negative case to refuse.
|
||||
func assertNonNilError() error { return errPlain }
|
||||
|
||||
type plainErr struct{}
|
||||
|
||||
func (plainErr) Error() string { return "plain" }
|
||||
|
||||
var errPlain = plainErr{}
|
||||
|
||||
func TestLocalReplayContentHash_StableAcrossReorderings(t *testing.T) {
|
||||
// Compile the same rules in two snapshots and verify the hash is
|
||||
// identical — sort order in AllActions() is implementation detail
|
||||
// and must not affect the cursor's content hash.
|
||||
rules := []*s3lifecycle.Rule{ruleExpirationDays(30), ruleNoncurrentDays(7), ruleAbortMPU(7)}
|
||||
snapA := newSnapshotWith(t, []engine.CompileInput{{Bucket: "b1", Rules: rules}})
|
||||
// Build snapB with the same rules in reverse order; hash must match.
|
||||
rev := []*s3lifecycle.Rule{rules[2], rules[1], rules[0]}
|
||||
snapB := newSnapshotWith(t, []engine.CompileInput{{Bucket: "b1", Rules: rev}})
|
||||
assert.Equal(t, localReplayContentHash(snapA), localReplayContentHash(snapB))
|
||||
}
|
||||
|
||||
func TestLocalReplayContentHash_ChangesOnTTLEdit(t *testing.T) {
|
||||
snap30 := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleExpirationDays(30)}},
|
||||
})
|
||||
snap60 := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{ruleExpirationDays(60)}},
|
||||
})
|
||||
assert.NotEqual(t, localReplayContentHash(snap30), localReplayContentHash(snap60))
|
||||
}
|
||||
|
||||
func TestLocalReplayContentHash_EmptyIsZero(t *testing.T) {
|
||||
snap := newSnapshotWith(t, nil)
|
||||
var zero [32]byte
|
||||
assert.Equal(t, zero, localReplayContentHash(snap))
|
||||
}
|
||||
|
||||
func TestLocalMaxEffectiveTTL_PicksLargest(t *testing.T) {
|
||||
snap := newSnapshotWith(t, []engine.CompileInput{
|
||||
{Bucket: "b1", Rules: []*s3lifecycle.Rule{
|
||||
ruleExpirationDays(7),
|
||||
ruleNoncurrentDays(30),
|
||||
ruleAbortMPU(14),
|
||||
}},
|
||||
})
|
||||
got := localMaxEffectiveTTL(snap)
|
||||
assert.Equal(t, s3lifecycle.DaysToDuration(30), got)
|
||||
}
|
||||
|
||||
func TestLocalMaxEffectiveTTL_EmptyReturnsZero(t *testing.T) {
|
||||
snap := newSnapshotWith(t, nil)
|
||||
assert.Equal(t, time.Duration(0), localMaxEffectiveTTL(snap))
|
||||
}
|
||||
435
weed/s3api/s3lifecycle/dailyrun/run.go
Normal file
435
weed/s3api/s3lifecycle/dailyrun/run.go
Normal file
@ -0,0 +1,435 @@
|
||||
package dailyrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
||||
"github.com/seaweedfs/seaweedfs/weed/util"
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
// LifecycleClient is the same RPC contract the streaming dispatcher
|
||||
// uses. Replicated here to avoid an import cycle with dispatcher; both
|
||||
// shapes target s3_lifecycle_pb.SeaweedS3LifecycleInternalClient and
|
||||
// neither owns the protobuf interface.
|
||||
type LifecycleClient interface {
|
||||
LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error)
|
||||
}
|
||||
|
||||
// Config bundles everything daily_run needs to make one shard-fan-out
|
||||
// pass over the meta-log. Fields with zero values use documented
|
||||
// defaults; required fields are checked at the top of Run.
|
||||
type Config struct {
|
||||
Shards []int
|
||||
BucketsPath string
|
||||
Engine *engine.Engine
|
||||
FilerClient filer_pb.SeaweedFilerClient
|
||||
Client LifecycleClient
|
||||
Persister CursorPersister
|
||||
Lister router.SiblingLister
|
||||
|
||||
// Workers caps how many shards run in parallel. Each shard owns its
|
||||
// own meta-log subscription and rate limiter is shared across all
|
||||
// of them, so the cap is about filer-side concurrency rather than
|
||||
// throughput. Zero or negative → 1 (serial, the legacy default).
|
||||
Workers int
|
||||
|
||||
// Limiter is the optional per-worker rate.Limiter shared across all
|
||||
// shard goroutines on this worker. nil = no rate limit (the legacy
|
||||
// "drain as fast as the filer can" behavior). Phase 3 wires the
|
||||
// per-worker share from ClusterContext.Metadata into this field.
|
||||
Limiter *rate.Limiter
|
||||
|
||||
// ClientName / ClientID identify this worker on the meta-log
|
||||
// subscription. ClientID 0 randomizes per-run.
|
||||
ClientName string
|
||||
ClientID int32
|
||||
|
||||
// Now overrides time.Now for tests. nil = production wall clock.
|
||||
Now func() time.Time
|
||||
|
||||
// EventBudget caps how many meta-log events each shard's reader
|
||||
// consumes before returning. Zero = unbounded (the run-until-now
|
||||
// behavior — see "stop condition" below). Tests set a small value
|
||||
// to bound deterministic runs.
|
||||
EventBudget int
|
||||
}
|
||||
|
||||
// Run executes the daily replay for every shard in cfg.Shards
|
||||
// concurrently. Returns the first non-nil error from any shard; other
|
||||
// shards still run to completion so a transient filer error on one
|
||||
// shard doesn't lose progress on the others.
|
||||
//
|
||||
// Phase 2 scope: replay only. A bucket whose compiled rules require
|
||||
// walker-bound dispatch (ExpirationDate, ExpiredDeleteMarker,
|
||||
// NewerNoncurrent) or any rule promoted to scan_only fails the whole
|
||||
// run with an UnsupportedRuleError — the handler reports this so admin
|
||||
// can either revert algorithm=streaming or wait for Phase 4.
|
||||
func Run(ctx context.Context, cfg Config) error {
|
||||
if err := validate(cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
now := cfg.Now
|
||||
if now == nil {
|
||||
now = func() time.Time { return time.Now().UTC() }
|
||||
}
|
||||
// Freeze "now" at the start of the run. Every shard, every match's
|
||||
// DueTime comparison, and every cursor-floor calculation uses the
|
||||
// same instant so a long-running shard doesn't see the boundary
|
||||
// drift relative to a short-running peer.
|
||||
runNow := now()
|
||||
|
||||
// Refuse runs whose snapshot includes a walker-bound action kind
|
||||
// or any rule the router won't route. Done once against an
|
||||
// immutable snapshot — every shard reuses this exact value so a
|
||||
// mid-execution Compile can't make shards disagree about the rule
|
||||
// set or the hash.
|
||||
snap := cfg.Engine.Snapshot()
|
||||
if unsupported := checkSnapshotForUnsupported(snap); unsupported != nil {
|
||||
return unsupported
|
||||
}
|
||||
|
||||
// Concurrency cap. cfg.Workers controls how many shards run in
|
||||
// parallel; the rate limiter (Phase 3) governs throughput. With
|
||||
// Workers=1 (the legacy default) the 16 shards process serially.
|
||||
workers := cfg.Workers
|
||||
if workers <= 0 {
|
||||
workers = 1
|
||||
}
|
||||
if workers > len(cfg.Shards) {
|
||||
workers = len(cfg.Shards)
|
||||
}
|
||||
sem := make(chan struct{}, workers)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
errCh := make(chan error, len(cfg.Shards))
|
||||
for _, sh := range cfg.Shards {
|
||||
sh := sh
|
||||
wg.Add(1)
|
||||
sem <- struct{}{}
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }()
|
||||
if err := runShard(ctx, cfg, snap, runNow, sh); err != nil {
|
||||
errCh <- err
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errCh)
|
||||
// Return the first error; the rest are logged at glog level so
|
||||
// they're recoverable in the operational stream.
|
||||
var first error
|
||||
for err := range errCh {
|
||||
if first == nil {
|
||||
first = err
|
||||
} else {
|
||||
glog.V(1).Infof("daily_run: additional shard error: %v", err)
|
||||
}
|
||||
}
|
||||
return first
|
||||
}
|
||||
|
||||
func validate(cfg Config) error {
|
||||
if cfg.Engine == nil {
|
||||
return errors.New("daily_run: nil Engine")
|
||||
}
|
||||
if cfg.FilerClient == nil {
|
||||
return errors.New("daily_run: nil FilerClient")
|
||||
}
|
||||
if cfg.Client == nil {
|
||||
return errors.New("daily_run: nil Client (LifecycleDelete RPC)")
|
||||
}
|
||||
if cfg.Persister == nil {
|
||||
return errors.New("daily_run: nil Persister")
|
||||
}
|
||||
if cfg.Lister == nil {
|
||||
return errors.New("daily_run: nil Lister")
|
||||
}
|
||||
if cfg.BucketsPath == "" {
|
||||
return errors.New("daily_run: empty BucketsPath")
|
||||
}
|
||||
for _, sh := range cfg.Shards {
|
||||
if sh < 0 || sh >= s3lifecycle.ShardCount {
|
||||
return fmt.Errorf("daily_run: shard %d out of [0, %d)", sh, s3lifecycle.ShardCount)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// runShard executes the daily replay loop for a single shard. The
|
||||
// algorithm is documented in DESIGN.md ("Algorithm" section).
|
||||
//
|
||||
// Phase 2 omissions vs. the full design:
|
||||
// - No walker invocation on rule-change / cold-start / retention
|
||||
// loss. The cursor is rewritten and the worker exits; Phase 4
|
||||
// wires the walker over engine.RecoveryView(snap) on these
|
||||
// branches.
|
||||
// - PromotedHash is always the empty hash. The partition-flip
|
||||
// trigger is dormant until Phase 4.
|
||||
// - All walker-bound action kinds and scan_only-promoted rules are
|
||||
// refused at validate-time (see checkSnapshotForUnsupported), so
|
||||
// replay only ever sees ExpirationDays / NoncurrentDays / AbortMPU.
|
||||
//
|
||||
// snap is the engine snapshot captured at the top of Run; reusing the
|
||||
// same value across shards guarantees they observe identical rules and
|
||||
// hashes. runNow is the frozen instant for due-time comparisons.
|
||||
func runShard(ctx context.Context, cfg Config, snap *engine.Snapshot, runNow time.Time, shardID int) error {
|
||||
persisted, found, err := cfg.Persister.Load(ctx, shardID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("shard=%d: load cursor: %w", shardID, err)
|
||||
}
|
||||
|
||||
rsh := localReplayContentHash(snap)
|
||||
maxTTL := localMaxEffectiveTTL(snap)
|
||||
|
||||
// Empty-replay sentinel: no replay-eligible active rules in the
|
||||
// snapshot. We persist the hash so a future rule addition is
|
||||
// detected as a content change. Use the hash rather than maxTTL
|
||||
// here as the explicit "no replay state" signal — both fire on
|
||||
// the same set in practice (action_kind.go only emits actions
|
||||
// when their Days field is > 0), but the hash captures intent.
|
||||
if rsh == [32]byte{} {
|
||||
return cfg.Persister.Save(ctx, shardID, Cursor{
|
||||
TsNs: 0,
|
||||
RuleSetHash: rsh,
|
||||
PromotedHash: [32]byte{}, // Phase 2: always empty
|
||||
})
|
||||
}
|
||||
|
||||
// Rule-change branch: hash mismatch (and a persisted cursor exists)
|
||||
// rewinds to now - max_ttl. Phase 4 adds the walker call here.
|
||||
if found && persisted.RuleSetHash != rsh {
|
||||
next := Cursor{
|
||||
TsNs: runNow.Add(-maxTTL).UnixNano(),
|
||||
RuleSetHash: rsh,
|
||||
// PromotedHash stays empty in Phase 2.
|
||||
}
|
||||
return cfg.Persister.Save(ctx, shardID, next)
|
||||
}
|
||||
|
||||
// Cold start with no prior cursor: start scanning from now - max_ttl
|
||||
// so a freshly-installed worker still expires already-due objects
|
||||
// whose PUT events sit within meta-log retention. Phase 4 would
|
||||
// invoke the walker here for the longer-than-retention case; Phase 2
|
||||
// trusts that retention >= max_ttl in the deployments this code is
|
||||
// enabled on.
|
||||
startTsNs := persisted.TsNs
|
||||
floor := runNow.Add(-maxTTL).UnixNano()
|
||||
if startTsNs < floor {
|
||||
startTsNs = floor
|
||||
}
|
||||
|
||||
lastOK, _, drainErr := drainShardEvents(ctx, cfg, runNow, shardID, snap, startTsNs)
|
||||
if drainErr != nil {
|
||||
// Drain failed (transport, ctx cancel, or limiter shutdown):
|
||||
// persist whatever progress we have so subsequent runs don't
|
||||
// re-process the events we already dispatched, then propagate
|
||||
// the error. Cursor save uses ctx — if ctx is already cancelled
|
||||
// the save will fail too; that's fine, we still surface the
|
||||
// underlying drain error.
|
||||
_ = cfg.Persister.Save(ctx, shardID, Cursor{TsNs: lastOK, RuleSetHash: rsh})
|
||||
return fmt.Errorf("shard=%d: drain: %w", shardID, drainErr)
|
||||
}
|
||||
|
||||
// drainShardEvents stops advancing lastOK at the first event with a
|
||||
// skipped (not-yet-due) match. Persisting that value means the next
|
||||
// run resumes from before the skipped event and re-scans it —
|
||||
// without that, future-DueTime matches would be lost. halted=true
|
||||
// adds nothing here because lastOK already reflects the
|
||||
// stuck-at-failure boundary.
|
||||
return cfg.Persister.Save(ctx, shardID, Cursor{
|
||||
TsNs: lastOK,
|
||||
RuleSetHash: rsh,
|
||||
// PromotedHash stays empty in Phase 2.
|
||||
})
|
||||
}
|
||||
|
||||
// drainShardEvents subscribes to the meta-log starting at startTsNs,
|
||||
// routes every event through router.Route, and dispatches each Match
|
||||
// whose due_time is past runNow. Returns (cursorAdvanceTo, halted, error):
|
||||
// - cursorAdvanceTo: the highest TsNs the persisted cursor may safely
|
||||
// advance to. Equals the TsNs of the last event whose matches were
|
||||
// ALL dispatched (DONE/NOOP_RESOLVED/SKIPPED_OBJECT_LOCK) AND every
|
||||
// prior event was likewise fully processed. Once an event with a
|
||||
// not-yet-due match is encountered, cursorAdvanceTo stops growing —
|
||||
// so subsequent runs re-scan that event and any after it.
|
||||
// - halted: true when an unresolved dispatch outcome (RETRY_LATER /
|
||||
// BLOCKED / transport error after in-run retries) stopped the loop.
|
||||
// - error: stream/setup failure or context cancellation; caller
|
||||
// persists cursorAdvanceTo and propagates the error so the run is
|
||||
// marked as interrupted rather than successful.
|
||||
func drainShardEvents(ctx context.Context, cfg Config, runNow time.Time, shardID int, snap *engine.Snapshot, startTsNs int64) (int64, bool, error) {
|
||||
clientName := cfg.ClientName
|
||||
if clientName == "" {
|
||||
clientName = "worker-s3-lifecycle-daily"
|
||||
}
|
||||
clientID := cfg.ClientID
|
||||
if clientID == 0 {
|
||||
clientID = int32(util.RandomInt32())
|
||||
}
|
||||
runUpTo := runNow.UnixNano()
|
||||
if startTsNs >= runUpTo {
|
||||
// Nothing to do — cursor already at or past the run boundary.
|
||||
return startTsNs, false, nil
|
||||
}
|
||||
|
||||
events := make(chan *reader.Event, 64)
|
||||
rd := &reader.Reader{
|
||||
ShardID: shardID,
|
||||
BucketsPath: cfg.BucketsPath,
|
||||
StartTsNs: startTsNs,
|
||||
Events: events,
|
||||
EventBudget: cfg.EventBudget,
|
||||
}
|
||||
|
||||
readerCtx, cancelReader := context.WithCancel(ctx)
|
||||
defer cancelReader()
|
||||
|
||||
readerDone := make(chan error, 1)
|
||||
go func() {
|
||||
readerDone <- rd.Run(readerCtx, cfg.FilerClient, clientName, clientID)
|
||||
}()
|
||||
|
||||
cursorAdvanceTo := startTsNs
|
||||
// stuck flips to true at the first event with a skipped (not-yet-due)
|
||||
// match. From that event onward, cursorAdvanceTo no longer rises —
|
||||
// future runs must re-scan everything from cursorAdvanceTo + 1 onward
|
||||
// so the future-due matches get re-evaluated when they age in.
|
||||
stuck := false
|
||||
halted := false
|
||||
|
||||
drain:
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
// Parent context cancellation (worker shutdown, MaxRuntime).
|
||||
// Return whatever progress was made so far and propagate the
|
||||
// error so the caller marks the run as interrupted.
|
||||
cancelReader()
|
||||
<-readerDone
|
||||
return cursorAdvanceTo, true, ctx.Err()
|
||||
case ev, ok := <-events:
|
||||
if !ok {
|
||||
break drain
|
||||
}
|
||||
if ev == nil {
|
||||
continue
|
||||
}
|
||||
if ev.TsNs > runUpTo {
|
||||
// Reached the run boundary: events past now belong to
|
||||
// tomorrow's pass. Cancel the reader so it doesn't keep
|
||||
// pulling live events, then exit.
|
||||
cancelReader()
|
||||
break drain
|
||||
}
|
||||
matches := router.Route(ctx, snap, ev, runNow, cfg.Lister)
|
||||
eventSkipped, eventHalted, eventErr := processMatches(ctx, cfg, runNow, ev, matches)
|
||||
if eventErr != nil {
|
||||
// A non-dispatch error (e.g. limiter wait cancelled by
|
||||
// shutdown). Cursor stays at the last fully-processed
|
||||
// event; surface the error so the caller treats it as
|
||||
// an interrupt.
|
||||
cancelReader()
|
||||
<-readerDone
|
||||
return cursorAdvanceTo, true, eventErr
|
||||
}
|
||||
if eventHalted {
|
||||
halted = true
|
||||
break drain
|
||||
}
|
||||
if eventSkipped {
|
||||
// First event with a not-yet-due match. Don't advance
|
||||
// the cursor past it; keep processing later events to
|
||||
// dispatch any due ones, but the persisted cursor stays
|
||||
// at the previous cursorAdvanceTo so this event is
|
||||
// re-scanned tomorrow.
|
||||
stuck = true
|
||||
continue
|
||||
}
|
||||
if !stuck {
|
||||
cursorAdvanceTo = ev.TsNs
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for reader to drain so its goroutine doesn't outlive us.
|
||||
cancelReader()
|
||||
if rerr := <-readerDone; rerr != nil && !errors.Is(rerr, context.Canceled) {
|
||||
glog.V(2).Infof("daily_run shard=%d: reader returned: %v", shardID, rerr)
|
||||
}
|
||||
return cursorAdvanceTo, halted, nil
|
||||
}
|
||||
|
||||
// processMatches dispatches every match emitted from one event. Returns
|
||||
// (skippedAny, halted, error):
|
||||
// - skippedAny=true if at least one match had a DueTime past runNow.
|
||||
// The caller must NOT advance the persisted cursor past this event
|
||||
// so the skipped match gets re-scanned in a later run.
|
||||
// - halted=true on an unresolved outcome (RETRY_LATER / BLOCKED /
|
||||
// transport error after in-run retries). Caller exits the drain
|
||||
// loop.
|
||||
// - error: propagated from limiter.Wait when ctx is cancelled.
|
||||
//
|
||||
// A future-DueTime match is silently skipped — it does NOT terminate
|
||||
// the loop or get cached as "done for this rule." A single event can
|
||||
// produce multiple matches for the same ActionKey against different
|
||||
// objects (routePointerTransitionExpand emits one match per noncurrent
|
||||
// sibling, each with its own SuccessorModTime derived from a different
|
||||
// demoting event), so a not-yet-due sibling must never gate a sibling
|
||||
// that's already past its DueTime. Without per-object state the only
|
||||
// safe behavior is per-match independence; the cursor-advance gate
|
||||
// upstream handles re-scanning the skipped events tomorrow.
|
||||
//
|
||||
// runNow is the frozen run-start instant. Using it (rather than a
|
||||
// fresh now() per call) keeps the boundary stable across all matches
|
||||
// in this run.
|
||||
func processMatches(ctx context.Context, cfg Config, runNow time.Time, ev *reader.Event, matches []router.Match) (skippedAny, halted bool, err error) {
|
||||
for _, m := range matches {
|
||||
if m.DueTime.After(runNow) {
|
||||
skippedAny = true
|
||||
continue
|
||||
}
|
||||
if cfg.Limiter != nil {
|
||||
if waitErr := cfg.Limiter.Wait(ctx); waitErr != nil {
|
||||
return skippedAny, true, waitErr
|
||||
}
|
||||
}
|
||||
outcome, dispatchErr := dispatchWithRetry(ctx, cfg.Client, m)
|
||||
if dispatchErr != nil {
|
||||
// Exhausted in-run transport retries: halt; tomorrow retries
|
||||
// from the same cursor.
|
||||
glog.V(1).Infof("daily_run: transport error on %s/%s %s: %v",
|
||||
m.Bucket, m.ObjectKey, m.Key.ActionKind, dispatchErr)
|
||||
return skippedAny, true, nil
|
||||
}
|
||||
switch outcome {
|
||||
case s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED,
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK:
|
||||
// Cursor advances. The dispatch's own metric (in the RPC
|
||||
// server) records the outcome; the daily-run side stays
|
||||
// metric-quiet to avoid double-counting.
|
||||
case s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER,
|
||||
s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED:
|
||||
glog.V(1).Infof("daily_run: %s on %s/%s %s",
|
||||
outcome, m.Bucket, m.ObjectKey, m.Key.ActionKind)
|
||||
return skippedAny, true, nil
|
||||
default:
|
||||
glog.V(1).Infof("daily_run: unknown outcome %v on %s/%s", outcome, m.Bucket, m.ObjectKey)
|
||||
return skippedAny, true, nil
|
||||
}
|
||||
_ = ev // ev kept available for future per-event logging
|
||||
}
|
||||
return skippedAny, false, nil
|
||||
}
|
||||
@ -18,6 +18,15 @@ type filerSiblingLister struct {
|
||||
bucketsPath string
|
||||
}
|
||||
|
||||
// NewFilerSiblingLister returns a router.SiblingLister backed by the
|
||||
// filer client. Exported so the daily-replay worker (and any future
|
||||
// non-dispatcher caller) can reuse the same .versions/ + null-bare
|
||||
// lookup logic without re-implementing it. Phase 5 deletes the
|
||||
// dispatcher's internal use; this constructor stays.
|
||||
func NewFilerSiblingLister(client filer_pb.SeaweedFilerClient, bucketsPath string) router.SiblingLister {
|
||||
return &filerSiblingLister{client: client, bucketsPath: bucketsPath}
|
||||
}
|
||||
|
||||
func (l *filerSiblingLister) Survivors(ctx context.Context, bucket, objectKey string) (router.Survivors, error) {
|
||||
bucketPath := strings.TrimSuffix(l.bucketsPath, "/") + "/" + bucket
|
||||
versionsDir := bucketPath + "/" + objectKey + s3_constants.VersionsFolder
|
||||
|
||||
@ -15,6 +15,19 @@ const (
|
||||
defaultRefreshIntervalMinutes = int64(5)
|
||||
defaultMaxRuntimeMinutes = int64(60)
|
||||
defaultBootstrapIntervalMinutes = int64(0) // 0 = walk once per process
|
||||
|
||||
// AlgorithmDailyReplay routes the worker through dailyrun.Run for
|
||||
// one bounded pass per Execute. Currently Phase 2 / replay-only:
|
||||
// buckets with walker-bound action kinds are refused. Phase 4
|
||||
// extends this to handle every kind. Default — the streaming path
|
||||
// stays in the tree as a runtime escape hatch only.
|
||||
AlgorithmDailyReplay = "daily_replay"
|
||||
// AlgorithmStreaming is the legacy event-driven dispatcher path
|
||||
// (reader + heap + per-shard pipeline). Kept as a fallback knob for
|
||||
// rollout; deleted by Phase 5 once Phase 4 walker integration ships.
|
||||
AlgorithmStreaming = "streaming"
|
||||
|
||||
defaultAlgorithm = AlgorithmDailyReplay
|
||||
)
|
||||
|
||||
// Config is the parsed AdminConfigForm + WorkerConfigForm view.
|
||||
@ -25,6 +38,7 @@ type Config struct {
|
||||
RefreshInterval time.Duration
|
||||
BootstrapInterval time.Duration
|
||||
MaxRuntime time.Duration
|
||||
Algorithm string
|
||||
}
|
||||
|
||||
// ParseConfig pulls the lifecycle Handler config from the merged
|
||||
@ -37,6 +51,7 @@ func ParseConfig(adminValues, workerValues map[string]*plugin_pb.ConfigValue) Co
|
||||
RefreshInterval: time.Duration(readInt64(workerValues, "refresh_interval_minutes", defaultRefreshIntervalMinutes)) * time.Minute,
|
||||
BootstrapInterval: time.Duration(readInt64(workerValues, "bootstrap_interval_minutes", defaultBootstrapIntervalMinutes)) * time.Minute,
|
||||
MaxRuntime: time.Duration(readInt64(workerValues, "max_runtime_minutes", defaultMaxRuntimeMinutes)) * time.Minute,
|
||||
Algorithm: readString(adminValues, "algorithm", defaultAlgorithm),
|
||||
}
|
||||
if cfg.Workers <= 0 {
|
||||
cfg.Workers = defaultWorkers
|
||||
@ -60,6 +75,12 @@ func ParseConfig(adminValues, workerValues map[string]*plugin_pb.ConfigValue) Co
|
||||
if cfg.MaxRuntime <= 0 {
|
||||
cfg.MaxRuntime = time.Duration(defaultMaxRuntimeMinutes) * time.Minute
|
||||
}
|
||||
switch cfg.Algorithm {
|
||||
case AlgorithmStreaming, AlgorithmDailyReplay:
|
||||
// valid
|
||||
default:
|
||||
cfg.Algorithm = defaultAlgorithm
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
|
||||
@ -27,6 +27,34 @@ func TestParseConfigDefaults(t *testing.T) {
|
||||
if cfg.BootstrapInterval != 0 {
|
||||
t.Errorf("BootstrapInterval default=%v, want 0 (walk-once-per-process)", cfg.BootstrapInterval)
|
||||
}
|
||||
if cfg.Algorithm != AlgorithmDailyReplay {
|
||||
t.Errorf("Algorithm default=%q, want %q", cfg.Algorithm, AlgorithmDailyReplay)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseConfig_AlgorithmStreamingExplicit(t *testing.T) {
|
||||
// Streaming stays available as a rollout escape hatch until Phase 5
|
||||
// deletes it. Operators must be able to opt back into it explicitly.
|
||||
admin := map[string]*plugin_pb.ConfigValue{
|
||||
"algorithm": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: AlgorithmStreaming}},
|
||||
}
|
||||
cfg := ParseConfig(admin, nil)
|
||||
if cfg.Algorithm != AlgorithmStreaming {
|
||||
t.Errorf("Algorithm=%q, want %q", cfg.Algorithm, AlgorithmStreaming)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseConfig_AlgorithmUnknownValueFallsBackToDefault(t *testing.T) {
|
||||
// Operators should not be able to silently activate a future
|
||||
// algorithm value by typo. Anything not in the enum falls back to
|
||||
// the default (daily_replay).
|
||||
admin := map[string]*plugin_pb.ConfigValue{
|
||||
"algorithm": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "future_algo"}},
|
||||
}
|
||||
cfg := ParseConfig(admin, nil)
|
||||
if cfg.Algorithm != AlgorithmDailyReplay {
|
||||
t.Errorf("unknown algorithm=%q must fall back to %q, got %q", "future_algo", AlgorithmDailyReplay, cfg.Algorithm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseConfigOverrides(t *testing.T) {
|
||||
|
||||
@ -11,6 +11,8 @@ import (
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||||
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dailyrun"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/scheduler"
|
||||
@ -81,11 +83,23 @@ func (h *Handler) Descriptor() *plugin_pb.JobTypeDescriptor {
|
||||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 1}},
|
||||
MaxValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 16}},
|
||||
},
|
||||
{
|
||||
Name: "algorithm",
|
||||
Label: "Algorithm",
|
||||
Description: "Daily Replay = bounded daily meta-log scan (Phase 2, replay-only — buckets using ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, or scan_only rules will fail the run until Phase 4 ships). Streaming = legacy reader+heap path, kept as a runtime escape hatch during the Phase 4 rollout; Phase 5 deletes it.",
|
||||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_ENUM,
|
||||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_SELECT,
|
||||
Options: []*plugin_pb.ConfigOption{
|
||||
{Value: AlgorithmDailyReplay, Label: "Daily Replay (default)", Description: "Bounded daily meta-log scan. Replay-only in Phase 2; buckets with walker-bound rules fail the run."},
|
||||
{Value: AlgorithmStreaming, Label: "Streaming (legacy fallback)", Description: "Long-running reader + per-shard heap + tick dispatcher. Pre-cutover behavior; removed in Phase 5."},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
DefaultValues: map[string]*plugin_pb.ConfigValue{
|
||||
"workers": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: defaultWorkers}},
|
||||
"workers": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: defaultWorkers}},
|
||||
"algorithm": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: defaultAlgorithm}},
|
||||
},
|
||||
},
|
||||
WorkerConfigForm: &plugin_pb.ConfigForm{
|
||||
@ -248,6 +262,10 @@ func (h *Handler) Execute(ctx context.Context, request *plugin_pb.ExecuteJobRequ
|
||||
defer s3Conn.Close()
|
||||
rpc := s3_lifecycle_pb.NewSeaweedS3LifecycleInternalClient(s3Conn)
|
||||
|
||||
if cfg.Algorithm == AlgorithmDailyReplay {
|
||||
return h.executeDailyReplay(runCtx, request, bucketsPath, filerClient, rpc, cfg, sender)
|
||||
}
|
||||
|
||||
sched := &scheduler.Scheduler{
|
||||
BucketsPath: bucketsPath,
|
||||
Engine: engine.New(),
|
||||
@ -269,6 +287,60 @@ func (h *Handler) Execute(ctx context.Context, request *plugin_pb.ExecuteJobRequ
|
||||
return nil
|
||||
}
|
||||
|
||||
// executeDailyReplay runs the bounded daily-replay path. Reuses the
|
||||
// streaming path's filer / s3 / engine setup but routes the per-shard
|
||||
// loop through dailyrun.Run instead of scheduler.Scheduler. Phase 2:
|
||||
// replay-only, refuses walker-bound action kinds with a typed error.
|
||||
func (h *Handler) executeDailyReplay(ctx context.Context, request *plugin_pb.ExecuteJobRequest, bucketsPath string, filerClient filer_pb.SeaweedFilerClient, rpc s3_lifecycle_pb.SeaweedS3LifecycleInternalClient, cfg Config, sender pluginworker.ExecutionSender) error {
|
||||
eng := engine.New()
|
||||
inputs, parseErrors, err := scheduler.LoadCompileInputs(ctx, filerClient, bucketsPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("daily_replay: load lifecycle inputs: %w", err)
|
||||
}
|
||||
for _, pe := range parseErrors {
|
||||
glog.V(1).Infof("daily_replay: %s: %v", pe.Bucket, pe.Err)
|
||||
}
|
||||
eng.Compile(inputs, engine.CompileOptions{PriorStates: scheduler.AllActivePriorStates(inputs)})
|
||||
|
||||
shards := make([]int, 0, cfg.Workers)
|
||||
// One pass per shard ID across [0, ShardCount). cfg.Workers governs
|
||||
// concurrency, not partitioning — every shard gets exactly one
|
||||
// goroutine and the rate.Limiter is the throughput governor.
|
||||
for i := 0; i < s3lifecycle.ShardCount; i++ {
|
||||
shards = append(shards, i)
|
||||
}
|
||||
|
||||
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
|
||||
JobId: request.Job.JobId, JobType: jobType,
|
||||
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
|
||||
Message: fmt.Sprintf("daily_replay shards=%d runtime=%s", len(shards), cfg.MaxRuntime),
|
||||
})
|
||||
|
||||
runErr := dailyrun.Run(ctx, dailyrun.Config{
|
||||
Shards: shards,
|
||||
BucketsPath: bucketsPath,
|
||||
Engine: eng,
|
||||
FilerClient: filerClient,
|
||||
Client: lifecycleRPCAdapter{c: rpc},
|
||||
Persister: &dailyrun.FilerCursorPersister{Store: dispatcher.NewFilerStoreClient(filerClient)},
|
||||
Lister: dispatcher.NewFilerSiblingLister(filerClient, bucketsPath),
|
||||
Workers: cfg.Workers,
|
||||
ClientName: "worker-s3-lifecycle-daily",
|
||||
// Limiter is wired in Phase 3 from ClusterContext.Metadata.
|
||||
})
|
||||
if dailyrun.IsUnsupportedRule(runErr) {
|
||||
// Surface the typed error verbatim so admin marks the run as
|
||||
// failed with the user-facing reason in the activity log.
|
||||
glog.Warningf("daily_replay: %v", runErr)
|
||||
return runErr
|
||||
}
|
||||
if runErr != nil {
|
||||
glog.Warningf("daily_replay: %v", runErr)
|
||||
return runErr
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// clusterS3Endpoints returns the master-discovered S3 gRPC addresses for the
|
||||
// cluster. The handler dials the first reachable one; the master refreshes
|
||||
// the list on KeepConnected so a stale entry self-heals on the next run.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user
