Problem:
In-flight out-of-line (OOL) writes did not clear t.pending_ool on successful
completion. If the transaction was subsequently marked conflicted during
commit/prepare, it erroneously marked the completed write as conflicted.
This caused:
1. conflict_pending_free_bytes to leak upward.
2. Physical extents to remain allocated but untracked, leaking space.
Eventually, this led to allocator exhaustion and OSD deadlock.
Solution:
Clear t.pending_ool on OOL write completion and transaction reset. Implement
a backpressure condition variable for pending deferred frees so that write
operations wait when accumulated conflicted frees exceed a 5% threshold.
Change:
* Added clear_pending_ool() to Transaction and called it from the
do_write() finally continuation in RandomBlockOolWriter::
alloc_write_ool_extents, so it runs on every exit path (success,
error, or conflict-driven unwind) and the conflicted check plus the
clear are atomic with respect to reactor task interleaving.
* Cleared pending_ool in Transaction::reset() to prevent leaks across
transaction reuse/retry.
* Tracked conflicted pending free bytes in RBMCleaner.
* Added wait_for_conflict_drain() to pause new OOL writes when deferred
frees exceed 5% of data capacity.
The pending deferred frees are also a release source for the reservation
gate introduced by the preceding commit: account them in
try_reserve_projected_usage()'s release-source check, and wake blocked IO
when they drain so the gate is re-evaluated.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
Problem:
Using the stats.used_bytes counter for RBM capacity check is unreliable.
Unmetered data paths (such as alloc_paddr or overwrites) cause it to drift
and diverge from the physical block allocator state. This leads to
premature ENOSPC rejections or capacity overcommits that crash the OSD.
Solution:
Implement is_storage_full() and try_reserve_projected_usage() using the
ground truth of physical allocator state (AVL tree's available size).
This replaces the interim statfs-based is_storage_full() failsafe check
introduced by the preceding OSDOp gating commit with the allocator's
authoritative view, as anticipated there.
Change:
* Updated BlockRBManager::get_free_blocks() to return the actual count
of free blocks tracked by the AVL allocator.
* Implemented get_allocator_used_bytes() in RBMCleaner to compute the
total allocated size across all RBM devices.
* Changed try_reserve_projected_usage() to query the allocator state
instead of stats.used_bytes, and reduced headroom to 1%.
* Overrode the virtual is_storage_full() hook in RBMCleaner to trigger
boundary gating using the same allocator state.
* Guarded is_storage_full() to return false until the background process
is ready (mirroring the is_ready assertion in try_reserve_projected_usage),
so capacity cannot be misreported as full while the allocator is still
being populated during mount.
* Gate only while a release source exists: RBM has no cleaner, so a
failed reservation can only be woken by an in-flight transaction
releasing its projected usage. If none exists, admit the IO instead of
blocking (which would deadlock, e.g. a sole writer on a full device)
and let the block allocator be the final authority: a genuinely full
device fails the allocation with ENOSPC through the existing
alloc-failure path. Zero-usage reservations are always admitted: they
cannot overcommit, and deletes -- the space-releasing transactions --
are metadata-only.
* Report statfs (get_stat) and the rbm_cleaner metrics from the same
allocator view via a get_used_bytes() helper, so operator-visible
capacity matches the admission gating; both fall back to
stats.used_bytes until the allocator is populated during mount.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
Add storage_full_failsafe_threshold to the RBM-backed transaction
manager tests: a fresh store reports not-full at the default ratio,
real extent allocations flip is_storage_full() once the used fraction
crosses osd_failsafe_full_ratio, and restoring the default ratio
clears the condition, verifying the threshold tracks the config knob.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
RBMCleaner::is_storage_full() hardcoded a 3% free limit. Read the same
config classic uses for its failsafe check (osd_failsafe_full_ratio,
default 0.97) instead, treating the store as full once the free fraction
drops below (1 - ratio), so the threshold tracks the knob rather than a
magic constant.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
When SeaStore runs out of space, allocations in object_data_handler.cc hit
enospc::assert_failure and abort the OSD. Gating on monitor OSDMap pool FULL
flags alone is insufficient: flag propagation lags by seconds-to-minutes and
cannot protect the local allocator from sudden exhaustion.
Add a local failsafe gate at the OpsExecuter boundary, Crimson's analog of
classic's osd_failsafe_full_ratio check (PrimaryLogPG.cc:2162). Data-allocating
ops (CREATE/WRITE/WRITEFULL/WRITESAME/APPEND/COPY_FROM2) are dropped with
-EAGAIN so the client resends, matching Crimson's existing mon-driven full path
(the "drop request" case in pg.cc) and classic's silent drop. This keeps
behavior uniform regardless of whether pool full flags have reached the client
yet. CEPH_OSD_FLAG_FULL_TRY is exempt; space-reclaiming ops
(ZERO/TRUNCATE/ROLLBACK) are deliberately not gated. The mon-driven pool
FLAG_FULL path in run_executer is untouched and remains authoritative.
The threshold uses the store's statfs accounting (stats.used_bytes). The
follow-on (#69352) moves RBM capacity checks to the allocator's authoritative
view and wires mon-side NEARFULL/FULL via statfs, so clients get a clean stop
rather than resend-until-resolved.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
smb: support altpaths for updating non-AD users/groups live while a cluster is running
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
* refs/pull/70212/head:
qa/cephfs-mirror: Fix tests for ISO sync_time_stamp
doc/cephfs-mirroring: document ISO-8601 sync timestamps
cephfs_mirror: format sync timestamps as ISO-8601 local time
tools/cephfs-mirror: report epoch time in last snap sync time
Reviewed-by: Karthik U S <karthik.u.s1@ibm.com>
when demoting them
It is possible that, when we do demote_region, the target region has
already been removed from the lba tree, for example, a temp recovering
object might have already been renamed to the real one, which means all
lba mappings within its original region have been moved away.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
to cache
This is to avoid an under-promotion extent from being added back to and
evicted from the cache again before the promotion ends.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
This is necessary for the promote transaction to find the transactions
that are retiring the extents promoted
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
rewrite transactions
So we need to merge the extents promoted/demoted with their counterparts
in client transactions, just like trim/cleaner transaction do.
The difference between demote/promote transactions and trim/cleaner
transactions is that the former also involves remapping extents while the
latter only involves rewriting and mutating extents.
With the logical bucket cache in place, all rewrite transactions need to
handle shadow extents carafully, too.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
In scatter_allocation, an extent is split into several
small ones, and the last allocation is supposed to fail,
in which case, the small extents that belong to the same
allocation wouldn't be attached to the parent lba leaf
node. Sumitting the transaction would trigger assertions,
and is not consistent with the behavior of nornal crimson
OSDs.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
read/write transactions
All client accessed extents are to be tracked by logical bucket cache
before eviction
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Write extents larger than some threshold to the cold tier directly
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
python-common/smb: work around socket path length limits for grpc lib
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Admin role:
- show confirmation modal that pool deletion is disabled
- allow enabling pool deletion
- fix pool deletion text
Pool mgr role
- show a modal that pool deletion is disabled
Signed-off-by: Afreen Misbah <afreen@ibm.com>
This is because, even for LRU caches, extents still have pin states,
although it's only Fresh. What's more, in the logical bucket cache
world, extents in both LRU and 2Q caches may be in the
Fresh/PendingPromote/Promoting state.
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Switch the trigger from `pull_request` to `pull_request_target` so that
the workflow definition on `main` is evaluated for PRs targeting active
release branches (main, umbrella, tentacle, squid). This allows the check
to run cleanly across release branches without requiring workflow file
backports to each branch.
Additionally:
- Update `src/script/verify-qa` to accept an optional target directory argument ($1).
- Run the trusted `verify-qa` script from `main` against the checked-out PR directory (`./pull_request`).
The concern of exfiltrating secrets is important but not relevant here:
we're not using the secrets in the definition and we explicitly
downgrade the github token to `read` permission.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Switch the trigger from `pull_request` to `pull_request_target` so that
the workflow definition on `main` is evaluated for PRs targeting active
release branches (main, umbrella, tentacle, squid). This allows the check
to run cleanly across release branches without requiring workflow file
backports to each branch.
While this workflow passes `github.token`, the token permissions are
strictly restricted to read-only (`contents: read` and `pull-requests: read`).
Because no PR code is checked out or executed, using `pull_request_target`
presents zero security risk here.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
The grpc library (at least the version on centos/rocky 10) has a limit
of 107 characters. Even a fairly short host name is going to be close
to this limit due to the UUID and service name in the path.
Because the tool doesn't do much else and it is safe, the workaround is
to chdir to into the parent dir of the socket and only pass the relative
path name of socket.
Fixes: https://tracker.ceph.com/issues/78857
Signed-off-by: John Mulligan <jmulligan@redhat.com>
PRs carrying these GitHub labels were silently dropped from both the
QA tracker's PR "Labels" column and the Redmine ticket's tag_list,
since SUPPORTED_QA_TAGS is a fixed whitelist and these were never
added to it despite being real, actively-used component/category
labels (e.g. ceph/ceph#70661, #70658 for crimson).
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
In order to update users (and groups) live without AD support we need
the configwatch sidecar to be configured for split (altfiles) nsswitch.
The capability to configure configwatch for nsswitch was added to
sambacc under a special mode of the run subcommand.
Switch to starting the confgwatch sidecar using the run subcommand and
configuring the appropriate user and nsswitch setup options.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Update environment for smb containers to specify arguments that sambacc
will use to configure the containers for a split configuration for
users (passwd) and group files. The writable copy will be store in
the samba state dir (/var/lib/samba in the container) and be shared
across the primary and sidecar containers so that configwatch can
pull new users and groups lists from ceph and apply them live.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Calling `list_images()` on a non-existent group falsely succeeded and
returned an empty iterator.This happened because `Group<I>::image_list`
ignored the negative error code returned by `group_image_list`, allowing
execution to proceed blindly. Furthermore, the C API `rbd_group_image_list`
intercepted `-ENOENT` and forced a success (0) return value.
Similarly other public C APIs `rbd_snap_list` and `rbd_group_snap_list`
also intercepts -ENOENT and forces a success (0) return value which
is incorrect.
Fix these behavior by:
- Propagating errors properly inside `Group<I>::image_list`.
- Removing the incorrect `-ENOENT` masking in `rbd_group_image_list`,
`rbd_snap_list`, rbd_group_snap_list`
Now `list_images()` consistently raise an `ObjectNotFound` error when
invoked on a non-existent group and public C APIs propagate ENOENT
without masking it into 0.
Also Extended the `TestGroups` test fixture with `self.dne_group` to
validate the expected error paths across all relevant APIs.
Fixes: https://tracker.ceph.com/issues/78108
Signed-off-by: VinayBhaskar-V <vvarada@redhat.com>
Use ref instead of external link definitions for intra-docs links.
Add labels if necessary in files within the install/ directory. Use
existing labels for links outside the install/ directory but do not add
new labels outside of the install/ directory.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
Substitutions were not enabled on a prompt directive that tried to use
the stable-release substitution so the substitution string was rendered
verbatim.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
When editing services such as oauth2-proxy, backend fields like scope
arrive as a single string. Spreading that string into the list turned
each character into its own row. Normalize string values to one item.
Fixes: https://tracker.ceph.com/issues/78808
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>
CherryPy's serve_file() uses Python's mimetypes module which returns
text/javascript for .js files (RFC 9239), but the gzip config only
listed application/javascript. This meant the main Angular bundle
was being served uncompressed. With Cheroot write timeouts,
the uncompressed transfer times out over slower networks, causing
ERR_CONTENT_LENGTH_MISMATCH in the browser.
Signed-off-by: Afreen Misbah <afreen@ibm.com>
crimson/osd/pg_recovery: call on_global_recover on replica-recover-deleted objects only when it's not missing on the primary too
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
* refs/pull/70574/head:
.github/workflows: bump checkout action version
.github/workflows/qa-symlink: sparse checkout verify-qa
.github/workflows/qa-symlink: run only for active branches
Reviewed-by: Joseph Mundackal <jmundackal@bloomberg.net>
Add documentation link in Learn more column for every alert in the
active alerts list table
Fixes: https://tracker-origin.ceph.com/issues/78747
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
The updated seastar deprecates the global smp::count and smp::all_cpus() in
favour of the per-instance smp::shard_count() and smp::all_shards(). Under the
CI -Werror build these deprecation warnings are fatal.
Replace the uses with the reactor-local free functions
seastar::this_smp_shard_count() and seastar::this_smp_all_shards(), which read
the current smp instance's shard count and shard-id range. crimson runs a
single smp instance and every migrated site executes on a reactor thread (the
alienstore worker threads do not touch these), so the behaviour is unchanged.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
At SEASTAR_API_LEVEL >= 9 seastar dropped output_stream::write(net::packet) in
favour of write(std::span<temporary_buffer<char>>). Convert Socket's writes to
the span form, writing the packet's released fragments and keeping them alive
across the write with do_with, then raise Seastar_API_LEVEL from 6 to 10 to
match the updated seastar. The span write overloads are unconditional, so the
Socket change is valid at every API level.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Update src/seastar to ceph-umbrella-25.05.0-1074-gcced0236, which is
scylladb/seastar master (fd71a5b4c7) with the ceph patches reapplied,
and adapt crimson to the future API changes the bump brings in:
- interruptible_future.h: seastar split its single ready_future_marker into
set_ready_future_marker and set_from_tuple_ready_future_marker; route
futurize::from_tuple through the matching markers.
- thread_pool.h: future_state::set() is now templated and get_value() is
ref-qualified; pass seastar::internal::monostate{} for the void case and
move the state before reading its value.
These crimson changes need the new seastar, so they share one commit with the
submodule bump. The build stays at Seastar_API_LEVEL 6 here; the level is
raised in the following commit.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
rely on 'bucket sync status' instead of looking up sync-status object.
it reports that there are no sync sources once the bucket's disabled
flag has propagated. also add a meta checkpoint after disabling.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
Add unit-tests to recreate the problems found by the IO Sequencer in the previous commit.
Some tests here rely on frameworks added in Umbrella and as such we are keeping them
independent (in their own PRs).
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Assisted-by: IBM Bob:Claude/GPT
Some binaries and man pages were not included in rpm builds or debian
builds. This adds those artifacts to ceph.spec.in
debian/ceph-common.install and debian/radosgw.install.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
mgr/smb: only resync clusters that reference a changed resource
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
The vendor stylizes it as NGINX.
Previously docs capitalized it as either NGINX, Nginx or nginx. nginx
was the preferred capitalization before commercialization of the
software.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
objects only when it's not missing on the primary too
The issue was introduced by 60537bdd2f
which didn't fix the issue it was supposed to because of the same error
this commit tries to fix.
Fixes: https://tracker-origin.ceph.com/issues/78676
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
ow_gap_from_last_entry() returns how much the node grows when the tail entry
is overwritten. However, it compared *value* lengths only.
overwriting pg-log key (31B) is longer than a _fastinfo key (9B) -
So the growth was under-counted by 22 bytes.
Also assert in _append()/_overwrite() that the write stays within capacity() to
avoid similar issues.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
period commit only triggers async push (RGWPeriodPusher)
from the a zone's gateway to the other zones. it doesn't block
until they've applied it. a reconfigure delay alone might not
be sufficient. add checkpoint retries to allow for lag.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
mgr/volumes: accept unit with subvolume resize
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Christopher Hoffman <choffman@redhat.com>
current io exerciser sequences can model standalone truncates and writes,
but cannot generate a single transaction that truncates an object and
then writes sparse regions. Add SingleTruncateWriteOp, DoubleTruncateWriteOp,
and TripleTruncateWriteOp, wire them through IoSequence, ObjectModel, and
RadosIo, and extend EC sequences/tests so truncate+write transactions can
be reproduced and validated.
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Assisted-by: IBM Bob:Claude/GPT
Problem:
Before this change, when the ISA plugin (classic build, dlopen'd),
would try to log, it'd segfault due to null g_ceph_context.
We need a way to make logging work for EC plugins in crimson.
Solution:
By building the ISA plugin WITH_CRIMSON, its logging is
routed to seastar's logger, solving the issue.
It is not build separately and dlopen'd like in classic because
doing so would require the plugin to have its own seastar logger
copy, an undesirable outcome.
It is a statically built library in the crimson-osd binary, and thus
it shares the single seastar logger. The library is linked with
--whole-archive so the self-registration constructor is not dropped.
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
This commit adds the functionality in EC plugin registry to support
built in (statically linked) plugins. This is done by 2 main changes:
1. introducing a new map, builtin_plugins, where plugins can register
a factory for itself at startup.
2. load() now first checks if the plugin is builtin and instantiates it,
otherwise it falls back to dynamically linking the plugin.
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
prepare_data_device() was calling nvme_utils.preformat() unconditionally
for every OSD slot. With --osds-per-device N, this means the second osd
reformats the device after the first OSD had already created its vg
on it which makes vgcreate fail with "device has a signature".
this commit fixes this issue by checking api.get_device_vgs() before
preformatting: if a ceph VG already exists on the device, skip the format.
This mirrors the logic create_lv() already uses to reuse an existing VG.
Fixes: https://tracker.ceph.com/issues/78615
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
The test_invalid_client_id was failing due to the
incorrect returns by handle_command and in get_perf_data()
in mgr/ststs/fs/perf_stats.py
Introduced-by: f8f972182c
Fixes:https://tracker.ceph.com/issues/76162
Signed-off-by: Neeraj Pratap Singh <Neeraj.Pratap.Singh1@ibm.com>
Monitor.cc logs a "Health check cleared: <CODE> (was: ...)" line
for any health check clearing mid-test, with the code bare (no
parens) -- while the "Health check failed"/"Health check failed
(unmute)" lines wrap the code in parens. Every \(CODE\) entry in
this file therefore only matches the raise/failed form and misses
the corresponding cleared form, the same gap just fixed for
POOL_FULL.
Drop the escaped parens from the remaining entries so each matches
both log forms, consistent with the bare entries already present
(OSD_ROOT_DOWN, MDS_INSUFFICIENT_STANDBY, POOL_FULL).
Non-code entries (reached quota, overall HEALTH_, slow request,
noscrub, nodeep-scrub, osds down, insufficient standby, etc.) are
unchanged.
Fixes: https://tracker.ceph.com/issues/78149
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
An initial_pending LogNode always append_kv()s and never overwrites.
expect_overflow() was still called with can_ow=true whenever the tail entry is
_fastinfo. It then used only the overwrite gap while the append
itself consumed a full entryi - so the node rolled over far too late and
used_space() overran capacity().
This would result in `assert(capacity() >= used_space()`.
Gate can_ow on !is_initial_pending() so the append is predicted with the full
entry size and rolls over in time.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
The previous implementation computed ow_gap_from_last_entry() but never used
the result.
Use it to check only the additional space required when the tail
entry grows, which matches overwrite logic.
Note, this is a cleanup - behavior remains similar.
See 825c8f149c
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Show sync_time_stamp and metrics_updated_at as human-readable
local timestamps in peer_status and mgr status. Keep epoch in
omap and counter dump.
Fixes: https://tracker.ceph.com/issues/76506
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Expand nvmeof mtls test to include cephadm-signed cert
(ssl=true + enable_auth=true, no certs)
Also improve wait_for_service() logic to assert all
gateways are running.
Fixes: https://tracker.ceph.com/issues/78295
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>
Switch all stat panels in the hardware and hardware compression
dashboards from lastNotNull to last reduceFunction. When a host is
unreachable or node-proxy stops, metric silence is ambiguous — it
could mean host down, agent crash, or scrape failure. lastNotNull
freezes the last known value (e.g. 4 NVMe drives), causing
inconsistency with the global Overview which reflects live count
reductions.
Use last so panels show N/A when no current data arrives.
Fixes: https://tracker.ceph.com/issues/78360
Signed-off-by: Afreen Misbah <afreen@ibm.com>
When fail_fs is enabled with max_mds > 1, _prepare_for_mds_upgrade()
issued fs fail and then continued unconditionally, skipping the
up:active wait. _complete_mds_upgrade() also set joinable=true without
waiting for in-rank MDS to recover.
Wait for all in-rank MDS to reach up:active before proceeding with the
upgrade and after re-joining the filesystem. Add fs.ready to the
mds_upgrade_sequence verify task so ceph-fuse teardown does not race
with MDS still in up:rejoin.
Fixes: https://tracker.ceph.com/issues/77835
Signed-off-by: Kobi Ginon <kginon@redhat.com>
On the systemd-resolved (debian) path, dnsmasq has only address= records
and no upstream, so queries it can't answer locally are forwarded via
/etc/resolv.conf which points back at systemd-resolved, which routes
the cname domains back to dnsmasq. The resulting loop burns a 5s
resolver retry per round trip which manifests itself in rgw s3tests
on Ubuntu running too slow hitting the job timeout.
Set no_resolv on that path so dnsmasq answers authoritatively and never
forwards. The rpm/NetworkManager path is unchanged, as dnsmasq there is
the system resolver and must forward upstream.
Fixes: https://tracker.ceph.com/issues/78572
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
crimson/osd: fix backfill deadlock in BackfillState::Waiting
BackfillState::Enqueuing unconditionally posts RequestWaiting{}
when budget available return false. Waiting can exit only via
ObjectPushed which requires an in-flight push. If the budget
is exhausted with no pushes in flight no objectPush will ever
arrive and the PG stalls permanently.
Solution: Introduce a dedicated BudgetBlocked state. When
budget_available() is false and tracked_objects_completed()==true
post RequestBudgetBlocked{} instead of RequestWaiting{}.
BudgetBlocked calls get_backfill_throttle() to suspend until a slots
free up, then fires BudgetAvailable{} which transitions directly back
to Enqueuing to dispatch the next batch of pushes.
Fixes: https://tracker.ceph.com/issues/77804
When a notification is clicked in the panel while the user is already
on the notifications page, the selected notification now updates
correctly by bypassing the preselect guard for query param changes.
Signed-off-by: Afreen Misbah <afreen@ibm.com>
librdkafka submodule
Vendors librdkafka v2.12.1 as a submodule (default) and teaches the
notification teuthology task to set up Kafka 4.x KRaft brokers. The
distro-shipped librdkafka 1.6.1 cannot SCRAM-auth against Kafka 4.x
brokers (#75900); the bundled version fixes that. Build with
WITH_SYSTEM_RDKAFKA=ON to opt back into the distro library (floor 2.6.1).
follwinf changes needed with teh submodule:
* build kafka_stub only in case of system librdkafka
* use the same header location in submodule and system
* supress compilation warning to avoid -WError issues
2.12.1 matches Fedora(44)'s current librdkafka-devel, so the vendored copy
tracks a version that distro packagers have already vetted rather than
whichever release happens to be latest upstream.
The teuthology version matrix picks one of {3.9.2, 4.1, 4.2, 4.3} per run.
3.9.2 covers the last Zookeeper-based release for the migration cohort.
4.1/4.2/4.3 track Apache's current three supported minor lines and are
resolved to the latest patch at test time via dlcdn.apache.org/kafka/, so
new patches within a supported minor need no PR update. A new minor line
(4.4) requires renaming one YAML.
* new java version (17) is needed for kafka 4.2 and up (we wil use it on
all version)
* kerberos task need to run before the kafka install task
After pulling this branch, fetch the new submodule:
git submodule update --init src/librdkafka
Fixes: https://tracker.ceph.com/issues/77336
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com>
Transaction::append() asserts both transactions share data_features.
The per-collection batcher merged queued client txns unconditionally,
so a data_features mismatch would be asserted in build_next_batch().
Use diffrent data_features as a grouping boundary.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Per-collection batching merges many client txns into one seastore txn.
Batched OP_OMAP_RMKEYS is not stable yet, possibly due to an exposed
internal bug that is seen with the new behavior.
Avoid batching OMAP_RMKEYS for now.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Fixes: https://tracker.ceph.com/issues/76507
Signed-off-by: pujaoshahu <pshahu@redhat.com>
Summary: Updated submit button labels in edit forms to display "Save" instead of "Edit User", "Edit Pool", or "Edit Host" for better UX consistency, while keeping form titles unchanged.
Modified Files
role-form.component.ts: Added submitAction property to control submit button label by mode; edit mode uses Save changes, create mode uses Create Role.
role-form.component.html: Submit button now binds to submitAction instead of action/resource text.
role-form.component.spec.ts: Added tests for submitAction behavior in create and edit modes.
user-form.component.ts: Uses submitAction property; edit mode label updated to Save changes.
user-form.component.html: Submit button uses submitAction binding.
user-form.component.spec.ts: Added/updated tests covering submitAction behavior.
pool-form.component.ts: Uses submitAction property; edit mode label updated to Save changes.
pool-form.component.html: Submit button uses submitAction binding.
pool-form.component.spec.ts: Updated submit label test expectation for edit mode.
hosts.component.ts: Edit host modal submit button text changed to Save changes.
hosts.component.spec.ts: Updated test to match Save changes button text.
app.constants.ts: In the current workspace snapshot, there is no SAVE constant added under ActionLabelsI18n.
Signed-off-by: pujaoshahu <pshahu@redhat.com>
Signed-off-by: pujashahu <pshahu@redhat.com>
fixes: https://tracker.ceph.com/issues/78332
Before
===
The configuration form had a TypeScript compilation error and could not properly support editing client-specific configurations. The form lacked the ability to:
- Configure different values for specific client entities (e.g., client.rgw.*, client.admin, client.radosgw-gateway)
- No way to Add/remove multiple client entity entries dynamically
- No way to provide autocomplete suggestions for existing client entities
After
===
Users can now edit configuration options individually for any client entity, with a clean UI that supports:
- Generic client entity support: Configure any client.* entity (RGW daemons, admin users, CephFS clients, etc.)
- Autocomplete with custom values: Select from existing client entities or enter custom ones (e.g., client.rgw.my-daemon, client.admin)
- Dynamic entry management: Add/remove multiple client configuration entries
Signed-off-by: Naman Munet <naman.munet@ibm.com>
mgr/dashboard: restricting admin user to revoke its own admin permissions
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
I checked a couple of gherkin linters as an alternative but all of them
are not really well maintained. There are couple of new projeects but
even they are simply maintained by a single guy so i don't think it'll
get enough contributions later on. So instead, I am moving it to
prettier. Although we miss some functional validation, it'll take care
of some basic formatting.
Fixes: https://tracker.ceph.com/issues/78528
Signed-off-by: Nizamudeen A <nia@redhat.com>
When a rewrite transaction publishes via no-conflict handoff, merge_content_to
can update an already CRC-stamped initial-pending fixedkv page (copy-dest of a
concurrent split). It refreshed last_committed_crc but not the in-extent phy
checksum, so the page could be written with new content and a stale stamp.
Debug asserts only compare last_committed_crc to calc_crc32c(), so the mismatch
survived until a later cold read aborted with "fixedkv extent checksum
inconsistent".
Recalculate the CRC and call update_in_extent_chksum_field together in both
LBALeafNode and FixedKVInternalNode merge_content_to, matching
apply_delta_and_adjust_crc / update_lba_mappings.
Fixes: https://tracker.ceph.com/issues/77022
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
When the `rgw_copious_policy_logging` option is set to true, a
detailed trace of policy evaluation is logged.
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
(I wanted to see how it performed on rebase conflict resolution.)
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
(I figured I should have it look at the policy evaluation code for
bugs while it's included, and it found a few mistakes in output.)
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Create a log entry where we can fill a buffer with `format` instead of
needing to go through stringstream.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Extract the Redmine auth check added in the previous commit into a
standalone verify_redmine_auth(R) function so it can be unit tested
in isolation, and add src/script/test_ptl_tool.py covering:
- verify_redmine_auth(): invalid key (AuthError), a key that
authenticates but lacks permission (ForbiddenError), a valid key,
and confirming unrelated Redmine errors (ServerError) still
propagate normally rather than being misreported as a credentials
problem.
- AuditReport.post_consolidated_review(): logs success on 2xx, logs
an error (rather than failing silently) on a non-2xx GitHub
response, never calls out to GitHub under --dry-run, and is a
no-op when the report has no issues to post.
No behavior change to verify_redmine_auth() itself -- this is a pure
extraction plus tests. ptl-tool.py has no existing test suite, so
these are self-contained (mocked Redmine/requests, no network access,
no real git checkout needed) and run with plain pytest; see the
module docstring for the exact invocation.
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Invalid credentials were handled inconsistently:
- A missing GitHub/Redmine token was caught with a clear error at
startup, but an invalid or expired one was not: the Redmine
client is constructed lazily by python-redmine, so a bad
~/.redmine_key was never actually exercised until
manage_qa_tracker() called R.project.get() -- which runs *after*
PRs have been merged and the integration branch/tag has already
been pushed to ceph-ci. The result was an unhandled traceback
well after real side effects had already happened, with no
actionable message.
- post_consolidated_review() posted the consolidated audit review
to GitHub without checking the response status at all, so a bad
or under-scoped token would fail silently there while every other
GitHub write call in the file already checks status_code.
Fix both:
- Immediately after constructing the Redmine client, make a cheap
R.user.get('current') call and catch AuthError/ForbiddenError,
exiting with a clear message before any merging or pushing
happens.
- Check the response status in post_consolidated_review() and log
an error on failure, matching the pattern already used by the
other GitHub POST call sites in this file.
Verified with py_compile plus live --dry-run runs against
tracker.ceph.com: a deliberately invalid PTL_TOOL_REDMINE_API_KEY
now exits 1 immediately with a clear message and no git side
effects, while a valid key proceeds exactly as before.
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
- Escape user-controlled content before Carbon innerHTML rendering
- Whitelist localStorage deserialization keys, prune stale read map
- Replace deep equality with ID comparison in removeToast
- Gate ngAfterViewChecked DOM queries behind a dirty flag
- Deduplicate by title+type+message; show occurrences count
- Fix bell icon unread dot clearing on new occurrences
- Fix view more pre-selection via route queryParams observable
- Save individual notifications instead of HTML-merged messages
- Convert notifications bell component to signals with OnPush
- Add OnPush to notification-item and toast components
- Add e2e tests
- Fix styling of notification panel
Signed-off-by: Afreen Misbah <afreen@ibm.com>
rgw_gc: don't fail when processing entry with nonexisting pool
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
mgr/smb: add support for RGW shares with external Ceph clusters
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
- adds view more link to toast which takes to view full toast message
- allow notifications to be clickable in notification panel taking user to the particular notification
- add routes to eahc notitification in notificatio page
- extend duration for error notifications - as per carbon error notifications should persist longer since users need time to read
Signed-off-by: Afreen Misbah <afreen@ibm.com>
Block-manager users were denied access to the NVMe/TCP page because
host listing APIs require hosts read permission.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
Root cause: On FIPS-compliant systems, the OpenSSL provider rejects hashlib.md5() calls that lack the usedforsecurity=False flag, because MD5 is not an approved algorithm for security-sensitive use. This causes the MOTD set command — and the /api/motd create endpoint — to raise a ValueError and crash whenever a new MOTD is saved.
Fix: Pass usedforsecurity=False to hashlib.md5(). This tells the FIPS provider that MD5 is being used only as a non-cryptographic checksum (to detect whether a user has already dismissed a specific MOTD message), which is a legitimate use-case that FIPS allows.
Fixes: https://tracker.ceph.com/issues/78233
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
_find_modifications() regenerated every cluster whenever a JoinAuth,
UsersAndGroups, TLSCredential, or ExternalCephCluster changed, since
these can be shared across clusters. Use the existing lookup
helpers to resync only the clusters that actually reference it.
For rgw creds resync everything(just like old way), as narrowing
it scope means scanning every share in the store (which may not scale well),
so not worth a change.
Fixes: https://tracker.ceph.com/issues/78245
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
When no OSDs within a CRUSH bucket can be found to upgrade, the earlier error
message indicated the minimum number of PGs affected if any OSD was removed
from the set. For e.g.,
"Error EBUSY: unsafe to upgrade osd(s) at this time (at least X PG(s) will
become offline if any OSD out of the Y in CRUSH bucket 'foo' is stopped)"
This was inaccurate in some cases because the number of PGs affected is
determined only from the last tested OSD in the CRUSH bucket based on the sort
order (i.e., OSD with least number of PGs per OSD). Based on the CRUSH rules
and placement, this doesn't mean that the least number of PGs are affected if
this OSD is stopped. There may be other OSDs in the set that affect a smaller
number of PGs if made offline.
But with the way ok-to-upgrade logic uses the convergence factor, not all OSDs
would be tested for the offline pg check. Therefore, it is not possible to
accurately determine the minimum number affected PGs.
In view of the above, the error message is modified as shown below to be
slightly more generic but still convey the reason:
"Error EBUSY: unsafe to upgrade osd(s) at this time (one or more PG(s) will
become offline if any OSD out of the Y in CRUSH bucket 'foo' is stopped)"
Fixes: https://tracker.ceph.com/issues/78425
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Issue: The Export Multi-site Realm Token feature in the Dashboard showed no token and no realm details for any realm that had a secondary zone configured (i.e. the realm had been replicated to another cluster). The API endpoint GET /api/rgw/realm/get_realm_tokens returned an empty list or raised a 500 Internal Server Error.
Fixes: https://tracker.ceph.com/issues/78232
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Previously, a real (non-dry-run) push of the integration branch/tag to
ceph-ci was silent -- only the "[DRY RUN] Would push ..." path logged
anything. Add matching log.info() calls on the real-push path, and
include the literal `git push <remote> <ref>` invocation in the message
so it can be copy-pasted and re-run manually if needed (e.g. after a
network blip, or to re-push without re-running the whole merge).
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
blk/kernel : skip AIO thread for devices with size < block_size
Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
os/bluestore: use _release_pending_allocations() during file migration
Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Add rgw_enable_dedup_threads (default true) and rgw_nfs_run_dedup_threads
(default false) config options to control whether a given RGW instance
participates in dedup background processing.
This follows the same pattern used by rgw_enable_gc_threads,
rgw_enable_lc_threads, and rgw_enable_restore_threads. The config is
evaluated at startup in init_dedup(); when disabled, the dedup Background
object is not created, the thread is not spawned, and the instance does
not register a RADOS watch on the dedup control object.
Disabled RGW instances are invisible to the dedup cluster coordination
mechanism -- they receive no notifications, cause no timeouts, and do
not grab shard tokens.
Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
Assisted-by: Cursor Opus-4.6 <cursoragent@cursor.com>
omap_complete in read_result_t can be uninitialized if ECSubReadReply
contains an empty map of omap_complete information.
Fixes: https://tracker.ceph.com/issues/76953
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
* refs/pull/52147/head:
PendingReleaseNotes: add an note about dmclock based scheduler for MDS
doc/config-ref: mention about MDS dmclock configurations
qa: use dmclock with fs:workload
mds: add MDS dmClock scheduler for subvolume QoS support
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Robin H. Johnson <robbat2@orbis-terrarum.net>
Implement support for creating RGW-backed SMB shares on external Ceph
clusters with user configuration.
Fixes: https://tracker.ceph.com/issues/77784
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
Cluster-manager users were denied access to the CRUSH map page because
the underlying API requires pool read permission.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
Read-only users could see the Settings icon but were denied access
to user management pages. Remove user read permission from the
read-only role so the administration menu stays hidden.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
The Help menu option redirected read-only users to an Access Denied
page. Hide it when the user lacks config-opt read permission.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
Remove SilenceFormComponent from the silences list page, which
incorrectly required create permission and caused a blank page for
users with prometheus read access only.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.html
- Added permissions
Add the flags listed for ceph osd pool set in ceph osd pool get.
Remove nonexistent flag debug_fake_ec_pool in ceph osd pool set.
Confirm the additions/removals from MonCommands.h.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
The can_ow branch of LogManager's expect_overflow computed:
```
remain = capacity() - get_last_pos() - reserved_len
```
all uint32 operands, once the log leaf node is near full
(get_last_pos() + reserved_len >= capacity()) this subtraction underflows.
The node never rolls to a new leaf, instead it overflows the
16 KB block at commit-time delta replay.
Captured at the replay overflow point (per-collection batching, 4K randwrite):
```
LogNode OVERFLOW at replay: pos=16641 size=58 reserved_len=3098
reserved_size=6 capacity=16328 LIMIT=16384
log_node.h : In function 'void LogKVNodeLayout::set_last_pos(uint32_t)',
ceph_assert(pos <= LOG_NODE_BLOCK_SIZE)
```
Here reserved_len (3098) is accurate: ~13543 committed + 3098 pending =
16641 = pos, i.e. the node was already ~13.5 KB full and the predictor
should have rolled over. Instead capacity - get_last_pos - reserved_len
went negative (16328 - ~13273 - 3098 < 0), wrapped in uint32, and the
check passed; pos reached 16641 > 16384 and set_last_pos asserted.
This was exposed by the per-coll batching where many overwrites append in
a single txn.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Previously, the per-collection ordering_lock provided both ordering and
single-holder exclusion, causing same-collection transactions to be serialized
one at a time.
Replace that with a per-collection queue and a single dispatch loop. When a
collection has a batch in flight, newly arriving transactions are queued.
The dispatch loop later drains the queue and merges transactions into one
SeaStore transaction (batch). Resulting in a one build, one ool_write and one journal record for the entire batch.
Only one batch is in flight per collection, so same-collection concurrency is
still prevented and ordering is preserved by the queue.
Callbacks are unchanged: FuturizedStore drains each caller's on_commit before
enqueueing, so only ops are merged. Each caller's promise is fulfilled when the
merged batch commits.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Better-defined generator semantics; many internal improvements,
a couple found bug fixes, minor feature enhancements.
Considerable simplification that streamlines and improves the
baseline performance of library internals. Updates tests and
example documentation.
- Add inclusive()/exclusive() selector endpoints.
- default is still: select { a, b } == [a, b).
- endpoint semantics work for reads, pagination,
range-erase, split planning.
- tests: selector endpoint combinations; endpoint handling; multiple
value types from generators..
- runtime byte-pointer helpers now actually just runtime
Assisted-by: Codex:GPT-5
Signed-off-by: Jesse Williamson <jfw@ibm.com>
The background sync-log-trim thread already skips bucket trim on zones whose
sync module does not export data , but the admin command had no equivalent
guard, so running it on an archive zone could delete instance metadata the
zone is meant to retain.
So, refuse the command on non-exporting zones unless --yes-i-really-mean-it is
given.
Fixes: https://tracker.ceph.com/issues/78364
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Since tentacle, the preferred base image for ceph containers has been
Rocky Linux 10, and the CI tag-naming logic in build.sh already assumes
rockylinux-10 is the default fromtag for every branch except reef and
squid. The actual build default was never flipped, though: anything
that ran build.sh without FROM_IMAGE set (e.g. the release container
job in ceph-build) still got a CentOS Stream 9 base.
Flip the Containerfile ARG and the build.sh fallback to
docker.io/rockylinux/rockylinux:10 so umbrella and later build from
Rocky 10 by default. Builds that want a different base can still pass
FROM_IMAGE explicitly, as the CI pipeline does.
Signed-off-by: David Galloway <david.galloway@ibm.com>
Aggressive Link-Time Optimization (LTO) and stricter linker defaults
in newer toolchains (like GCC 13 on Ubuntu 24.04) can strip or
localize external global variables if their visibility is not
explicitly defined. Tagging g_lockdep with default visibility ensures
it remains accessible in the dynamic symbol table for external
binaries like monmaptool.
Assisted-by: Gemini
Fixes: https://tracker.ceph.com/issues/78321
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
first_interval_in_write can be accessed before it has been initialized. This
is harmless as at worst it will just cause the 1st I/O to a FastEC PG to be
slightly slower than necessary by disabling some optimizations.
However it needs fixing so the teuthology Valgrind job can check for other
issues.
Fixes: https://tracker.ceph.com/issues/76953
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Raise a few low-frequency, high-signal bilog trim events so they are
visible at the default log level:
Trim pace is bounded by the rgw trim config options and these events
fire at most once per bucket or once per trim cycle, so they are not
expected to be chatty.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
The peer -ENOENT handling in RGWReadRemoteStatusShardsCR no longer
consults the local bucket layout (whether Deleted flag set or not),
so its bucket_info member and the matching constructor arg are
not used anymore.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
When a peer returns -ENOENT for a bucket's sync status, the old code only
purged the log if the local layout already carried BucketLogType::Deleted.
That flag may never arrive once the deleting zone removes its own instance
(#70858), so trim aborted with -EINVAL and left bucket.instance metadata
orphaned.
But an all-peer -ENOENT is also ambiguous: a live bucket whose sync status is
not yet initialized looks the same. The Deleted flag is unreliable here,
and the local entrypoint is not authoritative during creation (the
instance syncs to peers before the entrypoint). So on all-peer -ENOENT,
confirm the bucket is really gone by reading its entrypoint from the
metadata master, which is authoritative and available immediately on
creation; if this zone is the metadata master its local read already
suffices. Only on confirmed deletion remove the orphaned instance
directly; on any failure, skip and retry on the next trim cycle.
Fixes: https://tracker.ceph.com/issues/70858
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
An all-peer -ENOENT does not by itself mean a bucket is deleted: a live
bucket whose sync status is not yet initialized returns -ENOENT too. Add
two testcases that verifies we do not accidentally delete instance of a
live bucket:
- test_bucket_log_trim_live_empty_bucket_not_removed: a data-less bucket
never gets sync status, so every peer reports -ENOENT; autotrim must not
remove its instance.
- test_bucket_log_trim_new_bucket_entrypoint_not_synced: a new bucket
whose instance has synced to the secondary before its entrypoint must
survive autotrim. A delay_meta_sync_bucket_entrypoint_store injection
forces that window.
Tests: https://tracker.ceph.com/issues/70858
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Adds test_bucket_log_trim_enoent_race_after_reshard to reproduce the
exact failure from tracker #70858: when autotrim runs on the primary
zone first and fully cleans up (including removing its own bucket.instance
metadata), then an autotrim on the secondary zone asks the primary
and receives -ENOENT.
The secondary hasn't received the BucketLogType::Deleted
flag yet (metadata sync is stalled via delay injection), so the current code
left StatusShards{generation=0, shards=[]} which caused take_min_status()
to fail with -EINVAL, leaving the bucket.instance metadata permanently
orphaned.
Tests: https://tracker.ceph.com/issues/70858
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
awk '/^#/ {print; next} {print | "sort --stable -f"}' .githubmap > .githubmap_sorted
This commit also removes two duplicates for zmc and nmshelke.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/77369
Signed-off-by: Devika Babrekar <devika.babrekar@ibm.com>
Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html
-removed Directory field, as it is no longer needed in form
synchronization
Use seastar::shared_mutex to synchronize transactions when committing
no_conflict ones.
Specifically, this commit achieves conflict/no_conflict trans
synchronization by:
1. The no_conflict trans acquires locks on all the stable extents which
it modifies.
2. The conflicting trans acquires shared locks on all the stable extents
which it modifies.
3. The no_conflict trans releases the locks after its committing.
4. The conflicting trans releases the shared locks once it enter the
prepare pipeline phase.
This commits makes sure that:
1. only transactions accessing the extents that the no_conflict trans
is committing will be blocked.
2. avoid persisting extents that a committing no_conflict trans is
modified
Compared to the old approach, this commit's strategy is much more
straightforward and understandable.
This commit is more performant as the no_conflict trans only blocks
the trans the modified extent set of which intersects with it, while,
in the old approach, all other transactions will be blocked once a
conflicting trans is blocked because it's blocked after it enters the
prepare pipeline phase
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Add tests to validate that CopyObject honors x-amz-tagging-directive.
New tests:
- test_object_copy_retaining_tagging: default copy keeps the source tags
- test_object_copy_tagging_directive_copy: COPY keeps the source tags
- test_object_copy_tagging_ignored_without_directive: a tagging header is ignored without a REPLACE directive
- test_object_copy_replacing_metadata_and_copying_tagging: metadata REPLACE keeps the source tags
- test_object_copy_replacing_tagging: REPLACE applies the new tag set
- test_object_copy_replacing_tagging_with_none: REPLACE with no tags clears them
- test_object_copy_to_itself_replacing_tagging: retag onto the same key
- test_copy_object_replacing_tagging: replaced tags replicate across zones (multisite)
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
CopyObject ignored x-amz-tagging-directive, so REPLACE never took
effect and copies kept the source object's tags. Honor the directive
independent of the metadata directive: REPLACE applies the given tags
(or clears them), COPY keeps the source tags.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
mgr/cephadm: scope .smb pool read cap to its own namespace
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Shweta Sodani <ssodani@redhat.com>
mgr/dashboard: Fix the footer - view all button css in notification panel
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
Samba recently changed CTDB to derive its runtime paths from Samba's
configure options instead of using a monolithic CTDB_RUNDIR[1].
With current packaging[2], the CTDB socket path moved from
/var/run/ctdb/ctdbd.socket to /run/samba/ctdb/ctdbd.socket.
Update the CTDB runtime bind mount target from /var/run/ctdb to
/run/samba/ctdb to match the new CTDB_SOCKETDIR. Additionally,
create the CTDB pid directory, /run/ctdb, and lock directory,
/var/lib/samba/lock/ctdb, on the host side as they are hidden
when their parent directories are bind mounted over.
Ideally these directories should be created by sambacc but sambacc is
included in samba-container images that may be built with different
Samba packages where these path configurations might not hold true.
[1] https://gitlab.com/samba-team/samba/-/merge_requests/4631
[2] https://github.com/samba-in-kubernetes/samba-build/blob/main/packaging/samba-master.spec.j2
Fixes: https://tracker.ceph.com/issues/78309
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
When building with DWZ enabled, the debian packaging fails with:
```
dh_dwz: error: Aborting due to earlier error
```
Running the dwz command manually reveals the root cause:
```
$ dwz -mdebian/ceph-osd-crimson/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug \
-M/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug -- \
debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson \
debian/ceph-osd-crimson/usr/bin/crimson-store-nbd
dwz: debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson: Too many DIEs, not optimizing
dwz: Too few files for multifile optimization
```
The dwz tool has a limit on the number of DWARF DIEs (Debug Information
Entries) it can process. The ceph-osd-crimson binary, being a large C++
executable with extensive template usage, exceeds this limit, causing
dwz to exit with status 1 and fail the build.
This change excludes only ceph-osd-crimson from dwz processing using
the -X flag, allowing other binaries in the package to still benefit
from DWARF compression while avoiding the build failure.
Please note, we always disable DWZ in ceph-build's ceph-dev-pipeline.
Fixes: https://tracker.ceph.com/issues/78303
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The coroutine conversion accidentally passed nullptr to NVME_IOCTL_ADMIN_CMD,
so Identify always failed and SeaStore fell back to st_blksize (e.g. 128K),
breaking journal roll_start alignment on mkfs.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
This adds a Kerberos setup task, tells the Kafka task to emit GSSAPI listeners and broker principals, and updates notification tests to pass Kerberos credentials through to the bucket-notification suite. It enables the RGW Kafka notification coverage to run against GSSAPI-secured brokers.
Signed-off-by: sujay-d07 <sujaydongre07@gmail.com>
* refs/pull/68813/head:
tools/cephfs: make journal pointer/header failure explicit
doc/cephfs: add note explainin the new batching technique
tools/cephfs: document --max-rss usage in first-damage.py
qa: test cephfs-journal-recovery event recover_dentries summary
tools/cephfs: flush every journal event as soon as it is scanned
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
* refs/pull/68924/head:
tools/cephfs_mirror: Fix failed-to-syncing state priority across metrics paths
Reviewed-by: Karthik U S <karthik.u.s1@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Fixes: https://tracker.ceph.com/issues/77813
Signed-off-by: Devika Babrekar <devika.babrekar@ibm.com>
Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html
-conflict while rebasing for SSO related parameters
src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts
-conflict while rebasing for SSO related variables
pybind/rbd: handle non-existent groups properly in list_snaps()
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
smb: add a teuthology test that exercises the gRPC over TCP interface of smb remote-control
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
innernode_split_merge_balancing, omap_iterate, list and monotonic_inc grow
the tree to a fixed depth, so under ASan a larger value reaches the same
depth (same split/merge paths) with far fewer keys. It stays below the
per-kv limit so inserts keep the normal path.
Scale the extent-count bound in innernode_split_merge_balancing with the
value size accordingly.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Fix race hazard in PeeringState::proc_lease where it read
a std::optional value which could be modified by another
thread at the same time causing an abort.
There is already a lock to protect accesses to this state
to make them thread safe, but the code wasn't using it.
Fixes: https://tracker.ceph.com/issues/54932
Assisted-by: IBM Bob 2.0.1
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
fixes: https://tracker.ceph.com/issues/78247
why the changes were made:
The Services table had autoSave enabled by default, which persists table state (including sort configuration) to browser localStorage. This created a conflict because:
The table uses server-side pagination ([serverSide]="true")
The table auto-reloads every 5 seconds ([autoReload]="5000")
On initialization, the table would load stale sort configuration from localStorage
This cached sort state would override the intended default sort (by service_name ascending)
User sort interactions would be saved to localStorage but could conflict with server-side sort handling
Signed-off-by: Naman Munet <naman.munet@ibm.com>
After an OSD that has not crashed is marked down there is a race hazard
where a message can be deliever to the OSD a second time. Different
asserts are hit depending on which message is duplicated.
This issue is seen in tests which mark osds down as an error inject
and very occasionally in the field where there are other issues
(e.g. network problems) that are causing osds which have not
crashed to be marked down.
The fix is to change the order in which the OSD processes a
new epoch and OSDMap which marks the OSD as down, it must
first consume the map so that old messages will be filtered
out and then call rebind to reset all the connections.
Fixes: https://tracker.ceph.com/issues/58417
Assisted-by: IBM Bob 2.0.1
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
When add_fscrypt_key() failed it jumped to the err label, which then
read the uninitialized keyid buffer into a key specifier and issued a
key removal for it. add_fscrypt_key() only populates keyid on success,
so this read uninitialized stack memory. Return the error directly
instead, since no key was added and there is nothing to remove.
Also stop overwriting the original error with remove_fscrypt_key()'s
return value on the policy-set failure path, so the real failure is
reported to the caller.
Fixes: https://tracker.ceph.com/issues/78246
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
_wrap_name() returns bool, but two encryption error paths did "return r"
with r < 0, which converts to true. Callers then treated the failure as
success and used the unencrypted name. Return false instead.
Fixes: https://tracker.ceph.com/issues/78246
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
mount() holds client_lock through a unique_lock (cl), but the dummy
encryption path unlocked and relocked the underlying mutex directly,
bypassing cl. On fscrypt_dummy_encryption() failure it returns early
without relocking, so cl's destructor then unlocks an already-unlocked
mutex, which is undefined behavior. Only reachable when
client_fscrypt_dummy_encryption is enabled.
Unlock/relock through cl so its ownership state stays consistent.
Fixes: https://tracker.ceph.com/issues/78246
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
* refs/pull/69320/head:
qa:/cephfs: separate out ceph-fuse upgrade tests under fs:upgrade
qa/cephfs: reject ceph-fuse upgrade jobs to reduce failure noise
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
During heavy backfill, sweep_out_pending_msgs_to_sent() could accumulate
more than 65535 buffer fragments in a single bufferlist, overflowing
seastar's uint16_t packet fragment counter and triggering an assertion.
Limit the sweep to crimson_osd_max_send_buffers fragments
per write; remaining messages are sent in subsequent dispatch iterations.
Fixes: https://tracker.ceph.com/issues/75600
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Atollon return status as a plain string ("OK") instead of
the dict format ({"health": "OK", "state": "Enabled"}) used by
other BMCs. This caused KeyError/AttributeError.
Added _get_health() helper that handles both formats:
- dict with nested health key
- plain string status
- missing or non-dict components
Signed-off-by: Afreen Misbah <afreen@ibm.com>
Scope the .smb pool read cap to the same namespace already
used by the adjacent cluster-meta rwx cap.
Fixes: https://tracker.ceph.com/issues/78235
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Replication (objects) from Source Zone title is misleading, the panel actually shows How many objects are being replicated per second
Fixes: https://tracker.ceph.com/issues/78230
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Prefer the distro-packaged sccache (installed via dnf) over downloading a
release tarball with curl. Fall back to the curl download when the package
is absent or on non-dnf distros, so existing behavior is unchanged.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
when referencing the "Backport" field in tracker.ceph.com, we
used "Backports". but the field is named "Backport" not "Backports".
in this change, we replace "Backports" with "Backport" when it
references this field.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
We already re-opt into telemetry immediately after upgrading,
but this warning still sometimes comes up and is detected
by the badness check. Since we are following the normal
procedure to silence the warning, the next step is to
just ignore this.
Fixes: https://tracker.ceph.com/issues/77382
Signed-off-by: Laura Flores <lflores@ibm.com>
Store sync completion time (last_synced_snap.sync_time_stamp)
as utime_t (Unix epoch) instead of coarse monotonic time so
the metric is meaningful to operators.
Fixes: https://tracker.ceph.com/issues/76506
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Long-running local git operations and Redmine API calls can cause the idle
keep-alive connection to api.github.com to be closed by the remote server.
When the script later attempts to post automated PR comments, it fails with
http.client.RemoteDisconnected.
Fix this by mounting an urllib3 Retry adapter to the requests.Session,
automatically re-establishing dropped connections up to 3 times on
transient network or server errors (500, 502, 503, 504).
Assisted-by: Gemini
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
The "disabled" search only matches sccache's Disabled("disabled")
payload, not "dist-client feature not selected" or the NotConnected
states. Those reach the success branch, so the job counts become
SchedulerStatus-NOTFOUND, JOB_POOLS is never defined, and ninja
rejects the manifest. Key off the JSON lookup's error instead to
cover every non-dist state.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Recompute ro_start and ro_end with compute_ro_range() and compare them
against the cached values, and cover single-shard maps.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
slice_map() accumulated the slice bounds against the wrong operands,
and included parity shards in ro_start and ro_end, which only track
data.
Introduced by 9e2841ab16 ("osd: Introduce optimized EC").
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
distutils_add_cython_module() runs setup.py with WORKING_DIRECTORY set
to the source directory, so every build leaks
{rados,cephfs,rbd,rgw}_processed.pyx into the source tree as untracked
files. Write the file to CYTHON_BUILD_DIR (already exported by that
CMake function) when set, keeping the cwd fallback for standalone
invocations.
The .pyx no longer sits next to the bindings' .pxd files, so add each
binding's source directory to include_path. Drop the build_dir
argument: it has no effect on absolute source paths.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
BackfillState::Enqueuing unconditionally posts RequestWaiting{}
when budget available return false. Waiting can exit only via
ObjectPushed which requires an in-flight push. If the budget
is exhausted with no pushes in flight no objectPush will ever
arrive and the PG stalls permanently.
Solution: Introduce a dedicated BudgetBlocked state. When budget_available()
is false and tracked_objects_completed()==true post RequestBudgetBlocked{}
instead of RequestWaiting{}.BudgetBlocked calls get_backfill_throttle()
to suspend until a slots free up, then fires BudgetAvailable{} which
transitions directly back to Enqueuing to dispatch the next batch of pushes.
Fixes: https://tracker.ceph.com/issues/77804
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
crimson/osd: fix boot crash when a peer OSD was purged across an osdm…
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Previously, ConflictSimulationCheck only anticipated merge conflicts
when simulating upstream cherry-picks. If a branch-specific commit (such
as test repo branch adjustments or backport metadata changes) failed to
apply cleanly to the target base branch, the script threw an unhandled
GitCommandError and crashed with an unhandled traceback.
Wrap branch-specific commit cherry-picks in a try...except block. When a
conflict occurs, abort the cherry-pick, record a "Rebase Required"
simulation failure in the audit report, and either prompt the user in
interactive mode or gracefully halt check execution in CI mode.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
release.gpg no longer exists at download.ceph.com/keys/ (moved to
/keys/old/, last updated 2020-08-21). Only release.asc is current and
maintained. autobuild.gpg still resolves but is inconsistent with this
and could rot the same way.
repo_gpgkey() now returns the .asc URLs for both release and autobuild
keys. rpm --import and apt accept ASCII-armored keys directly, so no
other code changes are needed.
Fixes 404s seen in qa/workunits/cephadm/test_repos.sh and reported by
users running cephadm add-repo on current distros.
Fixes: https://tracker.ceph.com/issues/78179
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Previously, workflow routing and PR state management were split between
the GitHub Actions workflow and the underlying PTL script, leading to
stale branch protection states, API pagination bugs, and blocked merges
during manual overrides.
This refactors the workflow into a centralized state machine and decouples
status reporting from the audit tool:
* Decouple State Management: Remove `--audit-label` from `ptl-tool.py` and
move all commit status and label transitions (`releng-audit-pass`,
`releng-audit-fail`) into a dedicated workflow reporter step.
* Non-Blocking Review Comments: Update `ptl-tool.py` in CI mode to post
audit findings as `COMMENT` events rather than `REQUEST_CHANGES`,
preventing stale review states from blocking PRs after an override.
* Add Step Summaries: Implement `write_ci_summary()` in `ptl-tool.py` to
output rich Markdown diagnostic reports directly to `GITHUB_STEP_SUMMARY`.
* Atomic Override Handling: Add `setOverrideStatus()` and `executeOverride()`
to push an immediate `success` commit status when `/audit override` or the
override label is applied, unblocking required branch protection checks.
* Safe SHA & Label Lookup: Replace the unpaginated Bash `curl` request with
a reusable `getPrSha()` helper and evaluate overrides directly within the
JavaScript router.
* Handle Cancellations Safely: Map `cancelled` workflow outcomes to an
`error` commit status rather than defaulting to `failure`.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
mgr/cephadm: track SMB rgw_creds_uri as a daemon dependency
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Shweta Sodani <ssodani@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Probe download.ceph.com before each stable add-repo and skip releases
that are not published for the current distro. This avoids apt/yum
failures on newer OS versions (e.g. Ubuntu 24.04/noble) where older
stable trees like quincy and reef were never published.
Fixes: http://tracker.ceph.com/issues/77248
Signed-off-by: Yael Azulay <yazulay@redhat.com>
slow_op_pools can be empty (all poolids 0/out-of-range) while
slow_op_types is not, making max_element return end(); dereferencing
it is UB.
Fixes: https://tracker.ceph.com/issues/78184
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
When enable_haproxy_protocol is set, _haproxy_hosts() can collect the
same address more than once. Store collected IPs in a set and return a
list so HAProxy_Hosts contains only unique entries.
Fixes: https://tracker.ceph.com/issues/78094
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
osd: Enable support for omaps in EC pools
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Subsequent commit will introduce a dedicated ceph-fuse upgrade
sub-suite (under fs:upgrade) to still keep track of failing
jobs.
Fixes: http://tracker.ceph.com/issues/76780
Signed-off-by: Venky Shankar <vshankar@redhat.com>
The removed_ranges erase loop in omap_get() called erase(out_it->first),
which invalidates out_it, then did ++out_it on the dead iterator (UB).
When adjacent keys were erased this skipped keys that should have been
removed:
Expected: { "k01", "k05", "k06" }
Actual: { "k01", "k03", "k05", "k06" }
Extract the loop into a static remove_keys_in_ranges() helper so it can
be unit tested without an ObjectStore, and rewrite it as a range erase:
removed_ranges is a sorted map of non-overlapping [start, end) ranges.
Fixes: https://tracker.ceph.com/issues/78177
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
ceph_module.mgr_log() emitted every record at dout(0): a record either
passed the Python-side gate and went unconditionally to both the log
file and the in-memory recent buffer, or never reached the C++ log at
all. The two-level debug_mgr semantics (file level / memory level)
never applied, so capturing third-party DEBUG logs meant sending each
record to the log file and a log_max_recent slot; with the kubernetes
client logging multi-megabyte HTTP response bodies at DEBUG, the
recent buffer alone could grow to tens of gigabytes.
Pass the record's level through mgr_log() and emit at the mapped ceph
level: CRITICAL/ERROR at 0, WARNING at 1, INFO at 2, DEBUG at 20.
INFO maps to 2 so it stays visible in the log file at the default
debug_mgr=2/5; DEBUG maps to 20 so floods are dropped there, while
debug_mgr=2/20 captures third-party DEBUG in the in-memory buffer
without touching the log file. The Python-side gate follows the
higher of the two debug_mgr levels so memory-only records still
reach the C++ side.
Fixes: https://tracker.ceph.com/issues/77633
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
MgrRootHandler captures third-party logs from the Python root logger
and emits them through the _ceph_log method of whichever mgr module
last configured logging. A handler on the root logger outlives any
single module, so once that module unloads the handler emits through
a stale pointer until another module reconfigures logging.
Add a free mgr_log() function to the ceph_module binding, emitting
via dout() like PyModuleRunner::log but with no PyModule involved,
and rewrite MgrRootHandler to route through it. The handler no
longer references a module instance, so it cannot go stale.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Introduce a supported whitelist of component tags (e.g., rgw, cephfs,
rbd, cephadm) and automatically extract matching labels from the queued
GitHub pull requests during integration branch builds.
When building or updating a Redmine QA tracking ticket:
* Append matching component tags directly to each PR line in the ticket
description (e.g., `(tags: rgw, cephfs)`).
* Populate the `tag_list` attribute on the Redmine issue to seamlessly
integrate with the RedmineUP Tags plugin.
* When updating an existing QA ticket, preserve any pre-existing tags
already attached to the issue and merge them with the active PR tags.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Introduce the `--qe-label <pr-label>` option to streamline daily integration
testing and Redmine QA tracking for backports.
Using `--qe-label` automatically implies `--integration` and `--pr-label`,
and manages Redmine QA tracking tickets via custom field 45 (Ceph PR Label):
* Searches for open QA tickets matching the specified PR label and target
release branch.
* If exactly one open ticket is found, prompts the user to either update
the existing ticket or create a new one.
* If no open ticket is found, automatically defaults to creating a new
QA ticket with the label stored in its metadata.
* Adds pre-flight validation to ensure all labeled PRs target the exact
same base branch, preventing ambiguous or mixed-release merges.
* Halts execution if multiple ambiguous open QA tickets exist for the
same label and release branch.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Refactor the Commit Parity Visualizer generation to use a deterministic
ordered grouping loop instead of a state machine. Missing commits are now
reliably printed within their respective original PR blocks, and any extra
backport commits are isolated at the bottom of the table.
In addition, add automated checks to detect "scrambled" commits during
backports:
* Verify that commits from multiple upstream PRs are not interleaved or
scrambled together.
* Verify that commits within an individual upstream PR are applied in
their exact original chronological sequence.
If scrambled or out-of-order commits are detected, log a failure in CI mode
or prompt the user for interactive resolution to ensure clean backport
history and reduce regression risks.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
* refs/pull/70139/head:
releng-audit: clear state labels and set pending status on comment triggers
.github/workflows/releng-audit: cleanup branches
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
When /audit retest or /audit test-branch is invoked via an issue_comment event, GitHub Actions does not natively bind the execution to the PR's HEAD commit SHA. As a result, the check widget on the PR conversation tab remains in its previous state (often showing as failed), and any lingering releng-audit-fail or releng-audit-pass labels remain attached.
This introduces a triggerAuditRun helper function to the workflow router that:
- Removes any existing releng-audit-fail or releng-audit-pass labels from the pull request to reset the label state machine and prevent subsequent synchronize events from halting prematurely.
- Explicitly creates a pending commit status on the PR's HEAD SHA via the GitHub REST API, ensuring visual consistency and feedback inside the PR UI while manual or test-branch audits execute in the background.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
- Added a new shared overview component for the resource page
- Added a new host-action service
- Fixes: https://tracker.ceph.com/issues/76712
Signed-off-by: Syed Ali Ul Hasan <syedaliulhasan19@gmail.com>
Allow authorized maintainers and release managers to test audit workflow
changes on a pull request using an alternate repository branch.
When a user comments `/audit test-branch [branch-name]` on a PR:
* Verify user authorization and confirm the target branch exists via the
GitHub API (defaulting to `testing/releng-audit` if unspecified).
* If valid, activate audit execution and set the `checkout_ref` output
to dynamically override the repository checkout target.
* If the branch does not exist or the user lacks permissions, fail the
job cleanly and leave an explanatory comment.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Add a test case that uses the API directly instead of through the
ceph-smb-ctl command line. Ideally, this is validation of access tot the
api and a precursor to using the grpc api as a component throughout the
smb workunit tests.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
We recently added testing for the remote control service via the
local unix socket but had no coverage for the gRPC based
mode of operation. This adds a test case that configures and deploys
the needed TLS certs and sets up and tests remote control with gRPC.
Fixes: https://tracker.ceph.com/issues/76676
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add an argument to the cephadm shell wrapper function that
makes it easy and obvious where to add volume mappings.
This will be used in a future change to map in a directory
containing TLS certificates and such.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Ensure that the testdir (parent dir of generated tls certs)
is made available to the smb test configuration used by
the workunit tests.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Update template.py jinja2 templating, to add support for extracting the
content of tls certs, generated by the ssl_keys task.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
Fixes: https://tracker.ceph.com/issues/78111
Signed-off-by: David Galloway <david.galloway@ibm.com>
\(POOL_FULL\) only matches the "Health check failed/update" cluster-log
format (code wrapped in parens). It misses the "Health check cleared"
format, where the code is not parenthesized (see src/mon/Monitor.cc),
so a pool clearing its full state mid-upgrade-test was not ignorelisted.
Drop the escaped parens so the pattern matches both forms, consistent
with other bare entries already in this file (OSD_ROOT_DOWN,
MDS_INSUFFICIENT_STANDBY).
Fixes: https://tracker.ceph.com/issues/78149
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
`RecordSubmitter` dispatches a small record (or the pending batch) only
once the journal is fully drained (`state == IDLE`): `submit()` takes
the direct-write path only at IDLE, and `decrement_io_with_flush()`
flushes the pending batch only after *all* outstanding io has
completed. The remaining dispatch triggers never fire for 4K-class
records: `preferred_fullness` measures the padding ratio of a single
block, and small records do not reach the batch capacity.
At low-to-mid client queue depth this degenerates into strict
alternation: one journal io in flight, each batch carrying only the
records that arrived during the previous device write. Measured with
4K randwrite QD20 against a single crimson OSD (circular-bounded
journal): io_depth 1.07, 1.9 records/io, ~2.2ms `submit_record`
latency -- 1.5x the raw device write+flush latency, since each record
first waits ~0.5x for the in-flight io to drain. `io_depth_limit`
(default 5) is never approached.
Dispatch whenever the device has a free io slot instead:
- `submit()`: take the direct-write path whenever `state != FULL`;
records are written greedily while slots are free, and batching
kicks in once `io_depth_limit` is reached.
- `decrement_io_with_flush()`: flush the pending batch into the freed
slot instead of waiting for IDLE.
Ordering is unaffected: a batch's `write_base` is assigned at dispatch
from `journal_allocator.get_written_to()`, so the on-disk layout
follows submission order; completions are reported in order via the
`device_submission` exit turnstile and the exclusive `finalize` phase,
so a record is never acked before all earlier records are durable.
Replay stops at the first invalid record, and anything beyond such a
hole was never acked. The same concurrency already occurs through the
`SUBMIT_FULL` path under high load; this extends it to the
low-fullness regime.
## Benchmark
Ceph 21.3.0 dev (RelWithDebInfo), single crimson OSD, 4 reactors
pinned to 4 cores, fio rbd engine on 4 disjoint cores, size-1 pool,
24 GiB prefilled image, consumer NVMe without PLP (~0.7ms fdatasync).
4K randwrite QD20, 180s, `RANDOM_BLOCK_SSD` (circular-bounded journal):
| metric | before | after |
|-------------------------|-------:|--------------:|
| IOPS | 5,800 | 10,918 (+88%) |
| avg client latency | 3.4ms | 1.83ms |
| avg journal io_depth | 1.07 | 3.18 |
| records per io | 1.9 | 1.01 |
| `submit_record` latency | 2.2ms | 1.05ms |
4K randwrite, 45s per queue depth, `RANDOM_BLOCK_SSD`:
| iodepth | 1 | 4 | 16 | 64 |
|---------|----:|------:|------:|--------:|
| before | 452 | 1,270 | 3,470 | ~10,300 |
| after | 536 | 1,908 | 9,337 | 15,333 |
4K randwrite QD20, 180s, `SEGMENTED` (segmented journal and ool
writer, both built on `RecordSubmitter`):
| metric | before | after |
|----------------------|-------:|--------------:|
| IOPS | 2,728 | 6,123 (+124%) |
| avg client latency | 7.3ms | 3.26ms |
| avg journal io_depth | ~1 | 2.26 |
A classic BlueStore OSD on the same 4-core/QD20 harness does 10,243
IOPS; crimson previously trailed it by 1.8x and now matches it.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Add rgw_creds_uri as a Dep.FIELD in get_dependencies and include it in
needs_redeploy_prefixes so a change triggers REDEPLOY.
Also fix _save_pending_rgw_config to use cred_map.get() instead
of a bare dict lookup.
Fixes: https://tracker.ceph.com/issues/77763
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
as per the newer eslint rules. i limited it to only prettifying the code
instead of changing anything functional. running `npm run fix` did the
magic
Fixes: https://tracker.ceph.com/issues/77112
Signed-off-by: Nizamudeen A <nia@redhat.com>
mgr/cephadm: suppress CEPHADM_FAILED_DAEMON for services being removed
Reviewed-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
mostly accessibility issues like
- A label component must be associated with a form element
Fixed by adding `for` to labels and sometimes replacing label with
legend
- Elements with interaction handlers must be focusable
- click must be accompanied by either keyup, keydown or keypress event for accessibility
- <button> should have content
Fixes: https://tracker.ceph.com/issues/77112
Signed-off-by: Nizamudeen A <nia@redhat.com>
deprecate the older eslint packages and install the unified official
packages for eslint and migrate .eslintrc to eslint.config.js.
also bump prettier
Fixes: https://tracker.ceph.com/issues/77112
Signed-off-by: Nizamudeen A <nia@redhat.com>
_handle_osd_map() computes start = superblock.get_newest_map() + 1. A
freshly created OSD has get_newest_map() == 0, so start stays 1. When
such an OSD joins a cluster whose oldest map is already trimmed past
epoch 1, the gap [start, first - 1] is entirely below the trim lower
bound, so neither osdmap_subscribe guard above fires, and start stays 1.
store_maps() then stores only [first..last], but committed_osd_maps()
iterates from start=1 and calls get_local_map(1), which reads the
never-stored osdmap.1, hits ENOENT, and aborts. The OSD crash-loops
without ever activating a PG.
Clamp start to first in that case: the maps below first are gone, and
first is a full map, so it's a valid starting point. Classic OSD does
the same unconditionally via
start = std::max(superblock.get_newest_map() + 1, first).
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Translate sqlite3.DatabaseError into a generic StoreUnavailable in
sqlite_store.py and handle it in cli.py and module.py instead of
letting it surface as an unhandled exception
Fixes: https://tracker.ceph.com/issues/78125
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
The dashboard backend API suite occasionally fails in CI on the job's
7200 second timeout alone, with every test passing. Suite runtime
varies with builder load: identical runs of the same 311 tests took
anywhere from about 5400 to over 7000 seconds, and teardown needs
roughly another 180 seconds on top. On the runs that landed at the
high end, only 60 to 80 seconds of margin were left before the
timeout fired. Cutting setup overhead gives the suite that margin
back.
DashboardTestCase.setUpClass() restarted every mgr twice: once in
MgrTestCase.setup_mgrs(), and again right after in _assign_ports(),
whose only job is to set the per-mgr dashboard port, which has to
happen while the mgrs are down.
Let subclasses declare their module ports in a MODULE_PORTS class
attribute instead. setup_mgrs() assigns them while the daemons are
already stopped, so the single restart it performs picks them up and
the second stop/fail/restart cycle disappears.
The assignment loop moves to _assign_module_ports(). _assign_ports()
keeps its old stop/assign/restart contract for callers that reassign
ports outside class setup (test_prometheus, test_dashboard,
test_module_selftest) and delegates to the shared helper.
With 66 dashboard test classes per API suite run this removes one
full mgr bounce per class, roughly 10 to 15 minutes of wall clock in
the CI runs examined.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
When a service is deleted, a daemon belonging to it may transiently
enter a failed state before orphan cleanup removes it. This happens
when there is a race between dependency-changed REDEPLOY and a
service removal going on paralelly.
Skip daemons in update_failed_daemon_health_check whose service is
already in spec_deleted.
Fixes: https://tracker.ceph.com/issues/77944
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
RCA done by copilot inside the github actions
Root cause: the workflow is passing the default token to a third‑party action that calls PR/diff APIs, and in this run that token doesn’t have the required permission scope for that request.
From the failing job log:
There seems to be an error in an API request
This is usually due to using a GitHub token without the adequate scope
From the job definition (.github/workflows/check-license.yml):
It uses JJ/github-pr-contains-action
Token passed is ${{ github.token }}
Fix
Grant explicit read permissions needed for pull request metadata/content in this workflow (or job), so the token has consistent scope when the action fetches diff data.
Why this solves it
The action needs to read PR diff data. Explicitly setting pull-requests: read (and contents: read) prevents under-scoped default token behavior that triggers the API failure.
Signed-off-by: Nizamudeen A <nia@redhat.com>
mgr/smb: add fruit:nfs_aces=no for macOS compatibility mode
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
qa/workunits/smb: work around test failures by pinning smbprotocol version
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Disable type checking in the __preprocess function of
diskprediction_local. This function is already collecting a lot of
`type: ignore` comments and adding more is getting difficult due to
the structure of this function. This time we were hitting errors
like:
diskprediction_local/predictor.py:160: error: Value of type "NDArray?[Any]" is not indexable [index]
diskprediction_local/predictor.py:161: error: "NDArray?[Any]" has no attribute "shape" [attr-defined]
on python 3.12.
Fixes: https://tracker.ceph.com/issues/78117
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Calling `list_snaps()` on a non-existent group crashed the entire
Python process with a `free(): invalid pointer` error. This happened
because an unexpected `-ENOENT` error returned by `rbd_group_snap_list2`
and the exception was raised without updating the num_group_snaps to 0
from 10. Later the Cython `__dealloc__` wrapper attempted to free garbage
of uninitialized pointer spaces resulting in a crash.
Fix this behavior by resetting `self.num_group_snaps` to 0 inside
`GroupSnapIterator` before raising exception to prevent memory
corruption and accessing uninitialized pointers during cleanup call
Now `list_snaps()` consistently raise an `ObjectNotFound` error when
invoked on a non-existent group.
Also Extended the `TestGroups` test fixture with `self.dne_group` to
validate the expected error path for `list_snaps()`.
Fixes: https://tracker.ceph.com/issues/78034
Signed-off-by: VinayBhaskar-V <vvarada@redhat.com>
1. Make cmp_vals() to take in ObjectOperation to allow
it work with ObjectWriteOperation.
2. Add comments for cmp_set_vals() regarding how different
omap values may be set.
3. Add unit tests to verify the transactional nature of
ObjectWriteOperation used when setting different omap values.
Fixes: https://tracker.ceph.com/issues/76622
Signed-off-by: Yixin Jin <yjin77@yahoo.ca>
Implement intelligent action selection for SMB daemon updates by tracking
feature changes as dependencies. When SMB service features change (e.g.,
adding cephfs-proxy, remote-control, keybridge), the system now triggers
a REDEPLOY instead of RECONFIG, as these changes require container
structure modifications.
Fixes: https://tracker.ceph.com/issues/77762
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
Work around new tests failures (in the TestHostsAccessToggle1 tests)
where the error/exception handling has apparently changed by just
pinning the version (by excluding the new v1.17.0 and newer versions).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
When a mirrored directory hits the consecutive failure threshold, the
failed flag stays set until the next successful sync. On retry, the
directory can be actively syncing while failed remains true, but
dump_sync_stat() and update_directory_current_sync_perf_counters()
checked failed before current_syncing_snap and kept reporting "failed".
Introduce get_dir_sync_state() and use it in peer_status, omap
persistence, and per-directory perf counters so syncing takes precedence
over the sticky failed flag.
Fixes: https://tracker.ceph.com/issues/76616
Signed-off-by: Kotresh HR <khiremat@redhat.com>
UI has an async validator which calls the GET bucket API to make sure
the bucket name doesn't exist, but the proxy
was not properly handling the http_status_codes which results in raising
a massive traceback in logs whenever you type things in the bucket name
field. So handling that gracefully by capturing the proper status codes
for both RequestException and DashboardException
BEFORE
```
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in _call_
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get
result = self.proxy(daemon_name, 'GET', 'bucket', {'bucket': bucket})
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 213, in proxy
raise DashboardException(e, http_status_code=http_status_code, component='rgw')
dashboard.exceptions.DashboardException: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000f14e08c1af0d5615-006'
b'71f54a7-3bc6-default","HostId":"3bc6-default-default"}')
2024-10-28T09:08:55.990+0000 7f89e045b640 0 [dashboard INFO request] [::ffff:10.74.18.122:51853] [GET] [500] [0.007s] [admin] [200.0B] /api/rgw/bucket/bucket-das
```
AFTER
```
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.services.exception] Dashboard Exception: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000c029a81e5d154844-006'
b'a4e183c-14251-default","HostId":"14251-default-default"}')
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.tools] [::ffff:192.168.100.1:60408] [GET] [404] [0.078s] [admin] [200.0B] /api/rgw/bucket/testsa
```
Fixes: https://tracker.ceph.com/issues/78038
Signed-off-by: Nizamudeen A <nia@redhat.com>
Blob is tracking whole AUs in SharedBlob tracker.
But the extents may account for just a portion of AUs.
We need to compare what would Blob release against SharedBlob tracker.
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
Split BlueStore::_do_remove into 2 parts:
- part that removes object (in _do_remove)
- part that processes head object and tries to unshare blobs
(new function _maybe_unshare_on_remove)
No logic changes.
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
iIf the journal pointer and/or header is missing or corrupt, the tool silently
returns with no-op. There is no dentry recovery happening if any of the objects
are missing but it should be explicit to the user.
This still doesn't change returning errnos on failures to maintain status quo
until https://tracker.ceph.com/issues/77913 is addressed
One important change: The condition to scan events is now changed from
header_present to header_valid. The reason for this can be understood by
reading JournalScanner::scan_header -- if the header could not be decoded, it
is set to null, using this in JournalTool::recover_dentries would segfault and
other corruptions include bad magic or inconsistent offsets. A recovery should
not proceed with distorted magic/offset.
Fixes: https://tracker.ceph.com/issues/76422
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
NOTE: exclusive to only `event recover_dentries`
JournalScanner::scan() runs a loop and loads the entire decoded journal in an
`EventMap` called `events`. This can severely shoot up the RSS if the journal
is huge (unflushed journal) making the tool less efficient.
The way to curb the memory bomb is to scan and flush journal in batches. The
flush boundary is based on the batch size and at event EVENT_SUBTREEMAP or
EVENT_SUBTREEMAP_TEST(subtree event(s) mark a clear boundary). The knob to
control this setting is via providing `--max-rss <bytes>` with the journal
tool command. This patch divides the provided limit by three to create the
batch since it is observed that the decoded journal events weight on an avg
~3x the encoded stream.
Also: Currently, the events map retains all decoded events until all dentries
are recovered, causing unnecessary memory retention.
Fix this by erasing each event from the map right after it is processed.
While the primary RSS bloat is caused by dnbl holding the encoded stream
in memory, this incremental cleanup provides a minor optimization until
a permanent fix is implemented (https://tracker.ceph.com/issues/77909).
Partially-Fixes: https://tracker.ceph.com/issues/76422
Co-authored-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
As we do not yet scrub scheduling in Crimson, the
OSDs do not perform deep scrubs. last_deep_scrub_stamp thus
stays at epoch 0 for all PGs. The monitor flags these as overdue
immediately, causing spurious test failures — particularly in tests
that restart OSDs (e.g. crimson_fio_restart), where PGs reaching
active+clean state trigger the health check.
Fixes: https://tracker.ceph.com/issues/77929
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Add documentation explaining that rbd mirror resync only copies
image state and data up to the last mirror-snapshot for
snapshot-based mirroring. Document the requirement to create
a new mirror-snapshot on current primary so that new changes
(including resize operation) gets reflected on secondary after
resync.
Fixes: https://tracker.ceph.com/issues/78050
Signed-off-by: Miki Patel <miki.patel132@gmail.com>
CherryPy and packaging are listed as runtime dependencies of
ceph-mgr-modules-core, but none of the modules shipped in that package
import either one. CherryPy is used by the dashboard, cephadm, and
prometheus modules, which list it in their own .requires; packaging is
not imported by any shipped mgr code. Both entries therefore pull
python3-cherrypy and python3-packaging into ceph-mgr-modules-core for no
reason.
Drop them. The remaining entries (natsort, requests, python-dateutil,
pyyaml) match what the core modules import; prettytable is declared
separately in debian/control.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Crimson recovery could use a neighbor clone as a clone_subset source
during osd_recover_clone_overlap without checking whether that clone
was safe to read. Classic OSD takes a read lock on the neighbor clone
via try_lock_for_read() before using it.
Under snap trim + recovery stress, recovery could clone_range from a
clone that snap trim was modifying, producing wrong object content
(ceph_test_rados: incorrect buffer at pos 1462272).
calc_clone_subsets() and calc_head_subsets() now return clone overlap
candidates in preference order only. ReplicatedRecoveryBackend locks
the first usable neighbor clone from each list and holds those read
locks until push/pull recovery completes. RecoveryCloneLockManager
releases locks in its destructor so move-assign cannot leak a held
lock and block snap trim.
Fixes: https://tracker.ceph.com/issues/76499
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
AsioFrontend::pause() (and stop()) call ConnectionList::close() from
the realm reloader thread. That invokes socket.close() on every live
connection from a thread that isn't the connection's asio strand, so
the reactor deregisters each socket's descriptor_state racing against
the coroutine that handles the connection. If a keep-alive connection
is starting its next async_read_some() on its strand at the same time,
epoll_reactor::start_op() dereferences the descriptor_state that close
has already freed and radosgw segfaults in an io_context_pool thread.
This matches the race pattern fixed for timeout_timer in 86ad0b891b:
any caller that touches socket state from outside the strand needs to
be serialized with the coroutine.
Store the coroutine's strand executor on Connection and dispatch the
close onto that strand in ConnectionList::close(). The lambda captures
an intrusive_ptr to Connection so the object lives until the close
actually runs, since connections.clear() unlinks the list entry right
after dispatch.
Tracker: https://tracker.ceph.com/issues/76094
Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
Update `unittest_rgw_asio_frontend` to run the `io_context` and
execute in a coroutine to match the environment in RGW.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Previously if we received an rvalue function object we would call a
moved-from object on all but the first thread. In practice this didn't
cause an issue, but only because of the functions we were passing.
Fixes: https://tracker.ceph.com/issues/76094
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
This is essentially a special case of an optional and it should act
like one.
I asked the original author to updated it but they didn't want to, so
I did it myself.
Fixes: https://tracker.ceph.com/issues/76094
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Without this we can end up in an infinite loop where we just fill the
log with -ECANCELED from `get_next()`. Duplicates a check in the
`blocked_operations` loop.
Fixes: https://tracker.ceph.com/issues/76094
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
We still had a part of the document saying we only support the default
port. We added service colocation via custom_ports/bind_addrs a while
back so the doc has been out of date.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
This commit enables fast EC tests such that they are run for crimson
OSDs. This is done towards an effort to establish reliable fast EC
support in crimson.
These tests where previously enabled as part of another PR but
ultimately were not merged due to insufficient confidence.
Fixes: https://tracker.ceph.com/issues/77910
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
cephadm ls may transiently report state='unknown' for daemons that are
still starting up. The state mapping in _refresh_host_daemons mapped
'unknown' to DaemonDescriptionStatus.error, which caused
update_failed_daemon_health_check to fire CEPHADM_FAILED_DAEMON for
daemons that were simply mid-startup. The warning clears on the next
refresh once the daemon reaches a real state, but it causes spurious
QA failures across many test suites.
Map 'unknown' to DaemonDescriptionStatus.unknown instead — the enum
value already exists and is handled by to_str() and the dashboard.
get_error_daemons() only checks for .error, so unknown daemons no
longer trigger the health warning.
Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
Add support for configuring monitoring_addr and bind_addr
for NFS daemons managed by cephadm.
Also restores required RADOS configuration blocks and
updates tests accordingly.
Fixes: https://tracker.ceph.com/issues/74403
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>
Remove readthedocs-sphinx-search from the RTD build environment because
it seems to be unused and unsupported.
RTD has archived the readthedocs-sphinx-search repo since Apr 2025.
The docs website does not seem to show such function, the search box
in the top-left corner requires a search to be submitted and is not
live.
Fixes: https://tracker.ceph.com/issues/72918
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
When macOS client compatibility is enabled, add the global Samba
configuration option 'fruit:nfs_aces = no' to disable NFS ACE
handling in the fruit VFS module. This improves compatibility
with macOS SMB clients by preventing conflicts between NFS-style
ACLs and macOS extended attributes.
The configuration is automatically applied when:
- Creating a new cluster with --client-compat=macos
- Updating an existing cluster to macOS mode
Signed-off-by: Shwetha K Acharya <Shwetha.K.Acharya@ibm.com>
The do_transaction latency histograms were bucketed and summed in
microseconds, but with seastar::lowres_clock (~task_quota, ~0.5ms) and
the ms-scale buckets we keep - microsecond values only add noise to the
exported numbers.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
The do_transaction latency histograms now source their samples from
seastar::lowres_clock (~0.5ms default). Drop the sub 1ms buckets.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
do_transaction timing runs on the hot path and used for
latency histograms. std::chrono::steady_clock's per-call clock_gettime
is too costly for these needs. seastar::lowres_clock uses cached now()
at every task_quota poll (~0.5ms) - which is granular enough.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
on hot shards a small fraction of transactions dominate latency,collecting
one histogram for all is not enough.
Add two more histograms of the same strcutre for txns that take more
than 5ms and more than 10ms.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
collock_wait already tells us how long a transaction waits for the
per-collection ordering_lock, but not how long the lock is then held.
comparing hold to wait time can point at either slow in-lock or
contention of a hot pg.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
When there are multiple realms in rgw multi-site, the realm name/token fields overflow the modal content area
Fixes: https://tracker.ceph.com/issues/78051
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
The mon_host tier branched on "mon_host < rhs.mon_host" but returned the
same comparison, so mon_host > rhs.mon_host fell through to key, both
a < b and b < a could hold.
Only Mirror::m_pool_replayers reaches operator< (via std::pair key).
Fixes: https://tracker.ceph.com/issues/78049
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
queue_update_task() took the lock by rvalue reference, so its early
returns never released it. In handle_update_task() that left m_lock
held across the on_finish->complete() calls; during shutdown a
completion can re-enter try_remove_mirror_image_status() and re-lock
m_lock, deadlocking.
Take the unique_lock by value so it unlocks on every return path.
Fixes: https://tracker.ceph.com/issues/78047
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The mirror-mode switch default in handle_get_mirror_info() calls
finish(-EOPNOTSUPP), which deletes this, then breaks and falls
through to the common code, dereferencing the null *m_state_builder and
freeing the request a second time.
Fixes: https://tracker.ceph.com/issues/78043
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Fix 25 criticals, errors, and warnings from Sphinx.
Some of the changes did not change rendering but most fixed unintended
rendering, broken link etc.
Change Mon to Monitor in ceph-secrets.rst.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
angular 19.2.latest is aggressively tearing down the test env which
forces the active rxjs timer to emit one execution as it collapses which
produces a ghost http in the case of request wrapped in the timer
service. so for now flushing all the requests manually to prevent it
Fixes: https://tracker.ceph.com/issues/77729
Signed-off-by: Nizamudeen A <nia@redhat.com>
setup_ca_signed_keys() hardcoded `systemctl restart sshd`, which fails
on Debian/Ubuntu where the SSH server unit is named `ssh.service`
rather than `sshd.service`. Try `ssh` first, fall back to `sshd` if
that fails, grouped so the fallback only fires on restart failure and
doesn't mask a failed config write.
Fixes: https://tracker.ceph.com/issues/77991
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
When copying an object and a compressor/encryption processor is configured
for the destination, the bucket-index accounted_size was set from `ofs`. If
the source is compressed but the destination is not (re-)compressed (e.g.
compression is off, or rgw_compression_type=random rolled no-compression),
then `ofs` is the compressed size, not the logical size.
In multisite this diverges from the peer zone, which recomputes its own
accounted_size via fetch_remote_obj() (already corrected in #63306).
ListObjectVersions then reports different sizes per zone.
Pass along the source's logical size (astate->accounted_size, derived from the
object's own attrs and correct for plain/compressed/encrypted) into
copy_obj_data() and use it when the destination is not compressed. The
rewrite_obj() and transition_obj() callers pass 0 and keep the prior
ofs-based behaviour. This is the copy_obj analog of the fetch_remote_obj
fix in #63306.
No on-disk/encoding change; corrects newly copied objects only (existing
index entries are unaffected) and is safe across a mixed-version upgrade.
Fixes: https://tracker.ceph.com/issues/75716
Fixes: https://tracker.ceph.com/issues/75715
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
This used to be the case before commit d4b977afdc ("qa/distros:
rename centos_latest.yaml to rpm_latest.yaml") and subsequent changes
to enable Rocky 10. When paired with valgrind, python_api_tests* jobs
fail persistently on Ubuntu and Rocky, see [1]. Switch back until that
is resolved (or for as long as centos_9.stream remains in the mix).
[1] https://tracker.ceph.com/issues/74864
Fixes: https://tracker.ceph.com/issues/77982
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
mgr/dashboard: Implement backword support for Create existing nvme-oF gateway groups through UI
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
- Remove dead cyan_store.h includes from pg.cc and shard_services.cc.
- Fix CyanStore::mkfs to write type="cyanstore" instead of "memstore"
- Handle missing device class in OSD::start: stores that return an empty
device class (e.g CyanStore) now fall back to is_rotational=false with
a warning instead of throwing
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Before optimizing the onode lookup paths we should measure the existing
cost if each possible optimization.
The per-node search is already binary search(see search_result_bs_t
binary_search). Possible costs are:
- per-level node traversal
- STAGE_STRING ns/oid memcmp (rbd has long shared oid prefixes)
Adding the following stats to be exposed as seastar-metrics (not logs).
Keep the new sampling per-comparison overhead out unless needed for
measurment runs.
Candidate optimization leads from the numbers:
* cross-node string/prefix dedup (marked as TODOs in stage.h) this could
result in cheaper compares for shared (rbd) prefixes.
* shrink onode_layout_t - marked as onode.h TODO.
stop inlining oi and ss to get more objects per leaf.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Verify that the current version is updated correctly with the
bucket versioning feature.
Fixes: https://tracker.ceph.com/issues/77981
Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>
Fix RGW certificate generation so `zonegroup_hostnames` and wildcard
SANs are passed to `get_certificates()` using the correct keyword
argument. Previously, RGW built the expected SAN list from
`zonegroup_hostnames` and `wildcard_enabled`, but passed it
positionally so it was interpreted as ips list.
Also add a unit test to cover the RGW `zonegroup_hostnames` +
`wildcard_enabled` path and ensure SANs are passed as `custom_sans`.
Fixed other calls to enforce the usage of keywords for arguments just
to avoid similar issues in the future.
Fixes: https://tracker.ceph.com/issues/77980
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Mark the versioning tests in RGW s3-tests.
Fixes: https://tracker.ceph.com/issues/77981
Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>
Drop AT_STATX_DONT_SYNC from stat calls in the crawler thread that
populate SyncEntry objects. The flag could leave a stale file size from
readdir, causing datasync to truncate files to the wrong length.
For RemoteSync, use inode attrs from ceph_readdirplus_r instead
of a redundant ceph_statxat.
Fixes: https://tracker.ceph.com/issues/77393
Signed-off-by: Kotresh HR <khiremat@redhat.com>
do_request_primary_scan() is an async coroutine. Backfill can complete
(and on_pg_clean() can destroy backfill_state) while the coroutine is
suspended in scan_for_backfill_primary(), e.g. waiting on an object
context load. Resuming and delivering PrimaryScanned then dereferences
a null backfill_state.
Discard the scan result if backfill_state is gone; the backfill has
already finished successfully.
Fixes: https://tracker.ceph.com/issues/77623
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
OSD::committed_osd_maps() captures old_map, then for each peer in
old_map->get_all_osds() that transitioned up->down it marks the peer's
cluster address down. The address was fetched from the *new* osdmap via
osdmap->get_cluster_addrs(osd_id), but osd_id comes from old_map. If that
peer was purged (fully removed) in the new map, get_cluster_addrs hits
ceph_assert(exists()) and the OSD aborts on boot.
This makes it impossible to decommission a crimson cluster: purging any
OSD leaves surviving crimson OSDs unable to reboot once they replay an
osdmap epoch range spanning that purge, since they abort on this assert
during boot.
Look the address up in old_map instead: it is guaranteed to contain
osd_id (that is where the id came from) and holds its last-known cluster
address, which is exactly what we want to mark down.
Signed-off-by: Vitaly Goot <vgoot@akamai.com>
The diagram depicts the relationships between the main
Scrubber components and interfaces. The previous version of
the diagram was outdated, and it is now fixed to match the code.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Add a build target that builds standalone RGW in addition to normal RGW,
and add an RPM for it that only installs the necessary bits.
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>
Member access binds tighter than ?:, so the handler attached only to the
recover_object() branch and exceptions from recover_object_with_throttle()
escaped unhandled. Parenthesize the ternary to cover both branches.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
NFS and keepalived are deployed without systemd enable (DISABLED_SERVICES).
When a host exits maintenance, re-enabling ceph-<fsid>.target only starts
enabled units, so these daemons remain stopped.
Fixes: https://tracker.ceph.com/issues/77954
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
"
home/if/ceph.2/src/test/fio/fio_ceph_messenger.cc: In function ‘void
put_ceph_context()’:
/home/if/ceph.2/src/test/fio/fio_ceph_messenger.cc:144:9: error:
expected primary-expression before ‘(’ token
144 | dout(0) << ostr.str() << dendl;
| ^
In file included from /home/if/ceph.2/src/include/Context.h:20,
from /home/if/ceph.2/src/msg/Message.h:32,
from /home/if/ceph.2/src/msg/Messenger.h:29,
from
/home/if/ceph.2/src/test/fio/fio_ceph_messenger.cc:12:
/home/if/ceph.2/src/common/dout.h:190:5: error: ‘_dout_cct’ was not
declared in this scope
190 | _dout_cct->_log->submit_entry(std::move(_dout_e));
\
| ^~~~~~~~~
/home/if/ceph.2/src/common/dout.h:216:15: note: in expansion of macro
‘dendl_impl’
216 | #define dendl dendl_impl
....
"
Fixes: https://tracker.ceph.com/issues/77953
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
Only add deps when the user has set a non-default value. In
choose_next_action(), ignore False/None-valued entries in the symmetric
diff so daemons upgrading from the old behavior are not reconfigured
solely because default deps disappeared
Fixes: https://tracker.ceph.com/issues/77901
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
mgr/cephadm: Updated NFS default protocol to v4 and v3 is enabled only when enable_nfsv3 is set in the spec
Reviewed-by: Redouane Kachach <rkachach@ibm.com>
Reviewed-by: Adam King <adking@redhat.com>
9abf2388f2 rewrote "sh -c cat" as SHELL "-c cat", which concatenates
to "/bin/sh-c cat", so the subshell tries to execute a non-existent
"/bin/sh-c". The test still passed because the SIGTERM sent by kill()
usually wins the race against the shell startup; under CI load the
shell occasionally prints
/bin/sh: line 1: /bin/sh-c: No such file or directory
into the stderr pipe first, ASSERT_TRUE(buf.empty()) fails, and the
skipped join() turns this into a SIGABRT in ~SubProcess().
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Clearing a compilation warning:
The ternary operator created a temporary std::string to unify
const std::string& and const char* branches, leaving the
string_view pointing to a destroyed object. Replace with
find() + if/else to assign directly from stable storage.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
BackfillState::Waiting requires an ObjectPushed event to transition
back to Enqueuing, but drop (remove) operations are fire-and-forget
and never generate ObjectPushed. When a backfill round enqueues only
drops and no pushes, the FSM permanently stalls in the Waiting state,
holding the backfill reservation and blocking all other PGs from
backfilling to that OSD.
Fix by adding a pending-pushes counter to ProgressTracker to
distinguish "only drops remain" from "pushes still in-flight". When
all pushes are complete and the namespace is exhausted, drain remaining
drop entries and advance last_backfill to MAX so that
OP_BACKFILL_FINISH triggers the normal completion path via
RequestDone.
The classic OSD avoids this by design: drops are never tracked as
in-flight operations in recover_backfill() and are drained inline.
Fixes: https://tracker.ceph.com/issues/77936
Assisted-by: Claude <claude.ai>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
CacheSession::send() deleted the reply in the async_write completion
handler. On teardown asio drops the pending handler without running it,
leaking the reply. Delete it right after encoding into bl instead; bl
owns the payload, so the write is unaffected.
Fixes: https://tracker.ceph.com/issues/77552
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
BlueStore's PMEMDevice backend targets byte-addressable persistent
memory exposed as a DAX device, in practice Intel Optane DC persistent
memory. Intel Optane has been discontinued since 2021 and there is no
mainstream replacement, so the backend has effectively no hardware left
to run on. It was always experimental and has seen no functional change
since the DML/DSA offload landed in 2022; every commit to it since has
been a formatting sweep.
Keeping it builds in a chain of dependencies that nothing else needs:
libpmem, libndctl and libdaxctl, plus dml for the DSA offload that only
works on a Sapphire Rapids CPU. Remove the backend and everything tied
to it.
RBD persistent write-back cache (WITH_RBD_RWL) still uses PMDK's
libpmemobj, so the pmdk submodule, WITH_SYSTEM_PMDK, Buildpmdk and
Findpmdk stay. WITH_SYSTEM_PMDK now depends on WITH_RBD_RWL alone, and
Buildpmdk no longer builds ndctl support.
- remove the WITH_BLUESTORE_PMEM option, HAVE_BLUESTORE_PMEM and
HAVE_LIBDML
- delete src/blk/pmem/PMEMDevice.{cc,h} and the pmem block_device_t
- drop the pmem value from the bdev_type option
- remove the --bluestore-pmem vstart flag
- drop the libpmem, libndctl and libdaxctl build deps from debian and
rpm, keeping libpmemobj for RWL
- delete the Finddml, Findndctl and Finddaxctl cmake modules
- remove the DSA/DML and ndctl documentation
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
When AlienStore's thread pool is saturated (e.g. during PG splitting),
pool_statfs() calls inside OSD::get_stats() can get stuck behind
hundreds of queued transactions in the free_slots semaphore. Because
Client::report() dispatched get_stats() unconditionally every 5 seconds,
hundreds of stuck pool_statfs() calls accumulated, further exhausting
the semaphore and making recovery impossible.
Fix by tracking whether a get_stats() call is already in flight and
skipping new dispatches until the current one completes. Emit a
cluster-log warning every ~60 seconds when stats reporting is stuck.
This limits the damage from thread pool starvation to a single consumed
free_slot instead of an unbounded pile-up.
Fixes: https://tracker.ceph.com/issues/77930
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
the mon, osd and mds register a "heap" asok command that forwards to
ceph_heap_profiler_handle_command, but the mgr never did, so
`ceph daemon mgr.<id> heap stats` returns "no valid command found".
that leaves no way to tcmalloc-profile a leaking mgr through its admin
socket, which is what chasing mgr memory growth needs.
register the same command on MgrStandby's admin socket. its hook lives
for the whole process, so it works on both the standby and the active
mgr. heap output goes to the bufferlist, as on the other daemons, while
status keeps using the formatter.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
crimson/common: use the value_map passed to dump_metric_value_map()
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
get_dashboards_data() keeps every query in a single map keyed by
"<panel title>-<legendFormat>", but that id is not unique: several
dashboards have a panel titled "IOPS", "OSDs" or "Throughput". The
dashboard read last overwrites the earlier entry, so one query shadows
another and never gets tested. glob() walks the files in filesystem
order, so which query survives, and whether the test passes, depends on
readdir.
run-tox-promql-query-test hit this in "Test IOPS Read"
(ceph-cluster.feature): the scenario feeds ceph_osd_op_r but the id
resolved to a CephFS pool panel.
FAILED:
expr: "sum(rate(ceph_pool_rd{cluster=~"mycluster|", pool_id=~"UNSET VARIABLE"}[1m]))", time: 1m,
exp:"{} 2.5E+01"
got:"nil"
HOOK-ERROR in after_scenario: AssertionError:
pool_id is "UNSET VARIABLE" because the scenario does not set $mdatapool,
and no ceph_pool_rd series were fed, so the query returns nil. It only
fails when readdir returns cephfsdashboard after ceph-cluster-advanced,
so the earlier change that walked nested rows and made the winner
deterministic did not fix it; it fixed which query wins, not that the
wrong one can win.
Key the queries per dashboard (data['queries'][<dashboard>][<id>]) and
have each .feature name its dashboard in a Background step. The lookup is
confined to that dashboard, so a title/legend used elsewhere no longer
shadows it. A duplicate within one dashboard is still an error, unless it
is in a collapsed row.
ceph-cluster.feature covers ceph-cluster-advanced; the rest map to one
dashboard each.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Plus refactors txc dumping code to permit Unit Testing for it.
Here is the new output sample:
Short form:
DF:D*3,DF:O,MF:A,PF:O*2,PF:P,...*2
Verbose form:
PutCF(prefix = O, key = 'key', val len = 5)
PutCF(prefix = P, key = 'key2', val len = 6)
PutCF(prefix = O, key = 'key3', val len = 6)
DeleteCF(prefix = O, key = 'A1')
DeleteCF(prefix = D, key = 'A1')
DeleteCF(prefix = D, key = 'A2')
DeleteCF(prefix = D, key = 'A3')
MergeCF(prefix = A, key = 'A5', val len = 6)
<...>*2
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
mgr/smb: Allow pool creation for empty RGW buckets
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
bench_async_write_worker destroyed AsyncIOContext while the finisher
still ran async_io_callback after setting completed. Track in-flight
callbacks per worker and drain before tearing down the io_queue.
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
CnLog::_flush() calls alien::submit_to(...).wait(), which blocks the
log flusher thread until the seastar reactor finishes outputting the
log entries. When the reactor writes to stderr and the pipe buffer is
full (e.g. two OSDs on the same host both dumping at debug level 20),
the reactor itself blocks on the write() syscall. Meanwhile BlueStore
threads filling the log buffer past m_max_new block in submit_entry()
waiting for the flusher - which is waiting for the reactor - which is
blocked on I/O. The kv thread cannot fire commit callbacks while it is
blocked on logging, so store transactions (including osdmap persistence)
hang indefinitely.
Fix by moving the entries into the lambda capture so _flush() can
return immediately without waiting.
This is a corrected version of the approach in commit 511af83e27
("crimson/os/alienstore/alien_log: _flush concurrently", PR #55262),
which was reverted in 4d52d1d22b because its lambda captured the
EntryVector by reference - the caller would clear/reuse the vector
before the lambda finished, causing a "pure virtual method called"
crash (https://tracker.ceph.com/issues/64140). The fix here avoids
that by swapping the entries out of the caller's vector and moving
them into the lambda's capture, so the lambda owns them for their
full lifetime.
Fixes: https://tracker.ceph.com/issues/77826
Fixes (failure of the 1st attempt): https://tracker.ceph.com/issues/64140
Assisted-by: Claude <claude.ai>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
This serves as a reminder for maintainers to verify whether capability
changes in the RGW service should also be applied to SMB and NFS
services to maintain consistency across services that use librgw.
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
NodeProxySpec.__init__() only accepts service_type and placement,
but ServiceSpec.from_json() passes all stored fields (ssl,
certificate_source, unmanaged, ...)
This breaks 'ceph orch ls' after the mgr reloads a node-proxy spec.
This commit forwards the standard ServiceSpec parameters to the
parent.
Fixes: https://tracker.ceph.com/issues/77874
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
The clone API endpoint POST /api/block/image/{image_spec}/snap/{snapshot_name}/clone
only accepted a snapshot name, which uses rbd_clone3 (clone by name). This only works
for snapshots in the user namespace.
Group snapshots (with .group prefix, in RBD_SNAP_NAMESPACE_TYPE_GROUP namespace)
require rbd_clone4 (clone by snap ID). The rbd Python binding already supports this:
passing an int for p_snapshot triggers rbd_clone4.
Add an optional clone_by_snap_id boolean flag to the clone endpoint. When set, the
path parameter is treated as a numeric snap ID and passed directly as an int to
rbd.RBD().clone(). Clone format 2 is enforced because this flag is intended for
non-user namespace snapshots which cannot be protected (a prerequisite for clone
format 1).
Usage:
POST /api/block/image/{image_spec}/snap/{snap_id}/clone
{"clone_by_snap_id": true, ...}
Fixes: https://tracker.ceph.com/issues/77875
Signed-off-by: Imran Imtiaz <imran.imtiaz@uk.ibm.com>
Assisted-by: IBM Bob
doc: fix some small errors and oversights for new rgw shares
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
When an RGW bucket is empty, SMB clients cannot create directories,
files, or upload data because the default.rgw.buckets.data pool does
not yet exist. The previous CephX capabilities ('mon allow r') did not
permit pool creation during the first write operation.
The fix changes the mon capability from 'allow r' to 'allow *' to
enable pool creation on the first write to an empty RGW bucket. Once
the pool exists, subsequent operations succeed normally.
Fixes: https://tracker.ceph.com/issues/77764
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
crc32c_zbc's fold-by-four loop used gp (x3) and tp (x4) as scratch for
two folding temporaries. Save/restore around the loop is not enough:
they are ABI-reserved, so a signal delivered mid-loop runs its handler
with a corrupted tp, and the first TLS access there faults.
On riscv64 this reliably crashes the crimson/seastore unittests
unittest-transaction-manager and unittest-omap-manager, where seastar's
stall-detector timer fires often: the process dies with SIGSEGV in the
linker's TLS path with garbage in gp/tp.
Fixes: https://tracker.ceph.com/issues/77904
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The mgr volumes module handles fs volume ls/create, not the monitor, so
without a mgr (CEPH_NUM_MGR=0) the "wait for volume module to load" loop
spun forever. Open-coding fs new unconditionally, my first attempt,
changed pool creation for every vstart user and dropped the wait, so
cephfs tests calling `fs volume ...` right after vstart.sh returns could
race a module still mid-load.
create_fs_volume() branches on CEPH_NUM_MGR instead. With a mgr it still
calls fs volume ls/create. Without one it runs the plain mon commands fs
volume create wraps: osd pool create cephfs.<name>.{meta,data}, then fs
new <name> <meta> <data>. The wait stays at its one call site above the
per-filesystem loop, so it runs once regardless of CEPH_NUM_FS, not once
per filesystem.
Verified against a local vstart: mgr present, fs volume ls polls once
then creates CEPH_NUM_FS=2 filesystems with no repeated poll;
CEPH_NUM_MGR=0, no poll, fs new creates 'a' directly, HEALTH_OK.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
dispatch_context_transaction submits a real empty transaction to
AlienStore even when the peering context has no store ops. This
goes through the AlienCollection lock and thread pool slot machinery.
Under load from broadcast_map_to_pgs (many PGAdvanceMap operations
running concurrently with recovery), some of these empty transactions
get stuck inside AlienStore's pipeline and never reach BlueStore,
causing PGAdvanceMap to hang at complete_rctx indefinitely.
Fix by skipping the store round-trip for empty transactions. The
on_commit callbacks are still collected and fired, preserving the
fix from b0f5e1086a ("crimson/osd/pg: also trigger callbacks for
empty peering transactions"). The store flush that commit added was
incidental to the callback fix and is unnecessary here: the peering
pipeline's exclusive process stage already guarantees that prior
peering transactions on the same PG have completed.
Assisted-by: Claude <claude.ai>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Adding documentations and pending release notes for the mirroring
checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Implementation of helper functions and data structures for the
snapshot based mirroring checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
Signed-off-by: Afreen Misbah <afreen@ibm.com>
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Fixes https://tracker.ceph.com/issues/77723
Signed-off-by: Afreen Misbah <afreen@ibm.com>
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com>
Assisted-by: Claude
Signed-off-by: Afreen Misbah <afreen@ibm.com>
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com>
Assisted-by: Claude
Signed-off-by: Afreen Misbah <afreen@ibm.com>
Reset self.daemon_name for matched gateway, to ensure
correct daemon_name is set when server_address param
is passed to the NVMeoFClient object.
Right now, NVMeoFClient().daemon_name shows same daemon_name
for all gateways within the group.
Also add unit test for NVMeoFClient in
src/pybind/mgr/dashboard/tests/test_nvmeof_client.py
Fixes: https://tracker.ceph.com/issues/77469
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>
For each key in spec.config, validate the option name and run
config rm on the service's config section. Call this from
remove_service() before spec_store.rm() schedules daemon removal.
Fixes: https://tracker.ceph.com/issues/77891
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
mgr-dashboard-frontend-unittests has been failing on main since 2026-07-01,
unrelated to whatever PR triggers it. Bisected to cfe12305fb, the merge
commit for PR #68180.
6182e67ea4 ("mgr/dashboard: NVMe Onboarding Setup Cards") moved gateway
group selection out of NvmeofSubsystemsComponent and
NvmeofNamespacesListComponent into a new NvmeofGatewayGroupFilterComponent,
dropping gwGroups, updateGroupSelectionState, handleGatewayGroupsError,
onGroupClear, and getSubsystems from both. Clean on its own.
PR #68180 forked from main in April, before that refactor existed, and
added three new tests against the old API. Also clean on its own, and never
rebased. Both PRs only touch lines the other doesn't, so when #68180 merged
34 minutes after 6182e67ea4, git's 3-way merge combined them without a
conflict. The merge is the only place the bug exists: new tests calling
methods the sibling parent had already deleted. TypeError at runtime,
TS2339/TS2551 from tsc, every run since. CI never caught it: GitHub was
showing #68180's make check as green from a run a full day older than
6182e67ea4.
nvmeof-namespaces-list.component.spec.ts: the dedup test is real, current
logic, it just called the old listNamespaces() instead of fetchData().
Fixed the call. The other two duplicate NvmeofGatewayGroupFilterComponent's
own spec; removed.
nvmeof-subsystems.component.spec.ts: same story, all three duplicate the
filter component's spec; removed.
Both removed error tests also checked preventDefault() on the error object,
which had no coverage elsewhere. Added that to
NvmeofGatewayGroupFilterComponent's spec, and hit a second bug while writing
it: the file's throwError(() => err) is the RxJS 7 factory overload, this
project is on RxJS 6.6.3, so it was emitting the arrow function itself as
the error. The existing error test never noticed since it only checks side
effects, not the error's identity. Used throwError(err) instead.
Verified: jest passes on all three spec files (was 6 failing), tsc -p
tsconfig.spec.json --noEmit is clean (was 17 errors), full block/nvmeof
sweep is 29/29 suites, 216/216 tests.
Fixes: cfe12305fb
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
WebCache declared its cache lock as a raw std::shared_mutex, bypassing the
project's ceph::shared_mutex alias that every other rwlock user goes through
(OSD::map_lock, TrackedOp::lock, ...). Align it with that convention: use
ceph::make_shared_mutex() to construct the member, and drop the explicit
<std::shared_mutex> template argument from each std::shared_lock/lock_guard
so they deduce whichever type ceph::shared_mutex resolves to.
This is idiom alignment only. The preceding commit fixes the Windows CI
timeout; this one does not touch it. ceph::shared_mutex only differs from
std::shared_mutex in debug builds, where it becomes ceph::shared_mutex_debug
for lockdep tracking. On the mingw-llvm release build that times out
unittest_web_cache, it still resolves straight to libc++'s std::shared_mutex.
The O(N^2) writer-wakeup behavior under heavy exclusive-lock contention is
unchanged.
Verified locally: unittest_web_cache builds and passes, exercising the debug
ceph::shared_mutex_debug path.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
WebCacheConcurrencyTest sized its thread pool as hardware_concurrency() * 100,
which is 3200 on the 32-vCPU CI hosts. Linux runs that in a few seconds, but
the Windows (mingw-llvm/libc++) build times out unittest_web_cache at the
1800s per-test limit on every PR.
BasicAddSame reuses one key, so after the first insert every add() is a
shared-lock hit: 3200 threads log hit=3200000 miss=1 and finish in ~4s.
BasicAddUnique inserts a distinct key each iteration, so every add() misses
and takes the exclusive lock. libc++'s std::shared_mutex wakes all waiting
writers on every unlock, so N contending writers cost O(N^2) wakeups. At
N=3200 that alone overruns the 1800s budget, which is where the failing run
hangs. StampedeSyncCallOnce and StampedeMutex are exclusive-lock bound the
same way.
The tests only need enough threads to race; 4x oversubscription is enough.
Cap the count via a shared helper.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
python3_pkgversion picked the native python3 package name on RHEL7, where
python3 was python34/python36 before 7.6 (cea9d18c). RHEL7 and RHEL8 are
gone, so on every supported distro (el9, el10, fedora, suse, openEuler) it
is always 3.
Replace python%{python3_pkgversion} with python3 and drop the fallback
default. No supported distro overrides it to build against a non-default
python flavor, so that capability goes too.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Many Python deps were split across %if suse / %if rhel branches only
because openSUSE spells them CamelCase (PrettyTable, PyYAML, Sphinx,
CherryPy, Routes, Jinja2) and fedora/rhel/openEuler lowercase. The
python3dist() provides resolve the same on el9, el10, openEuler and
openSUSE, so those splits collapse to single lines.
Spell the deps python%{python3_pkgversion}dist(), matching the spec's
existing python%{python3_pkgversion}-Foo convention. Version constraints
and feature guards are preserved.
The dist name is the PyPI project name, not always the rpm name minus
python3-: python3-dateutil provides python3dist(python-dateutil), and
python3-saml provides python3dist(python3-saml).
Left as package names: the bare interpreter; python3-devel and numpy-devel,
which have no python3dist() provide; the build toolchain (setuptools,
Cython, pip, wheel); and the ceph-internal python3-* subpackages, which are
versioned = release deps.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Library BuildRequires used per-distro -devel package names, some split
across %if suse / %if rhel branches only because the names differ. The
pkgconfig() virtual provides resolve identically on every rpm distro; the
spec already relies on them for fuse3, systemd and udev.
Switch every library dependency that ships a .pc to pkgconfig(), preserving
version constraints and feature guards. This collapses the suse-vs-rhel
naming splits (nss, keyutils, openssl, ldap, ibverbs, rdmacm, lz4, thrift,
libcryptopp) and the dedicated split blocks (nlohmann_json, lttng-ust,
babeltrace, expat) into single lines, and folds the duplicated xmlsec1 devel
packages together.
Left as package names: libaio and xfsprogs ship no reliable .pc; gperftools
carries version and tcmalloc-variant logic; numa is unconditional on rhel
but crimson-only on suse; and the xmlsec1 runtime backend plugins are
runtime test deps, not build-time pkgconfig() deps.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
RHEL 8 and CentOS 8 left the supported test matrix in 2023 (449db416,
7a1dce1e). RHEL 8 has also been unbuildable since CMakeLists.txt began
requiring Python >= 3.9: it ships python 3.6 by default, so cmake refuses
to configure. The spec still carried the el8 conditionals, so assume
rhel >= 9 and drop them:
- remove branches reachable only on rhel <= 8: the gperftools 2.6.1
BuildRequires, the tracker-36508 tcmalloc-libs requirement, and the
dataclasses backport, which also targeted that dead python 3.6
- collapse "rhel >= 8" and "rhel >= 9" to plain "rhel"; with the floor at
9 this is a no-op for rhel 9/10, fedora, suse and openEuler
- keep the rhel 9-vs-10 checks (== 9, < 10, >= 10)
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Currently client.admin is passed for client_name and that doesn't
exercise client_name handling much as client.admin is the default.
When deploying multiple clusters the ceph task distributes keyrings
with only the key for the initial monitor and client.admin key. Other
keys (e.g. client.0) are present only on their respective clusters, so
client.0's key for cluster2 needs to be obtained on cluster1 explicitly
with "ceph auth get".
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
migration:
-take secret_key from spec
-take mon_host from the spec
-ignore source keyring, source ceph.conf - bypass them
-added validations of the new way schema
-get key from the KV DB if key in spec is actually key-ref
Fixes: https://tracker.ceph.com/issues/68177
Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
'export' sets file_fd to STDOUT_FILENO for '-', but cleanup guarded
close() on STDIN_FILENO (always true), closing stdout. Guard on
STDOUT_FILENO instead, matching the 'import' path.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
`if (!stdout)` tested the libc FILE* (never null) instead of the
use_stdout flag, so the ofstream was never deleted when writing to
a file, leaking it and leaving its buffer unflushed. Test use_stdout.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
do_put()/do_append() set fd to STDIN_FILENO for '-', but cleanup
guarded close() on STDOUT_FILENO (always true), closing stdin.
Guard on STDIN_FILENO instead.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
In TestFSCryptVolumes, make sure to use mount class to
compare directory trees and to read/write files.
Fixes: https://tracker.ceph.com/issues/74110
Signed-off-by: Christopher Hoffman <choffman@redhat.com>
Add --async-write-fsync and --shared-file benchmark options.
--async-write-fsync sets ceph_ll_io_info.fsync on each async write to model
Ganesha-style write+commit and skips the post-phase ceph_ll_fsync when enabled.
--shared-file directs all worker threads at one file to stress client_lock
contention on a single inode. Both options are recorded in JSON/text output.
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Added filesystem stats reporting to cephfs-tool.cc.
Helper functions
• fetch_statvfs() — uses ceph_ll_statfs on the root inode when --async-io
is set, otherwise ceph_statfs on mount_root
• report_fs_stats() — prints and optionally JSON-encodes:
• Total / free blocks
• File count
When it runs
• Before the iteration loop (after the bench directory is created)
• After all iterations finish (before the final report and optional cleanup)
Console output example
Filesystem statistics (before iterations):
Total blocks: ...
Free blocks: ...
Files: ...
JSON output (when --json is used) adds a filesystem_stats section with before
and after objects containing the same fields.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Set pthread names on all benchmark worker threads (wr-worker-N, rd-worker-N)
via ceph_pthread_setname so they are visible in top, perf, and lockstat
output during profiling.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --async-io to use ceph_ll_nonblocking_readv_writev for read and write
workers instead of synchronous ceph_read/ceph_write. Add --queue-depth to
limit outstanding async I/Os per thread via a counting semaphore and
pre-allocated AsyncIOContext pool. Async write uses ceph_ll_create; async
read uses ceph_ll_walk + ceph_ll_open and defers inode release until all
async ops including readahead complete. Reject --async-io with
--per-thread-mount since async workers require a shared mount.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
When built with CEPH_LOCKSTAT, add --lockstat-dump and --lockstat-threshold
to collect lock contention data during benchmark phases. Start collection
at benchmark begin, reset counters before each read/write phase, and dump
per-iteration/per-phase JSON files (e.g. base_iter1_write.json). Requires
a lockstat-instrumented libcephfs build.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --msgr-workers to override ms_async_op_threads before mount (1-24;
0 keeps the ceph.conf default). Controls async messenger I/O threads
independently of benchmark worker thread count. Validate the range at
startup and include the resolved value in console and JSON output.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --progress and --progress-interval to display live bandwidth, IOPS,
and ETA during read/write phases. A background progress_reporter thread
aggregates per-thread stats every 100ms and prints a single updating
status line. Progress percentage is based on elapsed time when --duration
is set, otherwise on files completed. Make ThreadStats fields atomic so
workers and the reporter can access them concurrently.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --client-oc and --client-oc-size to override libcephfs object cacher
settings before ceph_init. Enables benchmarking with the object cacher
enabled or disabled and with a configurable cache size. Resolved values
are shown in console and JSON output.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --perf-dump to write libcephfs client performance counters to a file
at the end of the benchmark. Uses ceph_get_perf_counters() while the
mount is still active so counters reflect the full benchmark run.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --duration to cap each read and write phase at N seconds instead of
running until all files are processed. Workers loop until the time limit
expires, cycling through their per-thread file set in duration mode.
Pass phase start time into worker threads for consistent cross-thread
duration checks. Only count each unique file once in stats even when
filenames are reused across cycles.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
Add --json to write structured benchmark results (configuration,
per-iteration read/write metrics, and cross-iteration summary statistics)
to a file using JSONFormatter. Extend print_statistics() to optionally
mirror mean, stddev, min, and max into the JSON document. Improve
parse_size() to accept both SI (1000-based, e.g. "4MB") and binary
(1024-based, e.g. "4MiB") size suffixes.
Fixes: https://tracker.ceph.com/issues/76183
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
validate_input() rejected the valid TOKEN_CODE_SIZE length and accepted
all others. Compare with != so only a wrong length is rejected.
Fixes: https://tracker.ceph.com/issues/77872
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
If-None-Match: * must fail when the object exists, but the posix writer
did the opposite, disabling overwrite protection. Return
PRECONDITION_FAILED when the object exists.
Fixes: https://tracker.ceph.com/issues/77871
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
A versioned delete carries a non-empty version-id, but the MFA guard
flagged the empty case: versioned deletes bypassed MFA while plain
deletes wrongly required it. Test !instance.empty().
Fixes: https://tracker.ceph.com/issues/77869
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The max_size and max_objects in the Admin-OPs REST interface were accidentally set to int32_t
This allowed to set only up to 2GB of quota for accounts
Signed-off-by: Theofilos Mouratidis <mtheofilos@gmail.com>
run_command() captured a reference to the just-emplaced mon_commands
element into the send_message() continuation. mon_commands is a
std::vector, so a concurrent run_command() could reallocate it and
invalidate the reference before the continuation ran, dereferencing
freed memory.
No caller issues concurrent commands today, but the pattern is unsafe.
Take the result future before issuing the message, then coroutinize
the method so it is awaited directly instead of being captured into a
continuation lambda.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
handle_client_symlink (-ENAMETOOLONG), reclaim_session (-EPERM) and
handle_client_readdir_snapdiff (-EINVAL) each sent an error reply on a
failure path but fell through, replying to the same MDRequest a second
time. reclaim_session additionally went on to bind reclaiming_from,
bypassing the auth_name check. Return right after sending the reply.
Fixes: https://tracker.ceph.com/issues/77862
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
mgr/dashboard: Add option should be enabled even if allow all host access is enabled in nvme/tcp
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Abhishek Desai <abhishek.desai1@ibm.com>
Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
Config rgw_sts_key expects a 16-character hexadecimal key. The option
long_desc has an example with openssl rand -hex 16 which generates
16 hexadecimal bytes that consist of 32 characters rendered in ASCII.
Modify the example openssl command to output 8 bytes that matches the
required length of 16 characters.
Remove a space after a slash.
Raised in the doc bugs pad.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
The handler passed the stale locate result (0) while returning -EINVAL,
so callers saw success on a bad_cmd_get failure.
Only reachable via the "scrubdebug" asok command (the sole user of
asok_route_to_pg) when a parameter fails type parsing.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
* refs/pull/66558/head:
qa/cephfs: minor fix in comment
qa/cephfs: give more time to tests in test_clone_stats.py
qa/cephfs: increase number of files to cloned in test_clone_stats.py
volumes/stats_util: improve log messages
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Hold a function-scoped DentryRef in Client::unlink() to keep the dentry
alive through the entire teardown sequence.
Without the O_DIRECTORY dentry pin (PR #60909, merged after v19.2.1),
opening a directory does not call _ll_get(), so the dentry stays at
ref=1 while inode->dir is still active. When trim_cache() evicts this
dentry, Dentry::unlink() calls put() for the dir pin, ref drops to 0,
and the dentry is freed while Client::unlink() still needs it for
detach/lru_remove.
A function-scoped DentryRef guarantees the dentry survives until after
it has been properly removed from the dir and LRU, regardless of pin
state.
Signed-off-by: Igor Golikov <igolikov@ibm.com>
Fixes: https://tracker.ceph.com/issues/74625
Add ceph_assert(dn->ref > 0) after Dentry::unlink() in Client::unlink()
to catch the case where internal put() calls drop the dentry reference
to zero before we proceed to detach/lru_remove.
This turns a silent use-after-free into a clean assertion failure with
a full stack trace, making it possible to catch the exact conditions
in a test environment.
Signed-off-by: Igor Golikov <igolikov@ibm.com>
Fixes: https://tracker.ceph.com/issues/74625
Seastar_SANITIZE defaults to "DEFAULT", which turns on ASan and UBSan for
Debug and Sanitize builds regardless of ceph's WITH_ASAN. A Debug build
with WITH_ASAN=OFF then instruments seastar while ceph links tcmalloc, and
the seastar tests crash before main() because ASan calls the allocator via
dlsym before it is initialized.
Force Seastar_SANITIZE to follow WITH_ASAN, next to the other forced
Seastar_* cache variables, so seastar is sanitized only when ceph is. This
keeps the fix on the ceph side instead of carrying a seastar fork patch.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Rename transate_transaction() to translate_transaction(). The function
translates an os::Transaction into a PGTransaction.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
transate_transaction() handled OP_OMAP_RMKEYRANGE in the same case
block as OP_OMAP_SETHEADER, decoding a single bufferlist and calling
omap_setheader(). But OP_OMAP_RMKEYRANGE encodes first/last as two
strings, so decode_bl() misreads the stream and the wrong op is
applied.
Give OP_OMAP_RMKEYRANGE its own case: decode the two strings and pass
them to the two-string PGTransaction::omap_rmkeyrange() overload, which
encodes them into the string[2] bufferlist internally.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
ECCrimsonOp::transate_transaction() lacked a break after OP_OMAP_CLEAR
and OP_CLONERANGE2, so both cases fell through to the next one.
OP_OMAP_CLEAR fell into OP_OMAP_SETKEYS, which decodes an attrset from
the transaction iterator that OP_OMAP_CLEAR never encoded, corrupting
the decode stream for every following op.
OP_CLONERANGE2 fell into OP_CLONE; clone_range() had already marked the
destination op non-empty, so the following clone() trips
ceph_assert(op.is_none() || op.is_delete()).
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The 'min' section uses sitem.max, which looks like a copy-paste slip;
if so, the computed min values would never be emitted.
Fixes: https://tracker.ceph.com/issues/77851
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The loop iterated over addr but always passed the list head *ifa
to matches_with_net, so it only ever tested the first interface
while returning the iterated interface's name. Match against
*addr.
Fixes: https://tracker.ceph.com/issues/77847
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
These updates will allow us to update a single page rather than two. The detailed member lists and categories have been removed. I've added a redirect for visitors to this page to https://ceph.io/en/foundation/members/
Signed-off-by: Anthony M <amiddleton@linuxfoundation.org>
This covers both removing them actively from now on when removing
daemons from the HostCache and also not loading entries like this
back in next time they are loaded when a user upgrades to a
version with this patch
Fixes: https://tracker.ceph.com/issues/77839
Signed-off-by: Adam King <adking@redhat.com>
When ImageState::open() fails with an error other than -ENOENT, it
asynchronously deletes the ImageCtx before returning. The trash purge
loop only skipped the -ENOENT case and fell through on every other
error, dereferencing the already-freed ictx in diff_iterate() and
state->close(). Add the missing continue so a failed open is skipped.
The fall-through dates back to 504f4e78, which split `if (r < 0)
continue` to log non-ENOENT errors but dropped the continue.
Fixes: https://tracker.ceph.com/issues/77836
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Guard the KMS key fetch so we only touch perfcounter when it exists.
The fetch latency PerfGuard now sits in an optional and the error
counters are wrapped in an if, so both are skipped when perfcounter
is null.
This keeps key reconstitution working from radosgw-admin (like lc
process), where the perfcounter global isn't set up the way it is
inside radosgw.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Do not carry RGW_ATTR_SHARE_MANIFEST or RGW_ATTR_BLAKE3 when object data
or manifest ownership is rewritten through admin rewrite, remote fetch,
CopyObject, or lifecycle transition.
Both attributes describe RGW dedup state for an existing RADOS layout:
the shared-manifest marker and the blake3 digest of the object's on-disk
bytes. Once a path writes a new layout or creates a new destination head
with its own refcount tag, preserving them leaves stale dedup on the
destination.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Without a facet marker, teuthology concatenated all three yamls into
one job, running plain, CBC, and GCM workunits back-to-back under
whichever rgw override won.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Extract the decrypt/decompress/recompress/re-encrypt filter pipeline
into a shared RGWRecompressDPF base class. RGWTransitionDPF is added
for lifecycle transitions and RGWCopyObjDPF is rewritten on top of
the same base, both as thin subclasses providing key retrieval and
compression config.
Add rgw_prepare_decrypt_object() for decrypting from stored attrs
without an HTTP request context (handles SSE-KMS/-GCM, SSE-S3/-GCM,
RGW-AUTO/-GCM), and rgw_prepare_reencrypt_object() which regenerates
the GCM salt before re-deriving the per-object key — avoids AEAD
nonce reuse when LC re-encrypts modified plaintext.
Includes a QA workunit that drives transitions through three
storage classes with different compression configs and verifies
data integrity, including with SSE-KMS encrypted objects.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
* refs/pull/69324/head:
libcephfs: increment extra version to indicate new flock functions
libcephfs: add public api for ceph_ll_flock
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
When scaling ingress down with a ranked NFS backend that still has more
daemons than its placement target, remove ingress from hosts without NFS
first and skip removing co-located ingress on NFS hosts until the next
iteration after NFS scales down.
Fixes: https://tracker.ceph.com/issues/76304
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
If the ingress service is configured with a placement count lower than the number of hosts,
then when one node goes down, Keepalived is configured on the remaining hosts.
When the failed host comes back up, if it was previously configured as the master,
the VIP may become active on two hosts. To prevent this, Keepalived is disabled by default
and started within the service loop.
Fixes: https://tracker.ceph.com/issues/76304
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
When colocation of NFS daemons is allowed, the existing rank calculation incorrectly derives the
rank based solely on the number of hosts. This leads to inaccurate rank assignment because the logic
does not account for the full set of potential candidates, especially in scenarios where multiple
daemons can reside on the same host.
Fixes: https://tracker.ceph.com/issues/73442
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
If the removal of an extra NFS daemon fails for any reason, that daemon should not be started, as the NFS daemon with the highest rank generation is already running.
Fixes: https://tracker.ceph.com/issues/73442
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
recovery.
The recovery procedure performs that itself by design.
Fixes: https://tracker.ceph.com/issues/75852
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
rotational/non-rotational OSD behavior for DB/WAL devices.
Prior to this commit this parameter determined what device class
settings to apply for BlueStore only. OSD had partially applied
actual journal/db/main device class for some operations, e.g. benchmark
on init or allocmap persistence, irrespective to this setting value.
This commit allows 'bluestore_debug_enforce_settings" to override the
actual device class completely.
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
mgr/cephadm: bump Python version path mapped in kcli dev env
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Kobi Ginon <kginon@redhat.com>
Skip influx selftests when the Python influxdb client is not installed.
Remove the ineffective testhost workaround and ignore the expected
MGR_INFLUX_NO_SERVER warning when the module runs without a server.
Fixes: https://tracker.ceph.com/issues/77250
Signed-off-by: Kobi Ginon <kginon@redhat.com>
ceph daemon command can hangs indefinitely when the target
daemon's admin socket is unresponsive. do_sockio() creates socket with
no timeout, so sock.recv(4) blocks forever if the daemon is slow to respond,
where AdminSocket::init() creates the socket file before store->mount()
returns).
The AdminSocketClient already uses a 10-second SO_RCVTIMEO / SO_SNDTIMEO.
Align the Python client by calling sock.settimeout(timeout) before connecting.
The default is 10s to match the C++ side; callers can pass timeout=None
to restore the previous blocking behaviour.
Fixes: https://tracker.ceph.com/issues/77743
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
During backfill recovery on Crimson OSD replica, Bluestore
aborts with _txc_add_transaction error ENOENT on operation
due to OP_TOUCH_TEMP. OP_TOUCH_TEMP is guarded by #ifdef WITH_CRIMSON
but WITH_CRIMSON is not defined when BlueStore is compiled.
OP_TOUCH_TEMP is a crimson specific op to manage temporary
objects and the op is not used by classic osd. It is a plain
integer constant defined in the shared transaction layer so
adding it to the common code path in Transaction.h and BlueStore.cc
requires no additional includes or compile time guards.Handling
it unconditonally is therefore safe for all Objectstore backends.
Solution: Handle OP_TOUCH_TEMP without any guard
Fixes: https://tracker.ceph.com/issues/75957
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
on a cold read, check_full_extent_integrity() compares the crc of the read
content against the crc stored in the LBA leaf (pin_crc). this aborts on a
RANDOM_BLOCK data extent that was overwritten via delta-based overwrite:
abort(extent checksum inconsistent)
4# Cache::check_full_extent_integrity()
5# Cache::_read_extent()::{lambda}
7# Cache::replay_delta() (or _read_pin on the live IO path)
the leaf crc is not a valid reference for these extents. delta overwrite
mutates the extent in place to avoid touching the LBA tree, so
update_lba_mappings() skips the leaf for is_exist_mutation_pending() extents
and the pre-mutation crc stays there. the new content lands on disk later and
out-of-band, when the cleaner rewrites the modified region in place
(RandomBlockOolWriter::do_write). that write is not atomic with the leaf, so a
cold read can pick up new content against an old leaf crc.
updating the leaf crc on overwrite is not the fix. it brings back the LBA-tree
mutation (and the txn conflicts and hot-path crc) that delta overwrite exists
to avoid, and it still does not make the leaf reliable: the data write and the
leaf update happen separately, so publishing leaf=new while disk still holds
old just moves the mismatch.
so stop trusting the leaf for these extents. their content is verified at
replay, where replay_delta() applies the deltas and
CircularBoundedJournal::replay() asserts last_committed_crc == delta.final_crc.
and the content on disk is always current: modified_region accumulates every
change via union_insert and is cleared only when the in-place rewrite commits,
so do_write writes the latest content; only the leaf crc lags.
drop the leaf-crc check on read for RANDOM_BLOCK in-place-rewritable (data)
extents, but only when delta-based overwrite is enabled. with it off (the
default), overwrites go through remapping, the leaf crc stays valid, and the
check still runs for every other extent. last_committed_crc is computed
regardless, so the delta-chain check is unaffected.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
spdk_env_opts_init() reinitializes the whole spdk_env_opts struct,
overwriting pci_allowed/pci_whitelist and num_pci_addr that were set
just above it. As a result the local PCI device restriction never took
effect. Initialize opts first, then fill in the PCI address fields.
Fixes: https://tracker.ceph.com/issues/77802
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The condition `if ((cur->n_aiocb == 1) {` has an unbalanced
parenthesis and fails to compile. This code is only built on the
HAVE_POSIXAIO (BSD) path, so the breakage is normally hidden.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Add scrubbing functionality to the OSD test fixtures to enable testing
of scrub behavior on erasure-coded pools. This implementation uses as
much of the scrub backend as possible, but avoids using pg_scrubber or
the interaction with peering due to the complexity of the dependency on
PG.
Changes:
- Add scrub_object() method to PGBackendTestFixture for testing scrub
functionality on EC pools
- Add create_random_buffer() utility for generating random test data
- Add corrupt_shard_data() method to intentionally corrupt shard data
for testing scrub detection
- Add MockScrubBeListener, MockPgScrubBeListener, MockSnapMapReader, and
MockLoggerSinkSet mock classes to support scrub testing
- Add three new test cases: ScrubClean, ScrubDetectsCorruption, and
ScrubPartialWrite to verify scrub behavior with clean data, corrupted
data, and partial writes
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
# Conflicts:
# src/test/osd/TestECFailoverWithPeering.cc
Refactored test fixture code to properly execute object context operations
within event loop context, fixing ceph_assert(osd != -1) failures.
Key changes:
- Split functions calling get_object_context() into implementation and
wrapper functions that schedule work on the primary OSD via event loop
- Added accessor functions (set_object_context, clear_object_contexts)
to encapsulate per-OSD object_contexts map access
- Fixed lambda capture bug in PGBackendTestFixture::update_osdmap()
- Refactored run_parallel_recovery_and_verify_callbacks() to execute
within proper event loop context
All object context operations now execute within the correct OSD context,
with get_object_context() as the single point querying the current OSD.
Added test ZeroSizeObjectWithAttributesRecovery to verify recovery of
zero-size objects with attributes after primary failover, ensuring
attributes are properly recovered on the failed shard.
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Assisted-by: IBM-Bob:Claude-Sonnet
# Conflicts:
# src/test/osd/ECPeeringTestFixture.cc
The test keeps random_num payload buffers (each up to max_message_len)
resident and sends copies concurrently; at 4MB the peak working set
runs ~600MB and overflows the test's 256M --memory arena with bad_alloc.
--memory only binds when seastar's own allocator is compiled in
(RelWithDebInfo/Release); Debug/Sanitize define
SEASTAR_DEFAULT_ALLOCATOR and use the heap, so the limit never bites
there.
Shrink max_message_len to 1MB, cutting the peak below 160MB so it fits
the 256M arena instead of enlarging it.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
mgr/dashboard: "Any" initiator entry appears in the Initiator table after selecting "All Hosts Allowed" during subsystem creation
Reviewed-by: Naman Munet <nmunet@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
journal tail on startup
Recover the correct journal tail during replay by scanning
JOURNAL_TAIL records instead of relying solely on the on-disk
header, which may be stale if a crash occurs between record
write and header update.
The journal header is stored *separately* from the (append-only)
journal and is not updated atomically with journal writes,
so it may lag behind after a crash.
Treat the journal as the source of truth and rebuild the tail from persisted records.
Replay is reorganized into three passes:
1) discover latest journal tail
2) rebuild allocation map
3) replay deltas
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
python 3.12 warns on unrecognized escape sequences in string literals.
fuse_mount's admin-socket pyscript and mount's nft payload carry regex
and shell escapes (\., \d, \;) that are meant literally, so make those
strings raw. filesystem's get_mds_addr docstring had a stray \/ in the
example address, which is just a slash, so drop the backslash.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
test_messenger_thrash put sharded_conf().stop() in the success
continuation, so an exception from the tests skipped it and left the
static sharded<ConfigProxy> holding live instances. its destructor then
asserts. a tester hit this when the run went OOM at --memory 256M:
ERROR [shard 0:main] test - Test failed: got exception
ceph::buffer::v15_2_0::bad_alloc (Bad allocation [buffer:1])
/ceph/src/seastar/include/seastar/core/sharded.hh:573:
seastar::sharded<Service>::~sharded()
[with Service = crimson::common::ConfigProxy]:
Assertion `_instances.empty()' failed.
terminate called without an active exception
run the body in seastar::async and stop the config through
seastar::deferred_stop, as crimson/osd/main.cc already does. the guard
is constructed only after start() succeeds, and its destructor stops the
config on any exit path, so a failing test exits cleanly instead of
aborting.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Add a WITH_MOLD cmake option that auto-configures the build for
the Mold linker. When enabled, cmake finds mold, sets CMAKE_LINKER,
injects -fuse-ld=mold into all linker flag variables, and sets
USING_MOLD_LINKER which gates the Mold-specific workarounds added
in the previous commit.
Usage: cmake -DWITH_MOLD=ON ...
Assisted-by: Claude <claude.ai>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Mold handles several linker features differently from ld.bfd:
- --exclude-libs,ALL hides .symver-aliased symbols (e.g. rados_*)
even when the version script lists them as global. Detect Mold
(via USING_MOLD_LINKER) and disable --exclude-libs; version
scripts already control symbol visibility.
- Duplicate symbols across static archives are treated as errors.
Merge crimson-alien-common into crimson-alienstore as a single
archive. For crimson-osd, link crimson-alienstore before
crimson-common with --allow-multiple-definition so the alien
thread's non-crimson definitions win (Mold picks first definition).
- Python/Cython extension builds (via Distutils.cmake) invoked the
system default linker. Pass -fuse-ld= via MOLD_FUSE_LD_FLAG so
extensions link with the configured linker.
- Add rados_* and _rados_* to librados.map global section for Mold
compatibility with .symver aliases.
Co-authored-by: Mark Kogan <mkogan@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
ceph orch ps - was showing stale or incorrect port information for `mgr` daemons after enabling or disabling mgr modules (e.g., dashboard, prometheus).
This fix ensures the displayed ports always reflect the actual active module endpoints.
Code Fixes:
-----------
Changes in src/pybind/mgr/cephadm/services/cephadmservice.py:
- Extract and refactor _get_mgr_service_ports:
The logic for reading active mgr module ports was inline inside prepare_create, using check_mon_command({'prefix': 'mgr services'}). It was extracted into a static helper method and refactored to read directly from mgr.get('mgr_map')['services'] instead -- which is faster and consistent with how get_dependencies reads the same data. The method is now shared by both prepare_create (existing use) and get_dependencies (new use).
- Extract `_get_mgr_service_ports`:
The logic for reading active mgr module ports from `mgr_map['services']` was inline inside `prepare_create`. It was extracted into a static helper method so it can be reused by both `prepare_create` (existing use) and `get_dependencies` (new use). Returns the port numbers currently registered by active mgr modules (e.g., `[8443, 9283]`).
- Override `get_dependencies` in `MgrService`:
The base class returns `[]` for all Ceph daemons. The serve loop compares `get_dependencies()` output against `last_deps` (saved after the previous reconfig). If they differ, a reconfig is triggered. By overriding this method in `MgrService`, we make the serve loop aware of mgr port changes. Returns a sorted list such as `['port:8443', 'port:9283', 'sd_port:8765']`.
- Override `generate_config` in `MgrService`:
After a reconfig completes, the result of `generate_config` is saved as `last_deps`. The inherited implementation returns `[]` as deps. If `[]` is saved but `get_dependencies` returns `['port:8443', 'sd_port:8765']`, they would always differ, causing an infinite reconfig loop. This override ensures that what is saved as `last_deps` matches what `get_dependencies` will return in the next iteration.
Changes in `src/pybind/mgr/cephadm/serve.py`:
- Update mgr cache on reconfig:
The existing logic skipped the cache update for all Ceph daemon reconfigs. A new `elif` branch for mgr reconfigs fetches the existing cached `DaemonDescription`, updates only its `ports` field, and saves it back. This makes `ceph orch ps` reflect the correct ports immediately after a reconfig, without showing a misleading "starting" status.
Changes in `src/cephadm/cephadm.py`:
- Update `unit.meta` on reconfig:
`unit.meta` is a per-daemon file on the host that stores metadata including ports. It is read periodically by `cephadm ls` and fed back into the orchestrator cache. Previously it was only written on initial deployment, so every periodic cache refresh would overwrite the corrected ports with stale data from `unit.meta`. This change updates the `ports` field in `unit.meta` during reconfig, ensuring the cache refresh confirms the correct state rather than reverting it.
Changes to src/pybind/mgr/cephadm/tests/services/test_mgr.py:
- Updated _prepare helper:
The original mocked check_mon_command to feed services data to prepare_create. Since our _get_mgr_service_ports now reads from mgr.get('mgr_map') instead of check_mon_command, the mock was replaced with mock_store_set('_ceph_get', 'mgr_map', ...) which sets the data in the right place.
- Added test_get_dependencies_changes_when_module_enabled:
A new test that verifies get_dependencies returns different dep strings before and after a module is enabled. This directly tests the core sync mechanism -- the serve loop compares get_dependencies output against last_deps to detect port changes and trigger a reconfig.
to run new tests:
cd src/pybind/mgr
tox -e py3 -- cephadm/tests/services/test_mgr.py -v
Fixes: https://tracker.ceph.com/issues/76565
Signed-off-by: Yael Azulay <yazulay@redhat.com>
Remove the full pg_map JSON dump at dout(30) which was flooding the
log queue with multi-MB entries on every tick. The much smaller
incremental dump is retained for debugging.
Fixes: https://tracker.ceph.com/issues/64082
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
docs: balancer: Explain PG shard count based balancing in more detail
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Without these breaks the switch fell through, recording a single
message's latency into every following per-type counter.
Fixes: https://tracker.ceph.com/issues/77751
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The wording "average" so far suggested a simple mean count, which it isn't.
It is important to document this, so that the user can follow
why the balancer is active or not.
Source for the logic (deviation computation and threshold check)
in `src/osd/OSDMap.cc`:
`calc_deviations()` - computes per-OSD deviation:
float target = osd_weight.at(oid) * pgs_per_weight;
float deviation = (float)opgs.size() - target;
Where `pgs_per_weight` is computed in `calc_pg_upmaps()`:
float pgs_per_weight = total_pgs / osd_weight_total;
This also has the early threshold + early return:
if (cur_max_deviation <= max_deviation) {
ldout(cct, 10) << __func__ << " distribution is almost perfect" << dendl;
return 0;
}
`fill_overfull_underfull()` classifies OSDs using `max_deviation`:
if (odev > max_deviation) {
overfull.insert(oid); // deviation > +5 -> source candidate
}
if (odev < -(int)max_deviation) {
underfull.push_back(oid); // deviation < -5 -> target candidate
}
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
crimson/os/seastore/transaction_manager: refresh `indirect_cursor` after the `direct_cursor` is removed in `TM::_remove()`
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Add a regression test that processes an insert so the keys are cached
in key_map, then processes a later clear_omap and asserts the cached
keys are returned as tombstones.
This would have caught the by-value bind in process_entries, where
clear_omap mutated copies and left already-cached keys untouched.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The clear_omap loop bound each key_map entry by value, so
update_value() mutated a copy and cached keys were never marked
removed.
As a result, clear_omap had no effect on already-cached keys: cleared
omap keys retained stale values and were returned on read.
Introduced-by: 7289e4d2ec ("osd: Add ECOmapJournal class and relocate OmapUpdateType enum class")
Fixes: https://tracker.ceph.com/issues/77750
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
mgr: fix MgrModuleRecoverDB to honor its retry budget on db cleanup failure
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
module_selftest fails on a cluster log warning that is not ignorelisted:
cluster [WRN] Health check failed: Failed to list/create InfluxDB database (MGR_INFLUX_DB_LIST_FAILED)
module_selftest is the only job that enables the influx module, and no suite
provisions an InfluxDB server, so once enabled the module cannot reach a backend
and raises a health warning: MGR_INFLUX_NO_SERVER when no hostname is set,
MGR_INFLUX_DB_LIST_FAILED from serve() once one is. test_influx sets a hostname,
so it hits the latter.
This surfaced only after the test image started shipping the influxdb python
module. Until then can_run() returned false, the module never enabled, and the
sole log line was "influxdb python module not found", which is already
ignorelisted. self_test() cannot run without enabling the module, so ignorelist
the warning it raises.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
req_data->mgr is null once the request finishes. Check it in
_set_write_paused and _set_read_paused before calling set_request_state,
otherwise a late streaming write segfaults.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
mgr/dashboard: NVMe-oF – Updated subtitle text on the Create Gateway page
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
The 'l.inc < r.inc' term in operator< lacked an 'l.name == r.name' guard,
breaking antisymmetry, so it was not a valid strict weak ordering.
Replace it with std::tie.
The only ordered container keyed on osd_reqid_t is
ECInject::write_failures0_reqid (QA/debug error injection), so no
production code path is affected and no backport is needed.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
If redmine_linkage_correct is True, _get_active_issues() is supposed to
ignore issues with "Multiple Source PRs" category. However that doesn't
happen because the category string is compared against (category, text)
pair.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Adds unit test for creating gateway service with encryption when enabled.
Fixes: https://tracker.ceph.com/issues/77359
Signed-off-by: pujashahu <pshahu@redhat.com>
fio-3.42 is the latest stable release. We switched to the ceph/fio fork in
10baab3fc8 to pick up a clang build fix; that
fix has since landed upstream, so switch back. This also gives riscv64
proper arch support (arch-riscv64.h, added in fio-3.36), which the fork's
fio-3.27-cxx lacks.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
dispatch_remote() and ceph_dispatch_remote() invoke pickle.loads/dumps via
PyObject_CallMethodObjArgs(pmodule, PyUnicode_FromString("loads"), ...). The
method-name string created by PyUnicode_FromString is a new reference that
CallMethodObjArgs does not steal, and it was never released, so every cross
-module remote() call leaked three PyUnicode objects on the active mgr.
Use PyObject_CallMethod(pmodule, "loads", "(O)", arg), which takes the method
name as a C string and manages it internally, so there is nothing to leak.
The "(O)" format wraps the argument in a one-tuple so it reaches the callee as
a single object. A bare "O" passes the argument through directly, and when it
is a tuple (as remote()'s *args always is, including the empty tuple from a
no-argument call) PyObject_CallMethod unpacks it and calls dumps() with the
wrong arguments, e.g. dumps() with none.
Fixes: https://tracker.ceph.com/issues/77724
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Refactored RGW credential management to use dedicated RGWCredential
resources instead of embedding credentials directly in Share resources.
This will help to reuse the same credential across multiple shares using
same user.
RGW credentials must not appear in the public RADOS store, which any
client with pool caps can read. Instead, write a config:merge stub
containing only the credential fields to the private mon config-key
store, and pass its URI to the container alongside the primary config
URI via extra_config_uris.
- Public config: ceph_rgw:access_key / secret_access_key are empty strings
- Private stub: stored under smb/config/<cluster_id>/config.smb.rgw,
contains only the real credential values via config:merge
- extra_config_uris: new Config field carrying supplementary URIs
appended to SAMBACC_CONFIG after the primary config_uri
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
- Add _generate_rgw_share() function for RGW share configuration
- Skip CephFS authorization for RGW shares in cluster assembly
- Configure Samba VFS with RGW-specific parameters:
- rgw_bucket, rgw_user_id, rgw_access_key, rgw_secret_key
- Validate bucket accessibility using radosgw-admin
- Ensure proper error handling in share creation flow
This enables the handler to generate proper Samba configuration for
RGW-backed shares along with validation and error handling.
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
- Add rgw.py module for RGW integration
- Implement fetch_rgw_credentials() to get user credentials from radosgw-admin
- Implement validate_rgw_bucket() to check bucket existence
- Handle bucket owner lookup when user_id not provided
- Add proper error handling and logging
This provides the foundation for RGW authentication in SMB shares,
similar to the NFS RGW export implementation.
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
- Add RGWStorage component class for RGW bucket storage backend
- Include fields: bucket, user_id, access_key_id, secret_access_key
- Add password conversion support for credentials
- Update Share class to accept optional rgw field and added convert
defination
- Update Share validation to support both cephfs and rgw backends
- Ensure exactly one storage backend (cephfs or rgw) is specified
This is the first step in implementing RGW storage support for SMB
shares, similar to the existing NFS RGW export functionality.
Signed-off-by: Shweta Sodani <ssodani@redhat.com>
common/async: wake async_cond waiters on their associated executor
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
The mgr dispatch throttle (ms_dispatch_throttle_bytes, default 100MB)
was far smaller than the OSD throttle (mgr_osd_bytes, 512MB),
making the dispatch queue the effective bottleneck on large clusters.
When the dispatch queue fills, OSD connections stall waiting for
throttle space. These connections time out and reconnect, causing
connection churn and heap growth from repeated connection setup/teardown.
The MGR also falls behind on PG stats processing, leading to unknown
and inactive PGs.
Add mgr_dispatch_throttle_bytes (default 1GB) and a
Messenger::set_dispatch_throttle_size() API to configure it at init time.
The new default is sized to accommodate the sum of all per-type policy
throttles, ensuring the dispatch queue is never the bottleneck.
Fixes: https://tracker.ceph.com/issues/66310
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
rbd-mirror: Remove old non-primary demoted image snapshots on the local cluster
Reviewed-by: VinayBhaskar-V <vvarada@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
crimson/os/seastore/lba: don't update the lba mapping if its child is an initial pending one.
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
When set\get\rm config values with leading or trailing spaces,
those spaces need to be removed before we are trying to get\set\rm
those keys.
add trim code before each approch and also for exist keys in config.
Fixes: https://tracker.ceph.com/issues/77598
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
ConnectionList::close() is called from the main thread during
AsioFrontend::stop(), but io_context pool threads may still have
pending async_write operations on those sockets. The old code called
socket.close() which is a boost::asio thread-safety violation:
- it NULLifies descriptor_data in the epoll reactor which causes
concurrent start_op calls to dereference NULL (SIGSEGV), or
- it deregisters the fd from epoll, orphaning pending operations so
their completion handlers would never fire (hang)
Fix by using socket.cancel() + native ::shutdown():
- cancel() dequeues pending operations from the reactor's
per-descriptor queue and adds completions with
operation_aborted. This should handle writes blocked in epoll.
- shutdown(fd, SHUT_RDWR) disables the transport so any
strand-queued writes that haven't entered epoll yet will fail
with EPIPE on the next send() attempt
Also remove connections.clear() since coroutines exited via I/O error
and their RAII Guard removes them from the list on destruction.
This follows the same principle as the fix for
https://tracker.ceph.com/issues/58670 which handles the same scenario
in timeout_handler path.
Bottomlime, socket operations must not be performed from outside the owning
strand.
Fixes: https://tracker.ceph.com/issues/77658
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Just like rgw_asio_frontend_timer.h, factor out the Connection struct and
ConnectionList class from the anonymous namespace in rgw_asio_frontend.cc
to a new header rgw_asio_frontend_connection.h (under namespace rgw::asio).
This is especially needed to get better coverage thru unittesting.
This is non-functional: no logic changes, the types, methods, and behavior
are identical as before. The frontend .cc uses type aliases to maintain the
same unqualified names.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
When an image is demoted on primary cluster and later promoted again,
a non-primary demoted image snapshot is created on the peer (secondary)
cluster. Each such promote/demote cycle on same cluster results in an
additional non-primary demoted snapshot being created on peer.
As a result, repeated promote/demote cycles on the same cluster can
lead to accumulation of stale non-primary demoted snapshots on the
secondary cluster. These snapshots are not removed immediately because
cleanup is only triggered when peer (secondary) is promoted and then
demoted.
The proposed changes ensures that such snapshots are proactively identified
and removed, preventing unbounded buildup. Also added test coverage to
verify cleanup behavior.
Fixes: https://tracker.ceph.com/issues/76155
Signed-off-by: Miki Patel <miki.patel132@gmail.com>
Add test_head_bucket_usage_multipart_incomplete to verify that
X-RGW-Object-Count (returned by HEAD Bucket with ?read-stats=true)
does not count in-progress multipart upload parts as completed
objects.
After uploading 2 parts without completing, HEAD bucket should
report 0 objects and ListObjectsV2 should return 0 keys.
Tests: https://tracker.ceph.com/issues/77509
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Test that multipart upload part heads are tracked under the
rgw.multipart category in radosgw-admin bucket stats, separate from
completed objects (rgw.main).
- Incomplete upload: rgw.main=0, rgw.multipart=X, rgw.multimeta=1
- CompleteMultipart: parts cleaned up, assembled object in rgw.main
- AbortMultipart: parts cleaned up, rgw.multipart returns to 0
- ListObjects returns 0 during incomplete upload
Also add quota enforcement testcase for multipart part category
- Verify that multipart upload parts tracked under RGWObjCategory::MultiPart
still count against both bucket and user quotas.
Tests: https://tracker.ceph.com/issues/77509
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Multipart upload part head objects are bucket-indexed with
RGWObjCategory::Main, making them indistinguishable from completed
objects in bucket stats. This is causing radosgw-admin bucket stats and
the RGW-extension X-RGW-Object-Count header on HEAD Bucket
(?read-stats=true) to report inflated object counts when in-progress
or abandoned multipart uploads exists.
So, this commit adds a new Object Category: RGWObjCategory::MultiPart
(value 5) and assigns it in MultipartObjectProcessor::complete() so
part heads are now tracked under a separate "rgw.multipart" category.
The X-RGW-Object-Count header, which reads only RGWObjCategory::Main,
now excludes in-progress parts. This is more correct as AWS S3 HEAD Bucket
does not expose any object count for incomplete uploads (the header itself
seems to be an RGW extension, not part of the S3 specification).
Quota enforcement, stats aggregation, and part cleanup are all
category-agnostic so should require no changes.
Also, updated the test_cls_rgw_stats simulator to use
the new category for multipart part entries.
Fixes: https://tracker.ceph.com/issues/77509
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
The Shadow enum value (2) is never assigned anywhere in the codebase.
Update the comment to clearly state it is unused and reserved for
backward compatibility, since the value is serialized as a raw uint8_t
in bucket index entries and may exist in on-disk data.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
PGBackend::getxattr() for EC pools reads from the attr_cache and
returns ENODATA when an attr is not found, without distinguishing
between "object exists but attr not set" and "object doesn't exist".
This causes cls methods like cls_refcount_get() to proceed as if the
object exists, enter the EC RMW pipeline, and hang.
Add an os.exists check in the EC path so non-existent objects return
ENOENT, matching the behavior of the replicated path which goes to
the object store and gets ENOENT directly.
Fixes: https://tracker.ceph.com/issues/77335
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Both close_db() and open_db() during retry were unguarded, so a failure
in either escaped the decorator immediately instead of retrying up to
MAX_DBCLEANUP_RETRIES.
Wrapped both in the same try/except, and logged the
failed attempt.
Fixes: https://tracker.ceph.com/issues/77635
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
The old scheduler spawned nproc jobs, then waited for the whole batch to
finish before starting the next, so cores that finished early sat idle
waiting on the slowest type in the batch. Keep a sliding window instead:
once the window is full, reap a single finished job before spawning the
next, which keeps the cores busy when per-type runtimes vary widely.
While here, pass vdir/arversion into test_object explicitly instead of
reading the loop globals, and give each job its own result file instead
of reusing slot-numbered files that the sliding window would otherwise
hand out to overlapping jobs.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Make the mgr root logger fallback follow debug_mgr and preserve the
original logger name, so third-party DEBUG logs are not emitted by
default or mislabeled as mgr logs.
Signed-off-by: Matthew N Heler <matthew.heler@hotmail.com>
OperationThrottler::get_throttle() returns a ThrottleReleaser guard whose
destructor calls release_throttle(), the mClock RequestCompletion that
decrements in_progress. The guard has to be held for the whole operation: a
caller that discards it at acquisition releases the slot in the same step, so
the operation never counts against max_in_progress and the throttle does
nothing.
Mark the type [[nodiscard]] so the compiler flags a discarded co_await of
get_throttle().
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
are being balanced
Before this commit, When multiple pools are being balanced,
pgs_per_weight only considers the last pool, which is wrong.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
According to our tests, this commit can speed up pg upmap calculations
by orders of magnitude
Performance analysis:
Suppose:
R = number of pg_upmap_items (the full candidate pool, after to_skip/only_pools filtering)
M = how many OSDs the deviation_osd loop visits before it breaks (less or equal to underfull count)
U = the number of distinct from OSDs in the upmap items. U is less or equal to number of OSDs, and should be less than R x k
k = mapping pairs per candidate (usually 1, small)
Time complexity of OSDMap::calc_pg_upmaps():
Before the commit: O(M * R)
After the commit: O(R * k + M * log(U)) ~= O(R + M *log(U))
Performant evaluation:
Existing upmap items Underfull OSDs Old path New path Speedup
50000 1000 6.51s 0.047s 137x
100000 1000 12.05s 0.086s 140x
200000 2000 63.93s 0.281s 227x
Fixes: https://tracker.ceph.com/issues/77563
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Assisted-by: Codex:GPT-5.5
This commit changes the output label of the bluestore allocator
score command from fragmentation_rating to fragmentation_score
Fixes: https://tracker.ceph.com/issues/77331
Signed-off-by: Prachi Lasurkar <lasurkarprachi@gmail.com>
New API with parameter checking was added in v2.25,
deprecating the original API used in this module.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Do not mark an upgrade complete until daemons in the filtered scope
match the target image. Limited (--limit) upgrades still complete
when remaining_count reaches zero.
Fixes: https://tracker.ceph.com/issues/62959
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
The pure-timing *_bench/BenchAlloc loops repeat a stateless path millions of
times. Under ASan that adds no memory-error coverage but inflates the run from
minutes to hours, so cut the iteration counts when built with ASan.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Drive the deref loop by get_remaining() >= step instead of comparing to
end(), so a total length that is not a multiple of step no longer lets
`iter += step` run past the end and throw "End of buffer".
The original counts were always a multiple of step, so the overrun only
showed up once the following commit shrinks the rounds under ASan -- a
false alarm rather than a real regression, which is why it went unnoticed
until now.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Collect FCM stats locally from NVMe drives (vendor log page 0xCA)
and expose them via node-proxy, the cephadm agent, and
`ceph orch hardware status --category fcm`.
Introduce a node backend that aggregates Redfish data with
node-local collectors, since FCM metrics are not available
from the BMC.
Fixes: https://tracker.ceph.com/issues/77521
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Let's use 'firmware' as the standard name in node-proxy, cephadm,
and orch hardware status.
We still accept 'firmwares' as a deprecated alias and read legacy
cache payloads transparently for backward compatibility.
Fixes: https://tracker.ceph.com/issues/77410
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
This adds AtollonSystem for AMI/Atollon BMCs:
- memory Id mapping,
- storage description fixes,
- StorageControllers based drive enrichment
It also wires vendor selection through cephadm (hw_monitoring_vendor)
and show slot/firmware in hardware status storage output.
Fixes: https://tracker.ceph.com/issues/77408
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
This adds the temperatures category and fan speed information.
Fixes: https://tracker.ceph.com/issues/77408
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
dump_metric_value_map() takes a value_map by const reference but ignored it
and iterated seastar::scollectd::get_value_map() instead. Both callers already
fetch the map and pass it in, so the map was fetched twice per call and the
argument was dead.
Iterate vmap. The output is unchanged, since the callers pass get_value_map(),
but the parameter now means what it says and the redundant fetch is gone.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Python 3.12 warns on invalid escape sequences in non-raw strings:
```
ceph-release-notes:45: SyntaxWarning: invalid escape sequence '\d'
cpatch.py:465: SyntaxWarning: invalid escape sequence '\;'
```
In `cpatch.py`, use raw strings for the two Dockerfile RUN lines so
their find -exec ... \; keeps the literal backslash-semicolon.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The ceph-api job failed under ASan: radosgw-admin aborted at startup with a
heap-buffer-overflow in boost.context's fiber resume().
==1155842==ERROR: AddressSanitizer: heap-buffer-overflow ... READ of size 8
#0 boost::context::detail::fiber_activation_record::resume() fiber_ucontext.hpp:153
#10 RGWSI_Notify::do_start(optional_yield, DoutPrefixProvider const*) svc_notify.cc:261
0x... is located 8 bytes after 1048-byte region allocated by
boost::context::detail::fiber_activation_record_initializer()
fiber_activation_record carries three extra members (fake_stack, stack_bottom,
stack_size) only when BOOST_USE_ASAN is defined. Under WITH_ASAN we build
boost.context with context-impl=ucontext and give consumers BOOST_USE_ASAN and
BOOST_USE_UCONTEXT through Boost::context's INTERFACE_COMPILE_DEFINITIONS, but
we never pass BOOST_USE_ASAN to b2. So libboost_context, which compiles
fiber_activation_record_initializer() and allocates the record, uses the
smaller no-ASan layout, while consumers that include the header (here rgw's
RGWSI_Notify::do_start via boost::asio::spawn) compile resume() with the larger
layout and read stack_bottom/stack_size past the allocation.
Pass define=BOOST_USE_ASAN to b2 so the library and its consumers agree on the
struct layout.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Getting it from our own service avoids exposing us to
supply chain attacks in our QA infrastructure.
Fixes: https://tracker.ceph.com/issues/76927
Signed-off-by: Laura Flores <lflores@ibm.com>
The CherryPy isolation refactor (PR #67227) accidentally changed the
dashboard bind address from wildcard (*:8443) to mon_ip:8443. The
get_mgr_ip() replacement was originally only for URI generation, but
the refactor passed the mutated address to CherryPyMgr.mount() as the
actual socket bind address.
This breaks the management gateway when its VIP is not on the same
interface as mon_ip, as the dashboard becomes unreachable on other
interfaces.
Preserve the original wildcard address for binding and only use
get_mgr_ip() for the advertised URI. Add regression test to prevent
future confusion between bind_addr and server_addr.
Fixes: https://tracker.ceph.com/issues/77491
Signed-off-by: Afreen Misbah <afreen@ibm.com>
The python binaries on some CI images and dev boxes ship stripped, so the
allocator frames in an interpreter-shutdown leak come through unsymbolised
(/usr/bin/python3.13+0x...) and the function-name matches above cannot apply.
leak:python3.10 already handled this for 3.10, and a stale comment claimed 3.12
does not leak.
Add leak:python3.12 and leak:python3.13, mirroring the 3.10 entry, so the
interpreter globals are suppressed whatever CPython the build uses.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
bin/ceph from a WITH_ASAN build aborts at exit, with LeakSanitizer reporting
CPython and Cython module-init allocations as leaks:
==2577940==ERROR: LeakSanitizer: detected memory leaks
Direct leak ... in PyObject_Malloc (/usr/bin/python3.12+...)
#4 __pyx_pymod_exec_rados rados_processed.c
SUMMARY: AddressSanitizer: 32113 byte(s) leaked in 30 allocation(s).
These are interpreter globals that live for the process lifetime. qa/lsan.supp
already suppresses them, but bin/ceph never loaded it: vstart.sh sets
LSAN_OPTIONS for the daemons it spawns, while a bin/ceph invoked separately
(ceph-api runs ./bin/ceph fsid once vstart.sh returns) inherits none and exits
non-zero. It stayed hidden until radosgw-admin stopped crashing in vstart and
the run reached that call.
ceph.in already re-execs with the ASan runtime preloaded under WITH_ASAN. Set
ASAN_OPTIONS and LSAN_OPTIONS first, from the CEPH_ASAN_OPTIONS and
CEPH_LSAN_OPTIONS that CMake also feeds add_ceph_test(), so the re-exec'd
interpreter starts with the suppressions loaded. Use setdefault so a value
from the caller still wins.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
add_ceph_test() spelled out the suppression-file paths and sanitizer flags
inline. bin/ceph needs the same options, so lift them into CEPH_ASAN_OPTIONS
and CEPH_LSAN_OPTIONS and have add_ceph_test() consume those. The environment
the tests run with is unchanged.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
This allows operators to do targeted GC work against specific shards, instead of working on all of them.
Signed-off-by: Nathan Hoad <nhoad@bloomberg.net>
AI-Assisted: Used AI to generate the initial implementation.
This check was failing to correctly identify whether backported commits
had actually been cherry-picked because it was checking the target branch
instead of the main branch. This change checks out the main branch
to look for the upstream cherry-picked commit.
Signed-off-by: David Galloway <david.galloway@ibm.com>
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Describe the mirroring module command that reads persisted omap
metrics, including syntax, output layout, stale detection, caching,
and comparison with the admin socket peer status interface.
Document --peer_uuid as a named argument for peer-only and
directory/peer filtering.
Also update PendingReleaseNotes
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Add snapshot_mirror_metrics_cache_ttl as a runtime mgr/mirroring module
option (default 15 seconds) instead of a hard-coded CACHE_TTL_SECS.
Both complete and partial lru_cache_timeout wrappers read the value
when caching omap metrics so operators can tune cache freshness without
code changes.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
When snapshot mirroring is not enabled for a filesystem, the
cephfs_mirror RADOS object does not exist and "ceph fs snapshot mirror
status" fails reading sync stat omap. Map rados ENOENT to a clear
MirrorException so the CLI reports that snapshot mirroring must be
enabled instead of a generic omap read failure or an uncaught exception.
Also catch unexpected errors in metrics_status like other mirror CLI
handlers, so the mgr module does not crash on failure. Return
-errno.EINVAL from the generic error path instead of the exception
message as exit code.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
When the directory is added for mirroring and the snapshot
is not taken yet, the peer_status show the following default
metrics.
{
'state': 'idle',
'snaps_synced': 0,
'snaps_deleted': 0,
'snaps_renamed': 0,
}
The mgr interface should also match that.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Persisted metrics in the cephfs_mirror omap can outlive the writing
daemon when cephfs-mirror stops or a directory is reshuffled to another
instance; mgr would keep reporting stale progress until the owning
daemon writes again.
Extend format_and_order_sync_stat_for_display() to compare persisted
_instance_id against InstanceWatcher live instances (via
FSPolicy.get_live_instance_ids()) and the directory's tracked instance
(via Policy.get_tracked_instance_id()). Mark metrics stale when the
persisted writer is no longer live (any state), or when it does not
match the tracked instance while persisted state is not "idle". Show
state "stale" with current_syncing_snap omitted.
Pass policy and live instance ids through load_sync_stat_metrics() and
fetch_sync_stat_metrics(), and into sync_stat_complete_cache and
sync_stat_partial_cache loaders. Cache hits serve already-formatted
(stale-marked) entries until TTL expiry without re-checking instance
liveness.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Add a short-lived in-memory cache for metrics returned by
"ceph fs snapshot mirror status", which reads persisted sync stats
from the cephfs_mirror object omap. Omap walks are relatively
expensive; caching reduces repeated reads when the CLI or multiple
clients poll at short intervals.
Two TTL-bucketed LRU caches (CACHE_TTL_SECS) are used instead of one
unified cache. The complete cache always holds every mirrored
directory and peer for a filesystem; the partial cache holds one
directory. A single directory-granularity cache cannot prove it has
all directories, so full-scan queries rely on the complete cache
contract.
Cache implementation (lru_cache_timeout decorator backed by
_TimedLRUCache, not functools.lru_cache):
- lru_cache has no TTL and no peek-on-hit API. Single-directory
queries must read the complete cache without loading on miss;
lru_cache.cache is also unavailable on Python 3.14+.
- complete (sync_stat_complete_cache): full omap prefix scan via
load_sync_stat_metrics. Cache key: (time_token, filesystem).
COMPLETE_CACHE_MAX limits filesystem entries per TTL window.
- partial (sync_stat_partial_cache): per-directory omap key load via
fetch_sync_stat_metrics. Cache key: (time_token, filesystem,
dir_path, peer_ids). PARTIAL_CACHE_MAX limits directory entries.
Bind cache_peek/cache_info/cache_clear via a CachedMethod descriptor
so TTL lookups receive (self, ...); otherwise single-dir status fails
with 'str' object has no attribute 'mgr'.
Serve logic (under the existing lock):
- status <fs> [--peer_uuid=<uuid>]: load complete cache on miss;
filter by peer when requested.
- status <fs> <dir>: peek complete cache (no load on miss); if the
entry contains the directory and peers, serve from complete.
Otherwise load partial cache (omap on miss).
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Reorder and format status output to match the output of asok
interface peer_status command. Return metrics under
metrics/<dir>/peer/<uuid> to match the asok peer_status layout,
including {"metrics": {}} when no peers are configured.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Add the following new interface to expose mirroring metrics
ceph fs snapshot mirror status <fsname> [<mirrored_dir_path>] [--peer_uuid=<peer_uuid>]
The cmd loads the persisted directory sync metrics from the
cephfs_mirror object's omap. Metrics are grouped by mirrored
directory and peer.
When --peer_uuid is specified, only metrics for that peer are
returned. peer_uuid is a named CLI argument (_end_positional_) so
peer-only filtering does not require a mirrored directory path.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
When a directory is removed from mirroring, the persisted directory
stats need to be removed. This patch handles the cleanup.
Omap keys must not be removed when mirrored directories are reshuffled
across cephfs-mirror daemons. The mgr release notify now carries a
purging flag (set only during permanent removal, not reshuffle), and
the daemon removes persisted stats only when purging is true. On
reshuffle with an in-progress sync, clear live current_syncing_snap
state and persist idle metrics so the acquiring daemon does not inherit
stale syncing omap entries.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Load last_synced_snap metadata from the cephfs_mirror object omap on
PeerReplayer initialization and when a mirrored directory is added.
Live current_syncing_snap metrics are not restored; they are rebuilt
when synchronization starts.
When the daemon restarts after a snapshot was synced on the remote but
metrics were not yet written to omap, loaded metadata may belong to an
older snapshot. Add reconcile_last_synced_snap() to compare against
the remote snap map, clear stale last-sync fields, and update
last_synced_snap id/name in memory.
Treat snaps_synced, snaps_deleted, and snaps_renamed as per-session
counters. Do not load them from omap; they start at zero for each
daemon session and are still reported via the admin socket. Persist
omap metrics unconditionally after reconcile so the mgr picks up the
new instance id and cleared session counters on restart.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Persist snapshot mirroring metrics to the cephfs_mirror object omap
for the mgr/mirroring module to support status command.
The tick thread only keeps omap up to date for in-progress syncs on
registered directories. Persist explicitly when stats change so omap
is updated before a directory unregisters:
- snap delete/rename propagation (inc_deleted_snap / inc_renamed_snap)
- after each successful snap sync (following set_last_synced_stat)
- after sync_snaps failure (following _inc_failed_count)
- after sync_perms failure (following _inc_failed_count)
Without the explicit call sites, omap can keep stale live or idle state
after sync completes or fails because directories leave m_registered
before the next tick.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Adds the capability to persist mirroring metrics.
The metrics are persisted in the omap of the cephfs-mirror
object. Metrics are persisted asynchronously.
Each mirrored directory path stores the corresponding
metrics as the value of a unique omap key representing
the mirrored directory. The omap key is as below.
sync_stat/<fsname>/<peer_uuid>/<mirrored_dir_path>
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
in 8ac962c6, we enabled ASan in script/run-make, which is part of
our CI workflow to build the tree. The goal was to enable us to identify
memory related issues early, but this introduced two kinds of problems:
- it's observed that some tests take around 4x time to complete in
comparison to the test time without ASan enabled
- api and dashboard e2e tests are failing because the ASan supppression
rules are not populated to them
This reverts commit 8ac962c698.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
a8ed43bfc0 replaced ct_memeq with memcmp in the ISA-L GCM accelerator,
making tag verification and the key-cache compare non-constant-time.
Restore ct_memeq for both; the OpenSSL and EVP paths already compare in
constant time.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Re-uploading the same part number in a GCM multipart upload encrypted the new
data under the same key and IV as the first upload, since the IV is
part_number||chunk_index and the part key came from the part number alone. GCM
requires a unique IV per key; reusing one to encrypt different data weakens its
confidentiality and integrity guarantees.
Generate a random 16-byte salt on each UploadPart and fold it into the part key,
HMAC(ObjectKey, BE32(part) || salt), so every upload gets a fresh key. The salt
rides RGWUploadPartInfo, and complete stores the selected part's salt in
RGW_ATTR_CRYPT_PART_NUMS, which now holds (part, salt) pairs. GET reads it back
to re-derive the key, and an empty salt reproduces the old derivation so unsalted
parts still decrypt.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Several CephFS pages described deprecated commands and features without
clear status or replacements, and one page documented an upgrade path that
predates every supported release.
* cephfs-mirroring: correct the peer_add deprecation note to reference the
actual replacement command, peer_bootstrap create, and link the Bootstrap
Peers section
* kernel-features, experimental-features: state that inline data has been
deprecated since the Octopus release and that enabling it triggers a
health warning
* upgrading: remove the "Upgrading pre-Firefly file systems past Jewel"
section; the tmap_upgrade command it referenced was removed in Kraken
Fixes: https://tracker.ceph.com/issues/77188
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
mgr/dashboard: skip the table when an nvmeof cli result has no columns
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Tomer Haskalovitch <tomer.haska@ibm.com>
Reduce the note to point existing users to the removal procedure, add a
ref target for the removal section, and rephrase the dm-cache mention to
reflect community adoption without implying official endorsement.
There are multiple problems fixed here, which are caused by operations
which perform a truncate-then-write in a single transaction.
NOTE: The only known scenario for these operations are CLS-sparsify operations.
I recommend backporting these changes to Tentacle, however, as they are
regressions in the RADOS API which can lead to data corruption.
PROBLEM: Cache not invalidated correctly:
If projected_size >= orig_size, the invalidate_cache flag is not set in the plan
This means there is potentially data in the RMW extent cache, which
may cause data corruption in subsequent writes (although not the write
being made). No actual use case for this operation is known, so it may not be as serious as
it sounds.
FIX: Invalidate cache on any truncate or "delete_first"
PROBLEM: Parity writes permitted with truncates:
Currently parity delta writes do not support operations which may have
truncates. However, if the object ended up the same size as pre-truncate, a
parity delta write may have been attemtped. This may lead to data corruption.
FIX: Block parity writes in this case.
NOTE: It is not worth the development effort to support PDW in this scenario, as
performance benefit would be minimal overall.
PROBLEM: RMW Reads can occur beyond first truncate point.
Such reads reflect the pre-truncate data (which is incorrect) and be
preserved, even if the invalidate cache flag is set (since the cache is
invalidated before the reads). This can lead to similar corruption as
the earlier "cache not invalidated"
FIX: trim the read to the lower truncate size.
PROBLEM:
When performing a truncate, the parity shards may need to be updated to
reflect truncates on other shards. These truncate writes in the plan were
overriding, rather than adding to, other writes in the op. The result was
a potentially truncated coding shard. This leads to assertions on reads.
FIX: Replace = with insert()
PROBLEM: Some shards not set to correct size on truncate-then-write
If projected_size is smaller or equal to the original size, then the
code which attempts to correctly size a shard will not run. However if
an operation performs a truncate then a partial write and does not
write to a particular shard AND the shard ends up smaller, then this
can lead to an incorrectly sized shard. This leads to assertion on reads.
FIX: Execute the shard-resize code on all truncates.
AI assistance was mainly used to write unit test. However, I cannot rule out a
contribution to the simple fixes found in this commit, so out of caution,
I place the Assisted-by tag.
Fixes: https://tracker.ceph.com/issues/77276
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Assisted-by: IBM-Bob:ClaudeSonnet/GPT
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
Fixes: https://tracker.ceph.com/issues/77589
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
readable.sh forks two short-lived ceph-dencoder processes per corpus
object(tens of thousands of processes overall). With ASAN's
LeakSanitizer enabled each process runs a whole-heap leak check at exit
that dominates runtime (~10s vs ~1s per process, measured on riscv64).
This test only validates encode/decode behaviour, not leaks, so disable
leak detection while keeping ASAN's other checks active.
check-generated.sh forks ceph-dencoder the same way, so apply the same
fix there.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Automatically run nvme-gw disable before stopping daemons on maintenance
enter and nvme-gw enable after daemons restart on maintenance exit so
gateways fail over in a controlled way.
Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
parse_known_args() silently consumes -w and all --watch-* flags
before subcommand arguments reach the command validator. When a
user ran "ceph orch ps -w", the watch handler entered cluster log
streaming mode while discarding the subcommand entirely, with no
error or indication that the orch ps command was ignored.
Add a guard at the entry of the watch handler: if any watch flag
is active and childargs is non-empty, print "Invalid command:
unused arguments: [...]" and return EINVAL. This matches the
behaviour already seen when -s/--status is combined with a
subcommand.
Assisted-by: Claude:claude-4.6-sonnet
Fixes: https://tracker.ceph.com/issues/77121
Signed-off-by: Yonatan Zaken <yzaken@redhat.com>
doc: update email address for Laura Flores
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Dan van der Ster <dan.vanderster@clyso.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
The x-amz-expiration response header describes the current-version
Expiration lifecycle action for an object. s3_expiration_header() instead
selected between the Expiration and the NoncurrentVersionExpiration rule
based on whether the object key carried an instance id:
const LCExpiration& rule_expiration =
(obj_key.instance.empty()) ? expiration : noncur_expiration;
This was wrong in two ways. OLH resolution writes the current version's
instance id into the key before the header is computed, so even a plain
GET/HEAD of the current version (no versionId) was treated as a
non-current request and reported a NoncurrentVersionExpiration rule. And
x-amz-expiration is not meant to surface NoncurrentVersionExpiration at
all. A versioned bucket whose only rule was NoncurrentVersionExpiration
therefore returned a bogus expiry-date and rule-id on the current object,
for both a plain request and a versionId request.
Compute the header from the current-version Expiration rule only, and
return no header when the request targets a specific version. The header
helper now passes the client request key (s->object_key) rather than the
OLH-resolved key, so a plain GET/HEAD of the current version is no longer
misclassified as a versioned request.
Add a unit test covering a noncurrent-only policy (no header for either a
plain or a versioned request) and a combined policy (the current
Expiration rule for the current version, no header for a versionId
request).
Tracker: https://tracker.ceph.com/issues/77564
Reported-by: Saravanan Palanisamy
Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
this task was manually editing /etc/resolv.conf to configure dns names.
on ubuntu 24, these changes were getting clobbered by NetworkManager or
systemd service restarts
instead, manage dnsmasq depending on host distro. rpm-based distros use
NetworkManager's dnsmasq plugin, while deb-based distros use
systemd-resolved to point at the local dnsmasq
Fixes: https://tracker.ceph.com/issues/76961
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Track per-OSD PG rebuild duration in prepare_stats_for_publish()
(primary OSD only). Three new counters are added to the
recoverystate_perf collection:
- pg_rebuild_duration: LONGRUNAVG time counter (sum+count pair);
This counter internally maintains 'avgcount' that tracks the
cumulative number of rebuild events.
- pg_rebuild_max_secs: maximum rebuild duration observed in seconds
- pg_rebuild_min_secs: minimum rebuild duration observed in seconds
The logic uses a per-PG in-memory latch (rebuild_start_time) to
capture the redundancy failure entry point. When a PG is first observed
to be in a vulnerable state (PG_STATE_DEGRADED,
PG_STATE_UNDERSIZED, or num_objects_misplaced/degraded > 0), it
latches info.stats.last_change as the start time, provided
last_change > last_clean, which ensures only genuine new failures
after the prior clean interval are tracked.
On recovery, the rebuild duration is computed as (now - rebuild_start_time)
and is only recorded if delta num_objects_recovered > 0 or the PG had
confirmed redundancy loss at latch time, filtering out spurious state
transitions. The latch is cleared after each recorded event.
The latch is also cleared in clear_primary_state() so that an interval
change or role transition (primary -> replica) does not carry a stale
start time or baseline recovered count into a future interval.
The new last_degraded is intentionally not used here to retain compatibility
with older Ceph branches where the field doesn't exist and requires encoding
changes. A future simplification can replace the latch with a direct
(last_clean - last_degraded) calculation once last_degraded is consistently
available.
This interim solution is a close approximation of the PG rebuild time.
These counters are scraped per-OSD by ceph-exporter and exposed to
Prometheus, enabling durability score calculations over user-defined
time windows.
Other Changes:
1. Add unit tests to TestPeeringState.cc that exercise the latch logic.
2. Add a standalone integration test to verify that the rebuild perf counters
are incremented on the primary OSD after a recovery event.
Fixes: https://tracker.ceph.com/issues/77493
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
A kernel CephFS or RBD mount can keep ``set-require-min-compat-client
reef`` from succeeding: the kernel client does not advertise the
pg-upmap-primary feature, so ``ceph features`` reports it as a luminous
client even when the daemons and every userspace client are newer.
ceph-fuse, libcephfs, and librbd advertise the full feature set and are
not affected.
Document this, and add a section on finding the holdout with
``ceph features`` and switching it to a userspace client.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
unittest-seastore runs the seastar reactor on a separate thread
(SeastarRunner) and stops it at exit without draining its pending
tasks, so a few cached extents those tasks still held are leaked at
shutdown. Suppress them by the three seastore subsystems they come from.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Several unittests fail LeakSanitizer on still-reachable allocations that
belong to third-party libraries, not to Ceph. These are boost.thread's
main-thread TLS, OpenSSL's one-time init (the ForkDeathTest children
_exit() before it is freed), and the cipher, DRBG and error-stack state
that OpenSSL and libcryptsetup keep behind the librbd encryption and
migration unittests. None get freed without OPENSSL_cleanup(), so suppress
them by their allocation entry points.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
AddCephTest.cmake runs unittests with
ASAN_OPTIONS/LSAN_OPTIONS=suppressions=qa/{asan,lsan}.supp, but vstart.sh
does not, so on a WITH_ASAN build `ceph-mon --mkfs` aborts on a still-reachable
leak that those suppressions cover and fails the "ceph API tests" job. Export
the same options when WITH_ASAN=ON.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Under WITH_ASAN Boost.Context is built ucontext-only, so consumers that
include its headers without linking Boost::context (e.g. libosd) were
still built for the fcontext backend and broke the link:
mold: error: undefined symbol: boost::context::detail::make_fcontext
Define the backend tree-wide so every consumer agrees on it.
riscv64's ASan runtime mis-handles makecontext/swapcontext, so the
ucontext fiber backend reports false-positive heap-buffer-overflows on
fiber switch that can't be suppressed. So exclude it.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
readable.sh wraps ceph-dencoder with `setarch $(uname -m) -R` to disable
ASLR on ASan builds, but the arch-qualified form also sets the personality
to that arch, which fails where setarch can't (e.g. riscv64). Use bare
`setarch -R` to only clear ASLR, and probe it first so the script falls
back to running ceph-dencoder unwrapped.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
With this build configured as RelWithDebInfo, seastar keeps its own
allocator instead of falling back to libc's. Under ASan that allocator
is called (via dlsym) before it is initialized and SIGSEGVs every
seastar/crimson unittest before main(). Define SEASTAR_DEFAULT_ALLOCATOR
under WITH_ASAN to keep seastar on the libc allocator.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
tcmalloc/jemalloc keep exporting the global operator new/delete even though
their malloc is shadowed by the sanitizer interceptor, so memory the sanitizer
allocated gets freed through tcmalloc and SIGSEGVs (e.g. seastar coroutine
frames). Force libc when WITH_ASAN is set.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
DeepCopyRequest::send_copy_object_map() holds the destination image's
owner_lock and image_lock (shared), issues an asynchronous
object_map->rollback(), and then releases the two locks through
m_dst_image_ctx. The rollback callback drives the rest of the request to
completion (handle_copy_object_map() ... finish() -> put()) on other
threads. That path needs image_lock, so it only proceeds once this method
releases it, and can then run to completion and free 'this' before the
following owner_lock.unlock_shared() executes, dereferencing the freed
DeepCopyRequest.
ASan reported this as a heap-use-after-free in
TestImageReplayer.StartReplayAndWrite: the object was freed on the finisher
thread (handle_copy_metadata() -> finish()) while the io_context thread was
still in send_copy_object_map().
The destination image ctx outlives the request, so operate through a local
reference to it and release the locks through that reference rather than
through 'this', as
operation/SnapshotRollbackRequest::send_rollback_object_map() already does.
Keep the explicit lock_shared()/unlock_shared() calls rather than scoped
std::shared_lock guards: unlike
operation/SnapshotRollbackRequest::send_rollback_object_map(),
send_copy_object_map() has four exit paths (send_copy_metadata(),
send_refresh_object_map(), finish(), and the object map rollback), each of
which must release the locks before running a different continuation, so
RAII guards would push that continuation out of the locked scope and read
less clearly than unlocking at each exit.
Fixes: https://tracker.ceph.com/issues/77551
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The mgr module install rule already excluded tests/*, but that glob only
dropped the files inside tests/ and left an empty tests/ directory in the
installed tree. The ci/ directories, which hold Dockerfiles, e2e test
scripts, and cluster specs used only for upstream CI pipelines, were not
excluded at all and were shipped by ceph-mgr-dashboard and ceph-mgr-rook.
Change the install pattern from tests/* to tests so the whole directory
is pruned, and add a ci pattern. Both patterns live in the exclude list
shared by every mgr module install rule (the module list, dashboard, and
rook), so this covers all of them in one place.
This supersedes the per-package excludes added in 8a2e0eb2fc and
d74b88500b, so drop them from ceph.spec.in and debian/rules.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The dedup subsystem was limited to 512 MD5 shards with 10% headroom,
capping effective operation at ~4 billion objects.
- Raise MAX_MD5_SHARD from 512 to 2048 and add new shard tiers for
1024 (2GB), 2048 (4GB) shards, supporting up to ~213 billion objects.
- Increase headroom from 10% to 20% to improve hash table load factor.
- Reject pools that exceed the maximum capacity with -EOVERFLOW
- Apply bucket filter in collect_all_buckets_stats() so that filtered
buckets do not inflate the object count used for shard sizing.
- Update sizing table, comments, and documentation to reflect the new
tiers and capacity limit.
Signed-off-by: benhanokh <gbenhano@redhat.com>
The async RDMA messenger reports each connection's per-connection eventfd
(notify_fd) as its socket fd(). notify_fd can legitimately be -1 -- under
fd-table pressure eventfd() returns -1, and the base RDMAConnectedSocketImpl
constructor skips queue pair *and* eventfd creation entirely when
ms_async_rdma_cm=true (the cm path is only implemented by the iwarp socket
classes). Either way an fd() == -1 socket reaches the success path of
Processor::accept and trips ceph_assert(socket.fd() >= 0) in
AsyncConnection::accept, or is fed to EventCenter::create_file_event() and
indexes file_events[-1] out of bounds. Both abort the OSD under connection
load.
Make the messenger surface the failure instead of handing back a broken
socket:
- Initialize RDMAConnectedSocketImpl::qp to nullptr. It was uninitialized and
read via get_qp() when ms_async_rdma_cm=true skips its assignment.
- Create the notify eventfd before the queue pair in the base constructor, and
adopt the cm id/channel before the eventfd in the iwarp constructor, so a
failure leaks nothing and the two failure causes stay distinguishable. Check
the rdma_create_event_channel()/rdma_create_id() return values on the touched
paths, and destroy the per-request cm id on the accept error paths (acking the
event does not free it).
- At each accept/connect construction site, reject a socket with fd() < 0
(-EMFILE) or a null queue pair (-EIO) instead of returning it. Free the
half-built socket on its owning worker thread via center.submit_to() so
~RDMAConnectedSocketImpl's center.in_thread() assertion holds and no RDMA/fd
state leaks (this also fixes the pre-existing leak on the rdma_accept()
failure path).
- Require ms_async_rdma_cm and ms_async_rdma_type=iwarp to be enabled together
in RDMAWorker::listen()/connect() (-EOPNOTSUPP with a clear message). The
two are coupled -- only the iwarp classes implement rdma_cm and they always
use it -- so any mismatch leaves a socket half-built and would otherwise
crash-loop the daemon.
- Guard EventCenter::create_file_event() against a negative fd as defense in
depth.
Fixes: https://tracker.ceph.com/issues/77547
Signed-off-by: Dax Kelson <daxkelson@gmail.com>
readable.sh can fail decoding pre-v3 corpus objects that predate the
unit field, e.g.:
**** reencode of .../objects/PerfCounterType/... resulted in a different dump ****
7c7
< "unit": 36
---
> "unit": 101
For struct_v < 3 unit is never decoded and had no in-class default, so
it read indeterminate memory and differed between the two ceph-dencoder
processes readable.sh compares. Default it to UNIT_NONE.
Also default-initialize type to PERFCOUNTER_NONE for consistency.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
A run that aborts before the cleanup fixture runs (timeout, OOM,
interrupt) leaves the clone dir behind, so the next build fails with
"destination path already exists". Remove it first.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
With WITH_ASAN, b2 runs as `b2 context-impl=ucontext headers stage` for the
build step and `b2 context-impl=ucontext install` for the install step. The
`context-impl` feature is declared in libs/context/build/Jamfile.v2, which
neither the headers/stage nor the install targets load, so b2 aborts with:
error: unknown feature "<context-impl>"
Name the context project as a target in both commands so its Jamfile (and the
feature) loads before the build request is expanded.
This works around https://github.com/boostorg/context/issues/297, fixed
upstream in
12ac945158
and first released in Boost 1.88; drop it once the bundled Boost is bumped to
1.88 or newer.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Both `tox_tests` and `env_vars_for_tox_tests` have been undefined since
f0079a1030, so this expands to `set_property(TEST PROPERTY ENVIRONMENT)`
-- a no-op. The actual per-test environment for tox tests is set in
add_tox_test() (cmake/modules/AddCephTest.cmake). Remove the leftover.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
With test venvs on system site-packages, mypy picks up the system
prettytable (3.4.0+, typed). It flags mgr's add_row(tuple) against the
list[Any] signature (src/mypy.ini) and qa's float_format = str against
the dict[str, str] property (qa/mypy.ini). Skip follow_imports in both.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Add a CEPH_PYTHON_SYSTEM_SITE switch (off by default). When set:
- setup-virtualenv.sh builds its venv with --system-site-packages;
- run_tox.sh exports VIRTUALENV_SYSTEM_SITE_PACKAGES=true for tox's venvs.
This lets distro packages satisfy test dependencies instead of pip building
them from sdist, which helps where prebuilt wheels are missing (e.g. scipy and
numpy on riscv64) by avoiding a slow rebuild when the RPMs are installed.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
GTEST_PARALLEL_COMMAND was set as an ordinary variable inside the
`if(NOT TARGET gtest-parallel_ext)` guard, so it only existed in the
first directory that include()s AddCephTest (src/common/options). Later
includes skip the guarded block and leave it empty, so PARALLEL
unittests under src/test silently ran serially.
Promote it to CACHE INTERNAL so it is visible across all directories
regardless of include order.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The py3 and coverage tox environments run the full mgr pytest suite
serially, which makes run-tox-mgr the longest test in CI. Add
pytest-xdist and pass `-n auto` to both so the suite is distributed
across the available CPUs.
pytest-xdist is constrained to <2 to stay compatible with the pinned
pytest-cov. Running in parallel also surfaced a hard-coded port in
cephadm's test_node_proxy, which now allocates an ephemeral port per
process.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
rgw: use local error code in handle_individual_object()
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
mgr/cephadm: Support cephadm certmgr with SMB/TLS configuration
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
59afb3d6 bumped rocksdb submodule in hope to address the FTBFS failure
when building rocksdb with GCC 16, but the tree still failed to build:
```
In file included from /ceph/src/rocksdb/include/rocksdb/trace_record_result.h:14,
from /ceph/src/rocksdb/trace_replay/trace_record_result.cc:6:
/ceph/src/rocksdb/include/rocksdb/trace_record.h:55:32: error: expected ')' before 'timestamp'
55 | explicit TraceRecord(uint64_t timestamp);
| ~ ^~~~~~~~~~
| )
/ceph/src/rocksdb/include/rocksdb/trace_record.h:63:11: error: 'uint64_t' does not name a type
63 | virtual uint64_t GetTimestamp() const;
| ^~~~~~~~
/ceph/src/rocksdb/include/rocksdb/trace_record.h:1:1: note: 'uint64_t' is defined in header '<cstdint>'; this is probably fixable by adding '#include <cstdint>'
```
in this change, we cherry-pick upstream fix to address this build
failure.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
when performing tests, we should enable sanitizers for detecting
potential issues. so, in this change, we enable ASsan, TSan and
UBSan.
script/run-make.sh is used by our CI job for testing PRs, so
enabling these sanitizers helps us to identify issues as early as
possible. because ASan cannot be used along with TSan, we prefer
using ASan for capturing memory related issue in favor of
detecting the multi-threading issues.
also, because of https://bugs.llvm.org/show_bug.cgi?id=23272, we
cannot enable multiple sanitizers. but we should enable UBSan as well,
once we can use a higher version of Clang than Clang-14. with
Clang-14, when enabling UBSan, we'd have following FTBFS
```
error: Cannot represent a difference across sections
```
when compiling `src/tools/neorados.cc`
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
ceph-volume inventory started including all LVM mapper devices after
c06bee965f. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
Fixces: https://tracker.ceph.com/issues/77486
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
mgr/dashboard : Support wildcard sans and zonegroup hostnames
Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Abhishek Desai <abhishek.desai1@ibm.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>
mgr/dashboard: align RGW role management with Carbon and fix API routing
Reviewed-by: Naman Munet <nmunet@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>
Ignore FS_DEGRADED in libcephfs_interface_tests and MON_DOWN in
rbd_python_api_tests to avoid false failures from transient cluster health
messages.
Signed-off-by: Adarsha Dinda <adarshadinda@Adarshas-MacBook-Pro.local>
Here we are registering smb features such remote_control and keybridge
with certificate manager and can be list with ceph orch certmgr bindings
ls
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Added function for get smb features TLSCredentials using _get_feature_certs
Added function _get_certificates_from_spec_ssl_certificates to get the
certificates for ssl_certificates
Updating features ssl certificates to default path of containers
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
AbstractWriteLog::shut_down() calls perf_stop(), which deletes
m_perfcounter, but the 5s periodic_stats timer is only canceled later
in the destructor. If it fires in between, periodic_stats ->
update_image_cache_state dereferences the freed m_perfcounter. Cancel
the timer under m_timer_lock first.
Destroying an AbstractWriteLog that was init()'ed without a matching
shut_down() is illegal, so the cancel_event() in the destructor is now
redundant and is dropped.
Fixes: https://tracker.ceph.com/issues/77501
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The ceph-mgr-dashboard and ceph-mgr-rook packages install their entire
mgr module directory, which includes a ci/ subdirectory containing
Dockerfiles, e2e test scripts, and cluster specs used only for upstream
CI pipelines. ceph-mgr-cephadm similarly ships a tests/ directory with
Python unit tests. These files have no runtime purpose on a deployed
system and should not be shipped in the binary packages.
Exclude mgr/cephadm/tests, mgr/dashboard/ci, and mgr/rook/ci via
%exclude directives.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The ceph-mgr-dashboard and ceph-mgr-rook packages install their entire
mgr module directory, which includes a ci/ subdirectory containing
Dockerfiles, e2e test scripts, and cluster specs used only for upstream
CI pipelines. ceph-mgr-cephadm similarly ships a tests/ directory with
Python unit tests. These files have no runtime purpose on a deployed
system and should not be shipped in the binary packages.
Exclude mgr/cephadm/tests, mgr/dashboard/ci, and mgr/rook/ci via
dh_install --exclude.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Add the ca_cert_required parameter to the lambda function mocking
CephadmService.get_certificates in test_prometheus_config_security_enabled
to match the updated method signature.
This ensures the test mock properly handles the new parameter that was
added to the get_certificates method.
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
with test
Here we have added a test case to test_smb.py to validate functionality
of certificate manager after apply certificates.
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Add ssl_certificates buffer for smb features like remote_control
and keybridge. when certificate applied it stores as a feature
name and SSLParameters as value where SSLParameters holds cert,
key and ca-cert.
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
A function is added _get_certificates_from_spec_ssl_certificates to get
certificates from ssl_certificates buffer. it returns TLSCredentials from
SSLParameters of ssl_certificates[feature]
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
A function call added to get ssl certificates from ssl_certificates
buffer. it returns TLSCredentials:
Signed-off-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
in order to silence the warnings like:
```
W: rbd-nbd: old-fsf-address-in-copyright-file
```
let's refere to the email address of FSF, instead of its old postal
address.
See https://lintian.debian.org/tags/old-fsf-address-in-copyright-file.html
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
~ObjectPlayer took m_timer_lock only to assert two invariants. but that
lock is borrowed by reference from the caller's SafeTimer, and an
ObjectPlayer can outlive it: a C_Fetch/C_WatchFetch completion on the
librados finisher may hold the last reference and run ~ObjectPlayer after
the timer and its lock are already gone. re-taking the freed lock is a
heap-use-after-free, which unittest_journal hits on arm64 under ASan:
==ERROR: AddressSanitizer: heap-use-after-free
journal::ObjectPlayer::~ObjectPlayer ObjectPlayer.cc:63
journal::ObjectPlayer::C_WatchFetch::~C_WatchFetch
journal::ObjectPlayer::C_Fetch::finish
the lock isn't needed though: at refcount 0 the watch has been cancelled
(m_watch_ctx == nullptr, asserted below) so no timer task references us, and
no fetch is in flight since a pending fetch holds a reference. nothing else
can touch our state. Furthermore, we can also skip acquiring `m_lock` as
well, because, in the destructor, it shouldn't really matter -- if one
of these asserts fails because the execution of the destructor races
with some `ObjectPlayer` mthod, we would get what the `assert()` was
added for. They are here to catch bugs and such a race just being
possible is a bug in itself.
Fixes: https://tracker.ceph.com/issues/77496
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
There was no coverage for extent leaks or double-frees during structural
omap B-tree transitions. While investigating a suspected extent leak, these
assertions were added and confirmed the caller-layer (parent-node) already
retires nodes correctly — no leak and no double-free.
Add count_live_omap_extents() to omap_manager_test_t, which scans the LBA
mapping space via scan_mapped_space() to count live OMAP_INNER/OMAP_LEAF
extents, and assert against it in leafnode/innernode_split_merge_balancing:
the live count returns to baseline after contraction, and is exactly 0 after
clearing the tree. This pins the caller-layer (parent-node) retirement
contract in CI.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
transaction when synchronously updating the lba mapping
Because the transaction can be captured by the functor itself
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
crimson/os/seastore/lba, TM: prevent of the users of LBACursor/LBAMapping from pass the paddr/lba_map_val_t across the boundaries of coroutines
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
The client_force_lazyio option is currently marked as supporting
runtime updates in the configuration schema, but this is misleading.
The value is read once during each file open/create and stored in
the file flags. There is no config observer registered to handle
dynamic updates, and there is no logic to propagate changes to the
already opened file handles.
This patch adds the NO_RUNTIME flag to the option definition to
correctly reflect reality.
Fixes: https://tracker.ceph.com/issues/77451
Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
Each bucket's lifecycle entry is hashed onto a shard by this count, so
changing it remaps existing entries and can leave duplicates behind.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Implements an opt-in backup mechanism for the monitor using
rocksdb::BackupEngine. Backups run on a schedule when
mon_backup_interval is set, or are triggered manually via
`ceph tell mon.* backup`. Cleanup keeps the last N, hourly,
and daily snapshots, with a free-space guard. Off by default.
Restore is offline: stop the mon and run
ceph-mon --restore-backup <dir> --yes-i-really-mean-it
optionally with --backup-version (BackupEngine logical version,
as shown by --list-backups). The mon keyring is stashed alongside
the RocksDB backup so a wiped mon_data is recovered end-to-end,
and kv_backend is stamped back when missing.
Co-authored-by: Daniel Poelzleithner <poelzleithner@b1-systems.de>
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
the last change dropped X509Req, but InternalCryptoCaller still uses
pyOpenSSL, which keeps deprecating and removing its OpenSSL.crypto and
OpenSSL.SSL APIs. pyOpenSSL's own README [1] says:
If you are using pyOpenSSL for anything other than making a TLS
connection you should move to cryptography and drop your pyOpenSSL
dependency.
none of this module makes a TLS connection, and we already depend on
cryptography and use it in cephadm/ssl_cert_utils.py.
so reimplement the caller with cryptography:
- create_private_key(): rsa.generate_private_key() dumped as PKCS8 PEM,
the same format pyOpenSSL produced.
- create_self_signed_cert(): x509.CertificateBuilder, with the dname
fields mapped to their NameOIDs.
- _load_cert() and get_cert_issuer_info(): load_pem_x509_certificate()
and cert.issuer.
- certificate_days_to_expire(): cert.not_valid_after.
- verify_tls(): compare the cert's and the key's public keys instead of
loading them into an SSL.Context.
password_hash() and verify_password() already used bcrypt, so they are
left as is. the output formats and error behavior are unchanged, hence
the existing tls tests pass without changes.
[1] https://github.com/pyca/pyopenssl/blob/main/README.rst
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The existing warning hedged with "this does not mean it will certainly
be removed", softening the message for a deprecated feature. Operators
scanning the page may begin a how-to setup without reading past the
introductory prose.
- Remove the hedging language; state plainly that cache tiering may be
removed without further notice
- Add a bold "Do not deploy new cache tiers" directive
- Add a note immediately before the how-to content scoping the page
to existing deployments only, with a pointer to preferred alternatives
Fixes: https://tracker.ceph.com/issues/77184
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
The page carried a deprecation warning on line 4 stating FileStore
"is no longer supported" but line 10 said "Filestore is preferred for
new deployments" -- a direct contradiction. Fix by:
- Expanding the warning to explicitly state FileStore must not be used
for new deployments and to use BlueStore instead
- Updating the intro paragraph to reflect that BlueStore is the
default and recommended back end, and that this page is for
pre-existing Filestore OSDs pending migration only
Fixes: https://tracker.ceph.com/issues/77183
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
The install overview cited Octopus (EOL 2022) as the cephadm minimum
and Nautilus (EOL 2021) as the Rook minimum. Update to Reef and Pacific
respectively, which are the currently supported release floor values.
Remove the repeated Nautilus/Octopus codename references in the
ceph-ansible paragraph, as these are historical release names not
needed to convey the message.
Fixes: https://tracker.ceph.com/issues/77192
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
All mirror URLs used insecure http://. Update to https://. The example
repository paths used debian-hammer and rpm-hammer (Hammer was EOL in
2016); update to the current stable release (tentacle). Also update
the GitHub mirroring link from the stale master branch to main.
Fixes: https://tracker.ceph.com/issues/77197
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
librados-intro.rst referenced ``python-rados`` for CentOS/RHEL.
rbd-openstack.rst referenced ``python-rbd`` for both apt and yum.
Python 2 reached end-of-life in January 2020; these package names
install the Python 2 bindings (or fail entirely) on current distros.
Replace with the correct Python 3 package names: python3-rados and
python3-rbd.
Fixes: https://tracker.ceph.com/issues/77200
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
This warning was added as part of Crimson pg merging and is supposed to be
meant just for Crimson pools. Correcting the if condition to reflect the same.
Fixes: https://tracker.ceph.com/issues/77459
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
this change migrate most of the OSD implementation to C++20 coroutine
when appropriate to improve the readability and maintainability.
there are two patterns in this migration:
- mechanic replacing future-promise then() chain with co_await/co_return
- replace `return <ready future>` with a `co_return` statement
- replace `seastar::do_for_each()` with a regular for loop, and
optionally restructure the code so it's more compacted.
- replace handle_exception() with regular try-catch block
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
no need to use co_return at the end of a coroutine which returns
future<>. despite that this does not incur performance penalty,
because:
co_await func() // func returns future<>
returns void
co_return is called with void operand, this statement is lowered to
promise.return_void()
and a bare "co_await func()" also calls return_void(). so they are
equivalent at runtime. but the extraneous co_return can be dispensed.
also take this opportunity to remove the bare co_return at the end
of a coroutine. it's unnecessary just like a `return` at the end of
a function returning `void`.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Previously, obsolete primary and demoted primary snapshots on the
secondary cluster were not cleaned up immediately after relocation.
Instead, old primary snapshots remained until a subsequent promote
operation triggered their cleanup, while old demoted primary snapshots
persisted until a later demote operation removed them.
Adding changes for proactive cleanup of obsolete primary and demoted
primary snapshots that are no longer required after relocation.
Also adding test coverage to validate the cleanup behavior.
Fixes: https://tracker.ceph.com/issues/76154
Signed-off-by: Miki Patel <miki.patel132@gmail.com>
Use norecover to pause recovery ops while inspecting reservation
state. Without it, recovery can complete before the admin-socket
dump is captured, causing intermittent test failures
Fixes: https://tracker.ceph.com/issues/77113
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
remove_obj() called ioctx.read() without a length, so only the first
8192 bytes of conf-nfs.* were read before write_full(). Removing exports
from a large cluster truncated and corrupted the shared config object,
leaving malformed entries such as '%url "rados:' while export objects
were deleted.
Fixes: https://tracker.ceph.com/issues/77463
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
seastar checks log format strings at compile time when
Seastar_LOGGER_COMPILE_TIME_FMT is on; the option is gated on
fmt_VERSION >= 9.0.0. ceph builds fmt via add_subdirectory(), which
leaves the version in fmt's own FMT_VERSION and never sets the lowercase
fmt_VERSION the option reads, so the check was silently off for crimson.
read the version off the fmt target and expose it so the option turns
on.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
two log calls in errorator continuations were missing an argument. the
compile-time fmt check surfaced these as a confusing "call to immediate
function ... is not a constant expression" deep in the errorator/composer
machinery rather than a plain "too few arguments", because the consteval
format failure escalates through the continuation templates:
- shard_services.cc handle_pg_create_info(): "ignore pgid {}" never
passed pgid.
- node.cc: the read_extent error handler logged "{} -- addr={},
is_level_tail={}" without the leading error; pass the std::error_code
and include <fmt/std.h> for its formatter.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
try_enter_out_dispatching()'s impossible default case logs io_state_t,
which would instantiate fmt::formatter<io_state_t> inside IOHandler,
before the formatter (declarable only after the class, since io_state_t
is nested) is visible. move just that abort path to io_handler.cc, where
the formatter is in scope, and keep the hot switch inline.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
the seastar logger checks log format strings with a consteval call at
the use site, which instantiates fmt::formatter<T> there. a
specialization declared at the bottom of a header, after inline methods
that already log T, is then "explicit specialization after
instantiation".
move the offending specializations ahead of the type: PG, LBALeafNode,
DummyNodeExtent and WatchTimeoutRequest get a forward declaration plus
the formatter before the class; overwrite_range_t, data_t and edge_t in
object_data_handler.cc move ahead of the functions that log them. while
here, trim_data_reservation() formatted a laddr_t base with 0x{:x}, but
laddr_t's formatter takes no spec, so use {}.
osd_types_fmt.h includes msg/msg_fmt.h: formatter<osd_reqid_t> formats
req_id.name (an entity_name_t), so it depends on that type's formatter
and must pull it in. otherwise logging a reqid instantiates
formatter<entity_name_t> before msg_fmt.h is seen.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
a few debug() calls had format strings that did not match their
arguments. with the seastar logger formatting at runtime these only
threw an fmt error at the matching log level, or went unnoticed:
- pg_backend.cc: "{}: object does not exist: {}" had two placeholders
but one argument; drop the stray leading "{}: ".
- pg.cc do_recover_missing(): the "need to wait for recovery ... version
{}" message never passed the eversion_t; add it.
- client_request.cc: a DEBUGDPP() carried an extra leading "{}: " on top
of the prefix the macro already injects (also fix the "rwoedered"
typo).
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Previously, LBAManager::remap_mappings() works as follows:
1. get the mapping's val;
2. remove the mapping;
3. insert the remapped mappings.
With pessimistic cc in place, during the above step 2 and 3, a rewrite
transaction that modifies the same mapping might be committed and miss
the pending lba leaf nodes because it doesn't contain the mapping at the
time.
This commit change the above workflow as follows:
1. replace the mapping with the first remapped mapping;
2. insert the remaining remapped mappings.
Note that all the remaining mapped mappings' paddrs are calculated based
on the mapping before it in the same coroutine as the insertion, which
means it'll always see the modification of a background rewrite
transaction.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
being passed across coroutines.
This is because rewrite transactions no longer invalidates other
transactions, so if paddr get passed across coroutines, it may
become outdated.
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
```
<cd-table #table
id="pool-list"
[data]="pools"
[columns]="columns"
selectionType="single"
[hasDetails]="true"
[status]="tableStatus"
[autoReload]="-1"
(fetchData)="taskListService.fetch()"
(setExpandedRow)="setExpandedRow($event)"
(updateSelection)="updateSelection($event)"
[customFilter]="true" # set this to true
(customFilterChange)="onCustomFilterChange($event)" #
get the new rules from here>
```
Fixes: https://tracker.ceph.com/issues/77290
Signed-off-by: Nizamudeen A <nia@redhat.com>
Documents the new rgw_rest_conn_connect_to_resolved_ips feature that
enables RGW to resolve HTTP endpoints for RGW services such as multisite,
into all IP addresses and distribute requests across them using
round-robin with per-IP health tracking, supporting DNS service
discovery deployments without external load balancers.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Verify counter dump registration, current-sync gauges while syncing
(full/delta), last-sync and summary counters after idle sync, and
extend mirror stats and remote-snap failure tests for per-directory
snaps_* and dir_state.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Add documentation for --client-compat parameter in 'cluster create'
command and new 'cluster update client-compat' command. This feature
enables macOS-specific SMB optimizations (fruit VFS, streams_xattr)
and can be set during cluster creation or updated for existing clusters.
Signed-off-by: Shweta Sodani <shsodani@redhat.com>
Describe per-directory labeled perf counters, labels, update behavior,
mapping to peer status, and counter reference tables. Document the
per-peer tick thread and cephfs_mirror_tick_interval, which refreshes
current-sync gauges on each tick. Add a PendingReleaseNotes entry for
the new cephfs_mirror_directory perf counter group.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Wire the remaining cephfs_mirror_directory labeled counters that were
registered in the prior commit but not yet refreshed. Live
current_syncing_snap gauges continue to be updated from the per-peer tick
thread; this commit updates last_synced_snap and per-directory snap
summary counters at the points where SnapSyncStat is actually modified.
Depends-on the cephfs_mirror_directory PerfCounters schema (dir_state,
current_*, last_*, snaps_*) added when each mirrored directory is
registered.
Prometheus / counter dump
-------------------------
These counters appear under "cephfs_mirror_directory" in "counter dump" with
the same labels as current-sync metrics (source_fscid, source_filesystem,
peer_uuid, peer_cluster_name, peer_cluster_filesystem, directory). ceph-exporter
exposes them as e.g. ceph_cephfs_mirror_directory_last_sync_bytes and
ceph_cephfs_mirror_directory_snaps_synced.
Unlike cephfs_mirror_peers, values are per (peer_uuid, directory) rather than
aggregated across all directories on the peer. Peer-level counters are
unchanged.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Introduce a new labeled perf counter group, cephfs_mirror_directory, so
per-directory snapshot mirror progress can be scraped via "counter dump" and
exported to Prometheus by ceph-exporter (e.g.
ceph_cephfs_mirror_directory_current_sync_bytes).
Design
------
* One PerfCounters instance per mirrored directory on a peer, keyed in
m_directory_perf_counters and registered on the daemon-wide
PerfCountersCollection.
* Labels on each instance (flat counter dump array entries):
- source_fscid, source_filesystem
- peer_uuid, peer_cluster_name, peer_cluster_filesystem
- directory (dir_root, e.g. "/parent/d1")
The peer_uuid label disambiguates the same directory path mirrored to
different peers.
* Counters are created in init() and add_directory(), removed in
remove_directory() and the PeerReplayer destructor.
* Priority follows cephfs_mirror_perf_stats_prio (same as
cephfs_mirror_peers).
Update path
-----------
Live / current_syncing_snap gauges are refreshed from
update_directory_current_sync_perf_counters(), called by
refresh_directory_current_sync_perf_counters() from the per-peer tick
thread (run_tick()). Each cephfs_mirror_tick_interval seconds (default 5)
the tick thread updates counters for each registered (actively syncing)
directory.
Counters registered (schema)
----------------------------
All of the following are added to the builder in this commit. Only the
"current sync" and dir_state fields listed under "Updated in this commit"
are written here; last_synced_snap and per-directory snap summary counters
are registered for a follow-up commit that updates them when stats change.
Directory state
dir_state (gauge u64)
0 = idle, 1 = syncing, 2 = failed
Maps peer_status top-level "state" (numeric; no string values).
Current syncing snapshot (peer_status "current_syncing_snap")
[Updated in this commit]
current_snap_id - snapshot id being synchronized
current_sync_mode - 0 = full, 1 = delta (snapdiff)
current_read_bps - bytes/sec read (raw, not formatted)
current_write_bps - bytes/sec written
crawl_state - 0 = N/A, 1 = in-progress, 2 = completed
crawl_duration_seconds - crawl duration; in-progress uses now - start
datasync_wait_state - 0 = none, 1 = waiting, 2 = complete
datasync_wait_duration_seconds
current_sync_bytes - bytes synced so far for this snap
current_total_bytes - total bytes for this snap
current_sync_bytes_percent - basis points (1745 = 17.45%)
current_sync_files
current_total_files
current_sync_files_percent - basis points
current_eta_valid - 0 = calculating, 1 = ETA available
current_eta_seconds - ETA in seconds when valid
Per-directory snapshot summary (peer_status snaps_*)
[Registered only; not updated in this commit]
snaps_synced, snaps_deleted, snaps_renamed
Last synced snapshot (peer_status "last_synced_snap")
[Registered only; not updated in this commit]
last_snap_id
last_crawl_duration_seconds
last_datasync_wait_duration_seconds
last_sync_duration_seconds
last_sync_timestamp - utime_t / seconds since epoch
last_sync_bytes
last_sync_files
When idle or failed, current_* counters are zeroed and dir_state reflects
0 or 2 respectively.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Introduce a per-peer tick thread controlled by cephfs_mirror_tick_interval
(default 5 seconds). The interval is re-read each iteration so configuration
changes take effect without restarting the daemon. The thread provides a
generic hook for future periodic mirroring work.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
'laddr_t' existing formatter did not support a ':x' format specifier
(actually - the output was always hexadecomal).
Here we remove the ':x', but also refactor the custom formatter to
avoid using the streambuf mechanism.
Note - SEASTORE_LADDR_USE_BOOST_U128 is no longer supported by the formatter.
Fixes: https://tracker.ceph.com/issues/77399
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
int32_t is already provided by "from libc.stdint cimport *", so the extra
ctypedef in the platform_errno.h extern block redeclares it:
warning: c_cephfs.pxd:29:4: 'int32_t' redeclared
Fixes: https://tracker.ceph.com/issues/77440
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
The seastar target exports -Wno-non-virtual-dtor as a PUBLIC compile
option, so it leaks to every target that links seastar, including C
sources such as crimson-common's reverse.c, producing:
cc1: warning: command-line option '-Wno-non-virtual-dtor' is valid for C++/ObjC++ but not for C
Guard the option with $<COMPILE_LANGUAGE:CXX> so that it only applies to
C++ compilations.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
gen_ctest_resource_file assumed online CPUs are a contiguous 0..nproc-1 range.
When a middle core is hot-offlined (e.g. for a hardware fault), that both hands
out the offline cpu id and drops an online one; crimson seastar unittests feed
the ctest resource id straight to seastar --cpuset, so hitting the offline id
aborts with "Bad value for --cpuset: N not allowed".
Use the CPUs actually schedulable by the process via sched_getaffinity().
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Add comprehensive test coverage for the new client compatibility
feature that enables macOS-specific SMB optimizations:
- test_enums.py: Add tests for ClientSupportMode enum values
(DEFAULT and MACOS) and string representation
- test_resources.py: Add tests for cluster client_compat field,
effective_client_compat property, and is_macos_compatibility_enabled
property with different mode configurations
- test_smb.py: Add integration tests for cluster_update_client_compat
CLI command including successful updates and error handling for
non-existent clusters
These tests ensure the client compatibility mode can be properly
set, retrieved, and updated at the cluster level.
Signed-off-by: Shweta Sodani <shsodani@redhat.com>
ExtentMap.split_blob, added in be93e121a9, creates the onode and buffer cache
shards as raw pointers and never frees them. once we build with ASan,
LeakSanitizer reports them (and the structures they own):
Direct leak of 9928 byte(s) ... BlueStore::OnodeCacheShard::create
Direct leak of 224 byte(s) ... BlueStore::BufferCacheShard::create
SUMMARY: AddressSanitizer: 10288 byte(s) leaked in 8 allocation(s).
every other test in this file already wraps these shards in a unique_ptr,
declared before the collection that borrows them so they outlive it. do
the same here.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Before this commit, Transaction::copied_lba_keys was never cleared
on reset, so when a transaction is reset, and another rewrite
transaction is committing, the rewrite transaction would try to
update the copied lba mapping which may not have been copied by
the reset transaction, which would result in unexpected errors.
This commit generally follows the regulations about resetting
transactions, which is clear everything on reset
Fixes: https://tracker.ceph.com/issues/76945
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
43dd4cbd37 bumped the rocksdb submodule to v7.10.2 for CVE-2022-23476,
dropping the <cstdint> includes the v7.9.2 pin carried.
db/blob/blob_file_meta.h uses uint64_t but no longer includes <cstdint>,
so it compiles only where another header pulls <cstdint> in transitively.
GCC with libstdc++ 16.1.0 no longer does, so the build fails:
db/blob/blob_file_meta.h: error: 'uint64_t' has not been declared
our targeted distros still pull it in, so the failure went unnoticed
there: ubuntu jammy (GCC 11.2.0) and noble (GCC 13.2).
bump the submodule to a cherry-pick of upstream rocksdb 72c3887167,
which fixes the same FTBFS.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
localweight to the cache backend. Removing the code
to update the localweight from GET and PUT requests.
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
These variables are only read inside assert(), which is compiled out
under NDEBUG. Mark them [[maybe_unused]] to silence the warnings while
keeping the debug-only assert() style used by the surrounding code:
src/crimson/os/seastore/lba/btree_lba_manager.cc:1078: unused variable 'orig_len' [-Wunused-variable]
src/crimson/os/seastore/omap_manager/log/log_manager.cc:73: variable 'ret' set but not used [-Wunused-but-set-variable]
src/crimson/os/seastore/transaction_manager.cc:382: variable 'intermediate_key' set but not used [-Wunused-but-set-variable]
src/mgr/PyModule.cc:166,186: unused variable 'r' [-Wunused-variable]
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Plain assert() is compiled out under NDEBUG, leaving the checked
variables unused. Use the always-evaluated gtest macros instead.
src/test/crimson/seastore/test_cbjournal.cc:586: variable 'old_written_to' set but not used [-Wunused-but-set-variable]
src/test/crimson/seastore/test_btree_lba_manager.cc:345: unused structured binding declaration [-Wunused-variable]
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Fixing these warnings:
src/crimson/os/seastore/seastore.cc:83: 'omaptree_initialize' defined but not used [-Wunused-function]
src/crimson/osd/replicated_recovery_backend.cc:733: 'nullopt_if_empty' defined but not used [-Wunused-function]
src/test/rgw/test_rgw_kms_cache.cc:63: 'rethrow' defined but not used [-Wunused-function]
src/test/librados/test_cxx.cc:215: variable 'cmd' set but not used [-Wunused-but-set-variable]
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Problem:
When multiple monitors are removed from the monmap in a single update,
the iterative rank adjustment in notify_rank_removed() fails to maintain
the invariant that our local rank matches the monmap's new_rank until
all iterations complete.
Problem:
The old code decremented rank for each removed rank:
```
for (auto i = monmap->removed_ranks.rbegin(); ...) {
int remove_rank = *i;
int new_rank = monmap->get_rank(messenger->get_myaddrs());
// Old logic (in ConnectionTracker):
if (remove_rank < rank) {
--rank;
}
// This assert would fail on first iteration when removed_ranks.size() > 1
ceph_assert(rank == new_rank);
}
```
When removed_ranks = {0, 1} and we are mon.d (originally rank 3):
- new_rank from monmap = 1 (final correct value after removing 0 and 1)
Iteration 1 (remove rank 1):
Before: rank = 3
Execute: rank_removed=1 < 3, so --rank -> rank = 2
Check: rank == new_rank? -> 2 == 1? FAIL
Iteration 2 (remove rank 0):
Before: rank = 2
Execute: rank_removed=0 < 2, so --rank -> rank = 1
Check: rank == new_rank? → 1 == 1? PASS
The invariant is violated during intermediate iterations because each
--rank adjustment only accounts for ONE removed rank, but new_rank
from monmap already accounts for ALL removed ranks.
Solution:
Trust the caller's new_rank from monmap instead of trying to compute it
iteratively. The monmap has already recalculated ranks correctly by
removing all monitors atomically, so we should just use that value.
Changed:
```
if (rank_removed < rank) {
--rank;
}
```
To:
// Trust the caller's new_rank from the monmap rather than trying to compute it.
// This handles cases where multiple ranks are removed simultaneously.
rank = new_rank;
Now the invariant holds on every iteration regardless of how many ranks
are removed.
Fixes: https://tracker.ceph.com/issues/77423
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Refactor get_perf_schema_python() and get_unlabeled_perf_schema_python() to use Formatter::ObjectSection and Formatter::ArraySection RAII helpers instead of manual close_section() calls. Manual closes have have led to crashes when close_section() calls were left unguarded, for example when an OSD with no perf counters had it's perf counters queried.
Fixes: https://tracker.ceph.com/issues/74693
Signed-off-by: stzuraski898 <steven.zuraski@ibm.com>
/home/runner/work/ceph/ceph/src/script/ptl-tool.py:1743: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
* refs/pull/69446/head:
python-common/cryptotools: stop using the removed X509Req API
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
rgw_a uses rgw::dmclock::* (rgw_schedulers) and kmip_* (kmip) but never
declared either, so each consumer relinked them by hand. ld.bfd hid this
via lazy archive extraction; mold pulls the members and fails with
undefined symbols. Declare it on rgw_a (PRIVATE) and drop the now
redundant explicit links from radosgw and the rgw shared library.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Make PeerReplayer::add_directory() idempotent when the mgr re-sends
acquire for a directory already in the replayer list.
Fixes: https://tracker.ceph.com/issues/77398
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Verify that reloading the mirroring module and removing a directory
does not leave a ghost replayer entry that keeps syncing snapshots.
Fixes: https://tracker.ceph.com/issues/77398
Signed-off-by: Kotresh HR <khiremat@redhat.com>
sccache is enabled via WITH_SCCACHE, but run-make.sh only printed
ccache statistics after the build. Print `sccache --show-stats` too
when sccache is available.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
MgrStandby only registered SIGHUP, leaving SIGTERM/SIGINT at OS defaults.
When a standby is stopped via `ceph orch daemon stop`, SIGTERM terminates
it with exit code 143, causing systemd to mark the unit as failed instead
of stopped, which triggers CEPHADM_FAILED_DAEMON.
Mirror the handler already used in Mgr::init() for the active mgr.
Fixes: https://tracker.ceph.com/issues/77116
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
test_remap_pin_concurrent races 32 transactions to remap or overwrite the
same extent; exactly one is expected to commit while the rest detect the
conflict and back out. The harness helpers try_get_pin() and try_get_extent()
treated only ct_error::eagain as a tolerable racing outcome -- returning an
empty result so the caller's is_conflicted()/null check counts the
transaction as conflicted -- and routed every other error to
ct_error::assert_all(), aborting the test.
When a competing transaction retires or remaps a mapping mid-race, a
subsequent get_pin()/read_extent() of that laddr legitimately resolves to
ct_error::enoent (ENOENT from BtreeLBAManager::get_cursor) rather than
eagain. The helpers then aborted with "get_extent got invalid error" instead
of treating it as a lost-the-race back-out. The failure is intermittent and
pre-existing: a gap in the test harness, not a fault in the store.
Handle enoent exactly like eagain in try_get_pin() and both try_get_extent()
overloads: return an empty result so the existing conflict/back-out path runs.
Reproduction:
ninja -C build unittest-transaction-manager
./bin/unittest-transaction-manager \
--gtest_filter='*test_remap_pin_concurrent*' --smp 1 --gtest_repeat=30
Before this change the run aborts within a few iterations with
"get_extent got invalid error"; with it, all 30 iterations across the four
device variants pass.
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
0fb99092ef renamed the cypress page-registry key "welcome" to
"onboarding" in urls.po.ts and updated 01-create-cluster-welcome.feature,
but missed the Background of 02-create-cluster-add-host.feature. so
urlsCollection.pages["welcome"] is undefined and the step fails with
TypeError: Cannot read properties of undefined (reading 'url')
at cypress/e2e/common/global.feature.po.ts:12
this fails the Background of every scenario in the spec, and cascades to
the later create-cluster specs that depend on the expanded cluster.
update the reference to "onboarding" to match the registry.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Module-mode FindProtobuf defines only the classic targets, so
gRPCConfig.cmake's find_dependency(Protobuf) fails on the partial
export set. Prefer config mode; fall back to module mode.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Fixes: https://tracker.ceph.com/issues/77391
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Update the mirroring documentation and also the
release notes with new metrics introduced and it's
availability via 'fs mirror peer status' asok
interface.
Fixes: https://tracker.ceph.com/issues/73453
Signed-off-by: Kotresh HR <khiremat@redhat.com>
The 'crawl' and 'sync-mode' metric is added.
sync-mode: full/delta,
"crawl": {
"state": "completed",
"duration": "37s"
}
sync-mode:
---------
The 'sync-mode: full/delta' is added to peer status.
The 'delta' means, blockdiff along with snapdiff is
being used to sync the files where as 'full' means
full directory is crawled and each file is synced
entirely.
crawl:
-----
The state can be in-progress/completed. This
identifies whether the crawler thread is done
queuing the files for data sync threads.
The time taken for the duration is also shown.
If the crawl is in-progress, the duration
would show the time taken till then from the
start of the crawl. If the crawl state is
completed, then duration indicates total
time taken for the crawl.
The crawl duration is shown in "d h m s" format.
The existing 'sync_duration' in last_synced_snap
is also formatted
The values are as below. When crawl state is
completed, the 'total_files' metric doesn't
grow anymore.
crawl_duration:
--------------
The crawl_duration of last snapshot is saved in last_synced_snap
section as well.
Sample outputs:
---------------
{
"/d0": {
"state": "syncing",
"current_syncing_snap": {
"id": 2,
"name": "d0_snap0",
"sync-mode": "full",
"crawl": {
"state": "in-progress",
"duration": "21s"
},
"bytes": {
"sync_bytes": "149.25 MiB",
"total_bytes": "176.47 MiB",
"sync_percent": "84.57%"
},
"files": {
"sync_files": 4931,
"total_files": 5845,
"sync_percent": "84.36%"
}
},
"snaps_synced": 0,
"snaps_deleted": 0,
"snaps_renamed": 0
}
}
------------------------------------------
{
"/d0": {
"state": "syncing",
"current_syncing_snap": {
"id": 2,
"name": "d0_snap0",
"sync-mode": "full",
"crawl": {
"state": "completed",
"duration": "37s"
},
"bytes": {
"sync_bytes": "891.39 MiB",
"total_bytes": "901.52 MiB",
"sync_percent": "98.88%"
},
"files": {
"sync_files": 29656,
"total_files": 30000,
"sync_percent": "98.85%"
}
},
"snaps_synced": 0,
"snaps_deleted": 0,
"snaps_renamed": 0
}
}
---------
{
"/d0": {
"state": "syncing",
"current_syncing_snap": {
"id": 3,
"name": "d0_snap1",
"sync-mode": "delta",
"crawl": {
"state": "completed",
"duration": "15s"
},
"bytes": {
"sync_bytes": "120.20 MiB",
"total_bytes": "149.94 MiB",
"sync_percent": "80.16%"
},
"files": {
"sync_files": 4032,
"total_files": 5000,
"sync_percent": "80.64%"
}
},
"last_synced_snap": {
"id": 2,
"name": "d0_snap0",
"crawl_duration": "17s",
"sync_duration": 45,
"sync_time_stamp": "5642.805770s",
"sync_bytes": "300.85 MiB",
"sync_files": 10000
},
"snaps_synced": 1,
"snaps_deleted": 0,
"snaps_renamed": 0
}
}
-------------
{
"/d0": {
"state": "idle",
"last_synced_snap": {
"id": 2,
"name": "d0_snap0",
"crawl_duration": "17s",
"sync_duration": "2m 38s",
"sync_time_stamp": "9259.225009s",
"sync_bytes": "901.52 MiB",
"sync_files": 30000
},
"snaps_synced": 1,
"snaps_deleted": 0,
"snaps_renamed": 0
}
}
Fixes: https://tracker.ceph.com/issues/73453
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Add following mirroring progress metrics to current_syncing_snap
as below
bytes:
sync_bytes - bytes synced till now
total_bytes - total bytes to be synced
sync_percent - Percentage of bytes synced till now
files:
total_files - Total files to be synced
sync_files - files synced till now
sync_percent - Percentage of files synced till now
sync_files and sync_bytes are also stored in last_synced_snap section
after the snapshot is synced.
The bytes is formatted as below.
Sample output:
--------
{
"/d0": {
"state": "syncing",
"current_syncing_snap": {
"id": 3,
"name": "d0_snap1",
"bytes": {
"sync_bytes": "120.20 MiB",
"total_bytes": "149.94 MiB",
"sync_percent": "80.16%"
},
"files": {
"sync_files": 4032,
"total_files": 5000,
"sync_percent": "80.64%"
}
},
"last_synced_snap": {
"id": 2,
"name": "d0_snap0",
"sync_duration": 45,
"sync_time_stamp": "5642.805770s",
"sync_bytes": "300.85 MiB",
"sync_files": 10000
},
"snaps_synced": 1,
"snaps_deleted": 0,
"snaps_renamed": 0
}
}
Fixes: https://tracker.ceph.com/issues/73453
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Set bluestore_allocation_from_file flags to 'startup'.
Without it, documentation claims the flag to be 'runtime updateable'.
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
PR #68428 plumbed force_delete_data through orchestrator daemon/service
removal; update the test_orchestrator stub so orchestrator_cli QA
(test_mds_rm, etc.) does not fail with an unexpected keyword argument.
Fixes: https://tracker.ceph.com/issues/77374
Signed-off-by: Kobi Ginon <kginon@redhat.com>
osd/scheduler: Classify subOp reads according to op priority for mClock
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
mgr/cephadm: Control cephadm files logging based on a mgr flag
Reviewed-by: Redouane Kachach <rkachach@ibm.com>
Reviewed-by: Adam King <adking@redhat.com>
Enable supports_omap flag in Fast EC pools.
Enable testing of omap operations in Fast EC pools.
Fixes: https://tracker.ceph.com/issues/77329
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Add documentation for the new cephadm binary logging level configuration
Fixes: https://tracker.ceph.com/issues/74872
Signed-off-by: Ashwin M. Joshi <ashjosh1@in.ibm.com>
Introduces a new 'cephadm_binary_logging_level' config option to control
the verbosity of cephadm logging to persistent destinations (cephadm.log, syslog).
- Adds --logging-level CLI flag (info, debug, error, warning)
- Adds mgr/cephadm/cephadm_binary_logging_level config option
- Applies logging level to file and syslog handlers
- Console handlers maintain their defaults for terminal UX
Fixes: https://tracker.ceph.com/issues/74872
Signed-off-by: Ashwin M. Joshi <ashjosh1@in.ibm.com>
Remove the warning that recommends using wpq scheduler as a fallback for EC
clusters. This issue is addressed by considering EC recovery reads as
background, assigning an accurate cost for those reads and tuning the QoS
parameters associated with best-effort class of operations.
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Adjust the 'background_best_effort' queue parameters across the
three standard mClock profiles (high_client_ops, balanced, and
high_recovery_ops) to ensure best effort ops are not starved.
Previously, the 'background_best_effort' queue carried a default allocation
of 0% (MIN) reservation and a weight of 1 under these profiles. When
concurrent client traffic is dense, the zero-reservation for example completely
starves backfill sub-ops (MSG_OSD_EC_READ) on pools with
'allow_ec_optimizations' set to false. This starvation forces the Primary OSD
to hold internal BlueStore transactions and PG object locks for extended
windows, causing severe client median (50th) latency inflation.
To prevent background starvation and resolve the effects of the primary lock
retention, the profile configurations are tuned as follows:
The following profile changes forces low-cost sub-ops to clear out of peer
queues rapidly to drop primary locks, which helps improve the client
completion latency and tail latency (95th, 99th and 99.5th) percentile.
1. high_client_ops profile:
- Grant 'background_best_effort' a safe 5% minimum reservation.
- Scale the queue weight to 4.
2. balanced profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
3. high_recovery_ops profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
4. Modify the mClock config reference documentation to reflect the tuning
changes to the best-effort QoS parameters across the profiles.
Note on Proportional Scaling Compatibility:
Configuring these changes shifts total reservations to 105% (e.g., 50%
client + 50% recovery + 5% best-effort under the Balanced profile). Under
heavy concurrent saturation, mClock's internal controls resolves this
gracefully via proportional down-scaling, preserving the underlying
device bandwidth limits for different classes of clients. For example instead
of the client being allocated 50% bandwidth, a slightly lower reservation is
allocated while shifting the remaining bandwidth to the best-effort queue.
This minor scaling shift is virtually unnoticeable to the client application,
but it prevents the internal queue deadlocks.
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Add perf counters to show the status pertaining to the number of ops,
dynamic queue lengths, queue latency and bytes read for the following
ops handled in the high queues and in the scheduler queues:
- peering
- client
- ec reads/writes
- ec recovery reads
Additional counters can be added in the future based on the requirement.
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Previously, sub-op reads returned a hardcoded cost of 0, bypassing
mClock's background bandwidth and tag calculation mechanisms. This
allowed backfill operations to proceed un-metered, occasionally causing
backend resource contention and driving up client tail latencies.
Cost is calculated based on whether the complete chunk/shard or a subchunk
needs to be read. The possible cases are:
1. Read the complete chunk aligned length:
- Cost is set to the length of the chunk aligned extent size.
2. Fragmented reads:
- Consider the subchunk length and count to calculate the cost.
- compute_cost evaluates the exact layout of fragmented shard bytes on
disk by summing up the active subchunk allocations exactly once
(`fragmented_shard_bytes += k.second * subchunk_size`).
- Linear Extent Scaling: Scale the baseline footprint cleanly by
multiplying it against the true count of read extents (`tl.size()`),
achieving a highly efficient O(N) time complexity.
This linear cost model is compatible with pools running with
'allow_ec_optimizations' set to true. Under the FastEC optimized
pipeline, most operations are unified and bypass fragment slicing,
meaning requests will primarily match the Case 1 chunk-aligned path.
In Case 2 where applicable, the O(N) loop ensures that cost will
scale proportionally according to the layout.
It is important to note that the amount of data to read was set to an upper
bound defined by osd_recovery_max_chunk (8 MiB) and was rounded up to the
stripe width. The reason for setting a higher than actual upper bound is that
there may be cases where the object doesn't have the xattrs yet to determine
its size. Therefore, the amount to read was ultimatly set to ~(8 MiB / k)
where k is the number of data shards. This can cause mClock to prolong
the recovery times as items stay longer in the queue. To address this, the
amount to read is set to the remaining length of the object to recover
if the object size is known. Otherwise, the amount to read is set to the
recovery chunk size as before. Therefore, in some cases, only the first
recovery read could be costly if the object context is not known.
The MOSDECSubOpRead class introduces the following:
- cost member. This necessitates an increment to the HEAD_VERSION and
appropriate handling within the encode and decode methods.
- compute_cost() that is called when creating the message by
ECCommonL::ReadPipeline::do_read_op(). This calls into ECSubRead::cost()
that performs the actual calculations to set the cost based on the cases
mentioned above.
- The same sequence applies to the EC optimized path in
ECCommon::ReadPipeline::do_read_op().
Fixes: https://tracker.ceph.com/issues/71655
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
The change brings MSG_OSD_EC_READ into the fold of mClock scheduler. This
improves the scheduling of client and other classes of operation as they
are no longer unnecessarily preempted by the 'immediate' queue.
EC SubOps are now handled as follows:
- EC SubOp reads generated during recovery will either go into the
'background_recovery' or 'background_best_effort' class based on
the recovery priority set for the op. EC SubOp reads generated due
to client will continue to be classified as 'immediate'.
- EC SubOp writes generated as a result of client operations will
continue to be classified as 'immediate'.
- EC SubOp replies are considered high priority and therefore
continue to be classed as 'immediate'.
Fixes: https://tracker.ceph.com/issues/71655
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
The earlier output formatting was resulting in the value and string
representation of the SchedulerClass being clubbed together for
e.g., "3client"
The formatting is now fixed to log SchedulerClass as "3 (client)".
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
we encode the description of a certain command, with comma-separated
values, which are represented in "key=value" form. the keys are a set of
configuration names, and their values' types are known. among them,
"positional" and "req" are of boolean type. before this change, we
consider "true" and "True" as true, otherwise we consider the value as
false. but this setting leads to confusion and could be a headache for
maintainers.
since all commands are determined at compile-time, and all commands'
descs use "req=true" or "req=false" if "req" is specified, we are
allowed to tighten the check to enforce that these values are "true"
or "false".
this change serves as part of a sanity check of all command
descriptions, as all command descriptions are formatted into JSON
with the "get_command_description" asok command and the mon command
with the same name. and it does not change the runtime behavior.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
in 46586b22, we introduced the "osd pool application get" command, but
in this command's desc field, we had a typo: "req=fasle". this didn't
cause any issue, because we always consider "true" or "True" as true,
and false otherwise when formatting a command desc into JSON.
but it's still not correct. let's fix this typo.
Fixes: https://tracker.ceph.com/issues/77251
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Skip default routes in ip -j BGP JSON parsing, fix black line-length
in loopback check, and update list-networks CLI help for --allow-lo-routes
and --allow-bgp-routes.
Signed-off-by: Kobi Ginon <kginon@redhat.com>
This reverts commit fd20467981.
kmip tests are failing again on ubuntu 24 because PyKMIP doesn't support
python 3.12. we'll be removing ubuntu 22 from main, so can't just pin the
test to that distro in the meantime
we're expecing the nvmeof team to add python 3.12 support in our
ceph/PyKMIP fork, and can reenable kmip testing once that happens
Fixes: https://tracker.ceph.com/issues/76995
Signed-off-by: Casey Bodley <cbodley@redhat.com>
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Fixes: https://tracker.ceph.com/issues/77272
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
PGBackend::remove() was returning success when asked to delete a
non-existent object or an already-whiteout object that must remain
a whiteout. The classic OSD returns -ENOENT in both cases. Fix both
paths to return enoent, and remove the duplicate !os.exists check.
Fixes: https://tracker.ceph.com/issues/76529
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Initialize the uuid member in the Peer constructor and include it in
stream output operator to ensure correct peer identification in logs.
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
The zone creation request from the rgw service creation form was missing
the tier_type, sync_from and sync_from_all properties as a result the
zone creation was failing. This PR tends to fix this issue.
Fixes: https://tracker.ceph.com/issues/77263
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Document CLI commands (set/get/get-value/ls/rm), the Python API via
CephSecretsClient, secret URI embedding and resolution, and epoch-based
change detection.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Add pytest coverage for the full stack: secret types and URI/path
parsing, storage backend contract, Mon KV store (CRUD, epoch,
serialization, corruption handling), SecretMgr (scan/resolve),
module RPC surface and CLI handlers, and the CephSecretsClient
wrapper. Gate test imports on the UNITTEST env var following the
SMB module pattern.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Add a thin typed client around mgr.remote() for consuming the
ceph_secrets module. Exposes get/set/rm, epoch and version queries,
batch version fetch, scan and resolve helpers. Lives alongside
ceph_secrets_types.py so any mgr module can import it without
depending on the ceph_secrets package directly.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
This commit has the following changes:
1) Add the ceph_secrets mgr module entrypoint and wire it to
SecretMgr. Implement the core RPC surface consumed by other mgr
modules (secret_ls/get/set/rm, secret_get_value, secret_get_version)
and keep the implementation focused on the internal API.
2) Add user-facing CLI commands (ceph secret ls/get/set/rm) using
parse_secret_path. Secret data is accepted via -i (inbuf) only for
script-friendly usage. Add secret get-value for plain-string output
without a JSON envelope. Ensure consistent JSON output and error
mapping to EINVAL/ENOENT, while preserving safe non-reveal defaults
unless explicitly requested.
3) Add the scanning and resolution helpers (scan_refs,
scan_unresolved_refs, resolve_object) through the ceph_secrets module
RPC API. This lets consumers reliably detect secret:/... references and
resolve them inside nested objects without duplicating logic. The
behavior is delegated to SecretMgr to keep parsing/resolution
consistent across the stack.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Introduce SecretMgr to encapsulate higher-level behavior on top of
SecretStoreMon: listing helpers, scan_refs, scan_unresolved_refs, and
resolve_object (walk nested dict/list structures). This keeps
parsing/substitution logic out of the mgr module entrypoint and makes
consumer behavior consistent. The module can now resolve secret://…
references deterministically and provide structured scan output.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Introduce SecretRecord (data, metadata, versioning, timestamps) and
the canonical KV prefix (secret_store/v1/…). Add JSON serialization
helpers (to_json) including the ability to omit secret data unless
explicitly requested. This commit defines the “what we store and how
it looks” without wiring any mgr interactions yet.
Add SecretStoreMon implementing the backend using mgr’s KV
store (get_store, set_store, prefix listing). Implement set/get/rm
semantics, version increments, and list-by-prefix queries for
namespace/scope/target. This isolates persistence logic from CLI/RPC
concerns and provides deterministic record behavior for later layers.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Change handling of reads_sent bool in do_read_op.
Use errors to determine omap_satisfied in handle_sub_read_reply.
When a head object is deleted, and a whiteout is created, a REPLACE op should be used so that the ECOmapJournal knows to handle the operation as a delete. This should avoid the scenario where omap updates that should be applied to a clone, are instead applied to the whiteout object.
Fixes: https://tracker.ceph.com/issues/77329
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Define a minimal backend protocol for secret persistence
operations (get/set/rm/list), keeping the module implementation
decoupled from the backing store details. For now we will start with
monstore-db as secure KV store but the idea is to extend this to other
backends such as Vault.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Introduce the shared types and parsing logic used across the secrets
module: secret scopes, secret references, and the exception hierarchy.
Includes validation for all supported addressing forms and clear
error messages on malformed input.
Fixes: https://tracker.ceph.com/issues/74562
Assisted-by: Claude <claude.ai>
Assisted-by: ChatGPT <chatgpt.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
node-exporter exits with status 143 when terminated via SIGTERM.
Systemd treats this as a failure by default, causing cephadm to
report an intentionally stopped daemon as being in an error state.
Allow daemon forms to provide daemon-specific success exit status
codes and generate a corresponding systemd SuccessExitStatus
drop-in when required.
For node-exporter, configure exit status 143 as successful so
intentional stops are reported as stopped rather than failed.
Fixes: https://tracker.ceph.com/issues/68863
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>
LC delete actions now emit LIFECYCLE.DELETE.OBJECT journal records
via no-req_state overloads of the bucketlogging frontend.
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
ceph-osd-crimson and ceph-osd-classic call update-alternatives in their
%posttrans and %preun scriptlets but don't depend on it. declare it as a
scriptlet dependency so the binary is there when they run.
Fixes: https://tracker.ceph.com/issues/77323
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
a37b5b5bde added %preun scriptlets that use ${_sbindir}, which is
shell syntax rather than an rpm macro, so it expands to empty at run
time and the scriptlet runs "/update-alternatives", failing on
uninstall/upgrade with:
/var/tmp/rpm-tmp.K1fvm3: line 2: /update-alternatives: No such file or directory
error: %preun(ceph-osd-crimson-2:20.3.0-5054.g33c1d671.el9.x86_64) scriptlet failed, exit status 127
Error in PREUN scriptlet in rpm package ceph-osd-crimson.
use %{_sbindir}, like the %posttrans --install lines already do, so it
expands to /usr/sbin/update-alternatives at build time.
Fixes: https://tracker.ceph.com/issues/77323
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Add a cephadm "deploy-file" subcommand that reads raw file bytes from stdin
and atomically installs them at an absolute destination (--fsid, --path,
optional --mode, --uid/--gid). Use it from the mgr for client conf/keyring
sync and tuned profiles instead of SSHManager.write_remote_file.
Keep staging the cephadm binary over SSH via _write_remote_file when sudo
hardening is off, the invoker deploy_binary path is unchanged when hardening
is on.
SSH: pass encoding=None to asyncssh conn.run when stdin is bytes so binary
payloads (deploy-file) are not UTF-8-encoded as str. Only add the input kwarg
when stdin is not None.
Fixes: https://tracker.ceph.com/issues/74045
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Added cephadm sysctl-dir with mutually exclusive --list (sorted basenames
under /etc/sysctl.d) and --apply-system (sysctl --system).
The cephadm mgr tuned profile logic calls sysctl-dir for listing and
reload, and remove-file for stray *-cephadm-tuned-profile.conf files
Fixes: https://tracker.ceph.com/issues/74045
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Add a cephadm remove-file subcommand that deletes only regular files.
Have the mgr call it from CephadmServe._write_client_files when pruning
stale client keyrings instead of running ssh rm, so removal goes through
the same cephadm/invoker path as other host operations.
Fixes: https://tracker.ceph.com/issues/74045
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Added a `cephadm check-online` subcommand that returns 0,
and use it from the mgr cephadm offline host watcher via
`CephadmServe._run_cephadm` instead of SSHing `true` directly.
Fixes: https://tracker.ceph.com/issues/74045
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Fixes: https://tracker.ceph.com/issues/74045
Revert "mgr/cephadm: Use 'cephadm exec' to execute bash commands, just commands requires to deploy cephadm binary and which command should be executed directly"
This reverts commit 3a7000aa5010a8a4ad862b321aa32f36a49ac283.
Revert "cephadm: added cephadm exec command to execute shell commands"
This reverts commit 53d87b0697b5a6bd08caa03d0d795880ba4b742b.
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Since we execute the cephadm binary directly with sudo,
we get exit code 1 with a “command not found” error if the binary does not exist.
Handled it in run_cephadm to generate cephadm binary in this case.
Fixes: https://tracker.ceph.com/issues/74045
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
When validating usernames, special characters such as '#' were not being URL-encoded before being appended to the user lookup endpoint.
Because the browser treats '#' as a URL fragment, the request path was truncated, causing the backend to return an incorrect response and the UI to display a misleading "User already exists" validation error.
Fixes: https://tracker.ceph.com/issues/77129
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Seastore cannot merge collections between reactor shards currently.
On cross-shard detection, tell the monitor the source PG is not ready
(via MOSDPGReadyToMerge{ ready=false }) so the unsafe pg_num decrement
is never proposed, then send MOSDPGStopMerge to clamp pg_num_target and
permanently disable further shrink for the pool.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Integrate PG merge handling into the map advancement pipeline.
When pg_num shrinks between epochs, check_for_merges() returns a
merge_result_t describing whether this PG is a merge source, target, or
not involved. start() stops advancing through later epochs once a merge
is detected, then either finish_merge_advance() or the normal activate
path runs so complete_rctx() always happens in one place.
- check_for_merges(): detect pg_num shrink and dispatch merge_pg().
- merge_pg(): merge-only work — Seastore eligibility, source handoff
setup, target rendezvous collection and PG::merge_from().
- finish_merge_advance(): commit rctx and complete the role-specific
steps (source: complete_rctx, stop, register_merge_source; target:
handle_advance_map, handle_activate_map, complete_rctx).
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Add PG::merge_from to execute the merge of source PGs into a target PG.
This function builds a transaction to remove source-specifc metadata
objects and merge source collections into the target collection.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Add infrastructure so source PGs can be extracted from their birth
shard, moved to the target shard, and collected by the target PG before
merge proceeds.
Cross-shard safety: PGs are tied to their birth_shard for destruction.
register_merge_source() uses extract_pg() to detach the source,
seastar::foreign_ptr to hop cores, and
crimson::local_shared_foreign_ptr on the target so release routes
destruction back to the birth shard.
Synchronization: replace the per-shard ShardServices merge_info_t
registry (shared_promise waiters, ready_pgs staging, and cleanup hooks)
with merge state on the target PG itself. Source-side
register_merge_source() delivers PGs via PG::add_merge_source(); the
target waits in PG::collect_merge_sources(n) on a per-PG semaphore.
Duplicate source registrations are ignored. PG::stop() breaks the
semaphore so shutdown does not hang.
ShardServices::register_merge_source() and extract_pg() live in
shard_services; rendezvous types and methods live on PG.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
This function ensures that when a PG is being removed or
merged and it calls stop() - it will clear primary state, and
notify the Monitor to clear any pending merge flags.
It will also call client_request_orderer.clear_and_cancel() ensuring
all remaining client requests are properly completed. This is needed
for merging in particular since on_change() is never called for the merge epoch
(handle_advance_map is skipped after merge detection), so
clear_and_cancel() is never invoked on the source PG's orderer.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Update ShardServices to inherit from seastar::peering_sharded_service.
This allows the service to access its own sharded container directly
via container() rather than manually storing a reference to it.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
When a PG is in the pending merge state it is >= pg_num_pending and <
pg_num. When this happens, IO is paused and once the PG peers we notify
the mon that we are idle and safe to merge.
Use Gated for merge notify callbacks.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Verify that rpcclient lookupsids resolves domain user SIDs correctly
inside the smbd container, preventing regressions on /run bind mount
permissions that break smbd to winbindd communication.
Fixes: https://tracker.ceph.com/issues/77120
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
This changes unlink_bs_symlinks to use os.path.lexists instead
of os.path.exists. It can happen that devices get renumbered,
in that case, the OSD symlink still exists but its target device
is gone which means os.path.exists returns False, so the symlink
is never cleaned up and ceph-volume activate can fail later.
Fixes: https://tracker.ceph.com/issues/77295
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
The host side 'run' directory under /var/lib/ceph/<fsid>/<svc_dir> bind
mounted into SMB containers is created with mode 0770, preventing any
non owner processes from accessing unix domain sockets or named pipes
under /run. This breaks smbd to winbindd communication, causing SID to
name resolution failures for AD joined deployments. Therefore change
permissions to 0755 to match standard /run semantics.
Fixes: https://tracker.ceph.com/issues/77120
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
choose_oid fed ceph_str_hash_linux straight into % max_objs, and the
low bits of that hash are weak enough that similar object names keep
landing on the same shard. LC takes the hash mod HASH_PRIME first for
exactly this reason, do the same here.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Remove mention of overlapped roots failing to scale and added
example of adding overlapped roots.
Added mention of changes in PendingReleaseNotes
Fixes: https://tracker.ceph.com/issues/73892
Signed-off-by: Eric Zhang <emzhang@ibm.com>
No longer need to detect which roots are overlapping. Instead test PG budget allocation when roots overlap
Fixes: https://tracker.ceph.com/issues/73892
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Remove references to overlapped roots. Assign PG budget per root as sum(mon_target_pg_per_osd/# roots that contain OSD)
Fixes: https://tracker.ceph.com/issues/73892
Signed-off-by: Eric Zhang <emzhang@ibm.com>
RGWDeleteMultiObj spawns child coroutines via spawn_throttle to delete
objects in parallel. each child coroutine carries the connection strand
as its associated executor, serializing concurrent operations on shared
state like the response formatter and ops_log_entries.
but in multisite env, concurrent deletions in the same bucket shard
contend on async_cond in RGWDataChangesLog::add_entry(). when notify()
fires, waiting coroutines resume on the raw io_context executor
instead of their connection strand, breaking the serialization that prevents data
races in send_partial_response()
any_completion_handler doesn't support post(). so post() to the
default executor and dispatch to the associated executor from there
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
doc/README.md was a 0-byte file with no content; remove it.
doc/nvmeof/ha.md described the internal NVMeofGwMon/NVMeofGwMap
HA implementation (Paxos messaging, ANA state machines, blocklist
logic) -- a developer design document, not user-facing content.
User-facing NVMe-oF documentation already exists in doc/rbd/.
Move to doc/dev/nvmeof-ha-design.md where developer design docs live.
Fixes: https://tracker.ceph.com/issues/77208
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
Both securitylead.rst and workinggroup.rst existed in the tree but
were not referenced in the security/index.rst toctree, making them
invisible in the built documentation. Add RST document titles and
include both pages in the toctree.
Fixes: https://tracker.ceph.com/issues/77206
Signed-off-by: Emmanuel Ameh <eameh@contractor.linuxfoundation.org>
BackgroundProcess::run() blocks on blocking_background when
background_should_run() returns false. When the loop wakes up —
typically because arm_blocking_io_and_wake() kicked it — the loop
re-checks background_should_run(). If that still returns false
(e.g. the cleaner has no pending work), the loop goes straight back
to sleep.
But the wake may also have been triggered by a change in the space-
availability condition (should_block_io) — and there is no path that
re-evaluates blocked user IO when background_should_run is false. The
blocked IO has no future trigger to re-check, and stays stuck until
the next unrelated wake-up happens to come at the right moment, or
never.
Add a maybe_wake_blocked_io() call right after the wake. It's a no-op
when nothing is blocked, and unblocks IO that the space condition has
already cleared otherwise.
Observed in sustained random-write benches that wedged with IO blocked
indefinitely while the cleaner sat idle (should_run=false).
Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>
t2c.py translates teuthology benchmark YAML into CBT configuration for
run-cbt.sh. Switch input parsing from yaml.load() to yaml.safe_load()
so the module works with current PyYAML and avoids unsafe deserialization.
Add test_t2c.py covering get_cbt_tasks(), Translator.translate(), and
main(), and register it with make check via add_ceph_test. Document the
translator in doc/dev/crimson/index.rst and describe the
ceph-perf-pull-requests Jenkins workflow in
doc/dev/continuous-integration.rst.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
previously, we got the mono_clock::now() in OSD::start() and passed it
to PerShardState. this worked fine. but it was a little bit convoluted
-- we pass the startup_time all the way to PerShardState.
in this change, we just use call mono_clock::now() in the contructor
of PerShardState. simpler this way.
the startup_time has two consumers:
- the PGs hosted by the sharded_service use it as a reference for the
monotonic timestamp
- Heartbeat::send_heartbeats() uses it as for the mono_ping_stamp.
because, strictly speaking, we cannot gurantee that all PerShardState
sharded services share the identical startup timestamp, as they are
constructed on different shards. but this does not matter, as PGs
always use the hosting shard service for the referencing timestamp,
and OSD always uses the shard service on local shard for sending
heartbeats.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
OSD::startup_time was added in 91c0df81, in which was added to provide
a monotomic increasing timestamp representing the startup time. but
later, we decided to keep track of this timestamp in PerShardState.
but we didn't remove OSD::startup_time when adding
PerShardState::get_mnow().
in this change, we remove the unused OSD::startup_time.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
OSD::start() is a long, deeply nested .then() continuation chain. let's
rewrite it as a coroutine to make it readable. the chain was already
sequential and the one concurrent step keeps its when_all_succeed(), so
the rewrite preserves both ordering and concurrency.
start() runs once at boot, off the i/o path, so the small overhead of
co_await over a hand-rolled continuation chain is a fine price for the
readability.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
eaa2013 added a generic suppression that exposed the fact that the newer
compiler was inlining CephContext::do_command(). In order to avoid such
scenario in the future, and to keeo the behaviour constant between
daemons we'll disallow inlining.
Fixes: https://tracker.ceph.com/issues/76473
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
unittest_rgw_cors used GMock::Main without global_init, so dout() in
RGWCORSRule::matches() dereferenced a null g_ceph_context and segfaulted
on the first test. Link unit-main like unittest_rgw_compression.
Assisted-by: Cursor:composer-2.5-fast
Signed-off-by: ramin.najarbashi <ramin.najarbashi@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
get_multi_cors_method_flags() calls ceph::for_each_substr(), but
rgw_cors.h did not include str_list.h. The CORS unit test includes
only this header, so the build failed with an undeclared identifier.
Qualify the call and add the missing include.
Signed-off-by: ramin.najarbashi <ramin.najarbashi@gmail.com>
we might be able to reduce onode lookup stage by storing the prev
lookup already made when getting the obc.
according to to the stats I've colleceted onode lookup takes 40%
of total build time (which is ~20% of total txn time)
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Modified the logging system to show which OSD is executing in test logs,
making it easier to debug multi-OSD test scenarios.
Changes:
- Added Log::set_prefix_hook() to allow tests to customize log prefixes
- Modified Log::_flush() to use prefix_hook when set, replacing thread IDs
- Added EventLoop::get_log_prefix() to return 'osd.X' or 'harness' based on context
- Updated EventLoop to track current_executing_osd during event execution
- Simplified PGBackendTestFixture to use NoDoutPrefix directly
- Removed redundant 'shard X:' prefix from ShardDpp (info already in PG identifier)
- Removed verbose console output from EventLoop (now redundant with OSD prefixes)
- Converted EventLoop message scheduling to use dout at level 20
Benefits:
- Test logs now show 'osd.X' instead of thread IDs during event execution
- Shows 'harness' for test setup/teardown code outside events
- Message scheduling visible in logs at debug level 20
- Makes multi-OSD test scenarios much easier to follow and debug
- No changes to production logging code
AI assisted code, with careful human review and checking.
AI-assisted-by: IBM Bob IDE:Claude Sonnet (Anthropic)
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
This test was talking about testing invalid ops, however with the inclusion of sync reads in EC (https://github.com/ceph/ceph/pull/67079), it is valid to perform class reads in EC. In addition, work was done around illegal ops here: https://github.com/ceph/ceph/pull/66258 and the existance of TEST(ClsHello, BadMethods) in test_cls_hello.cc covers illegal ops in that PR leading me to think this is unneccisairy. Because of these reasons, I think its better this test is removed as it is incorrect and also not working.
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>
Add a num_replays counter to Transaction, incremented in
Cache::mark_transaction_conflicted whenever a transaction is marked
conflicted.
Only relevant for user-MUTATE path (do_transaction_no_callbacks).
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Select the first CORSRule whose AllowedOrigin, AllowedMethod, and (for
preflight) AllowedHeader all match the request. Previously host_name_rule
matched origin only, so a second rule with the same origin but different
methods was never used.
Add RGWCORSRule::matches() and RGWCORSConfiguration::match_rule(), wire
generate_cors_headers and RGWOptionsCORS to them (including global CORS), and
add unittest_rgw_cors.
Signed-off-by: ramin.najarbashi <ramin.najarbashi@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The test uses add-repo --release 17.2.6 to verify version-string repo
handling, but debian-17.2.6 only has focal and bullseye suites and
jammy packages weren't built until 17.2.7. This causes apt-get update
to fail with a 404 on ubuntu_22.04 nodes.
see: https://download.ceph.com/debian-17.2.6/dists/
This fix bumps the version to 17.2.7 which includes a jammy suite.
Fixes: https://tracker.ceph.com/issues/77130
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Do not trigger alert NVMeoFMultipleNamespacesOfRBDImage
for same pool/image name used in multiple nvmeof namespaces,
if they are in different rados namespaces (rados_namespace_name)
These are valid repeation of pool/image name:
- mypool/rados_ns1/myimage1
- mypool/rados_ns2/myimage1
- mypool/myimage1 (default rados namespace)
Fixes: https://tracker.ceph.com/issues/77128
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>
The checks that decide whether to call set_endpoint_unconnectable()
were comparing against -EIO, but the actual error codes returned on
connection failure changed after commit 37352a9074 ("rgw: change
rgw_http_error_to_errno default to -ERR_INTERNAL_ERROR").
- complete_request() -> wait() returns req_data->ret which is set to
rgw_http_error_to_errno(0) = -ERR_INTERNAL_ERROR when http_status is 0
(TCP connect failed). Fix the six call sites to check -ERR_INTERNAL_ERROR.
- forward_request() returns tl::unexpected(-ERR_SERVICE_UNAVAILABLE)
when http_status == 0 (no HTTP response received at all). Fix the two
forward/forward_iam conditionals to check -ERR_SERVICE_UNAVAILABLE.
Without this fix, connection failures are never detected in the sync
paths, so set_endpoint_unconnectable() is never called and the
IP failover / retry logic is effectively dead.
The .h coroutine paths were already fixed by dbb409e21b ("rgw: fix
endpoint detection in RGWRESTConn") but that commit missed all .cc
sync paths.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Replace raw string manipulation in RGWEndpoint with boost::urls::url.
URL path/query/host changes now use set_path(), set_query(), set_host()
instead of string concatenation.
Rename original_url to endpoint_url_lookup_id to clarify its role as a
health-tracking key for ResolvedEndpoint lookup in
set_endpoint_unconnectable().
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
the shared error code `ec` was not cleared before continuing
past a non-existent shard (ENOENT from get_meta). and if the last
shard returned ENOENT, `ec` retained that value and the
fired EIO even though all shards were cleanly removed.
also switch to sys::system_error(ec) to preserve the real errno
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
Fix the bug where an empty lifecycle filter was stored
identically to one with an empty old-format prefix rule--
which caused the recovered rules to be formed incorrectly
as prefix rules.
Found by Kyle Bader and Tekton workflow. Updated to not declare
virtual methods in LCFilter.
Fixes: https://tracker.ceph.com/issues/76806
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Sub-ms ops rounded to zero under the old millisecond cast, making
the average meaningless for fast NVMe workloads. sample_sum is now
in microseconds; divide by 1000 for ms average.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Initialize lat_hist_bounds_us bucket boundaries (0.25ms–20ms in 10
log-ish steps) for the op_lat Seastar histogram in register_metrics(),
and populate them in add_latency_sample(). Previously op_lat only
tracked sample_count and sample_sum, giving an average but no
percentile visibility.
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
The DencoderPlugin destructor calls dlclose() on each loaded plugin .so
before main() returns. When LeakSanitizer's atexit-registered leak
check then runs, the plugin .so is no longer mapped in the process
address space, and any leak whose allocation stack passes through
plugin code is reported with "<unknown module>" — useless for
investigation.
This is not an ASan bug or a missing -g flag; it is the unavoidable
ordering of dlclose-before-leak-check. Skipping dlclose under
__SANITIZE_ADDRESS__ leaves the plugins mapped until process exit,
restoring symbolisation. The functional cost is zero: the kernel will
unmap the .so segments when the process exits anyway.
Without this, a developer triaging dencoder leaks under ASan has no
visibility into which plugin or which static initialiser is
responsible. With this, the leak stacks resolve to real symbols and
real bugs become triagable.
This commit does not change leak-detection results in non-sanitiser
builds and is a no-op on FreeBSD (which already skipped dlclose).
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
The cloud-tier transfer only treated 503 (-EBUSY) as retryable. A 500 from the
remote surfaced as -ERR_INTERNAL_ERROR and failed the object outright, but S3
backends use 500 as a transient "try again" signal. Retry it the same way we
retry -EBUSY. The helper is no longer EBUSY-specific, so rename it to
retry_on_transient_error; the restore path uses it too and gets the same
behaviour.
Even then a 500 never reached the retry: cloud_tier_create_bucket mapped an
error response to -EIO and an error with no body to 0 (success). Return the
real errno for transient failures, and stop treating an empty-body error as a
successful create.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
In this variant of get_primary_shard, the function attempts to undo
the primaryfirst ordering. This is incorrect, because pg_to_acting_osds
has already applied this.
The fix is simply to remove this code.
This is unlikely to be a significant problem, as this function is not
used in the critical locations. We are building new test harnesses
which will rely on this behaviour.
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Expose the confval information for the confval "rgw_rest_conn_connect_to_resolved_ips"
and "rgw_rest_conn_ip_fail_timeout_secs" so that they can be seen in the
"Ceph Object Gateway Config Reference" as these are meant to be client-facing configs.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Rename endpoint_round_robin_counter to endpoint_rr_index and
endpoint_ips_round_robin_counter to ip_rr_index for shorter,
cleaner variable names while maintaining clarity.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Adds a new admin socket command to dump zone connection details
including endpoints, resolved IPs, and health status. Useful for
debugging multisite connectivity issues.
Usage: ceph daemon <radosgw.asok> zone connections
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Introduce a new radosgw option 'rgw_rest_conn_ip_fail_timeout_secs' to
be able to set the constant CONN_STATUS_EXPIRE_SECS dynamically.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Previously, when a connection to a zone endpoint failed, the entire
endpoint was marked as unavailable for a timeout period. Since we now
resolve endpoints to all their IP addresses (via DNS A/AAAA records),
we can be more granular: track failures at the individual IP level.
Introduce ResolvedIP struct that pairs each IP's connect_to string
with its own failure timestamp. When selecting an IP for a request,
round-robin skips IPs that have recently failed, allowing traffic to
continue flowing to healthy nodes even when some are down.
An endpoint-level last_failure_time is maintained as a fast-path
optimization to avoid scanning all IPs when none have failed recently.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Previously RGWRESTConn stored endpoints in two data structures:
- endpoint_urls: vector<string> for ordered round-robin iteration
- resolved_endpoints: unordered_map<string, ResolvedEndpoint> for lookup
This was redundant since the URL was stored in both places.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Add ostream operator<< to RGWEndpoint struct for convenient logging of
endpoint details (url, original_url when different, and connect_to).
Update log statements across rgw_http_client.cc, rgw_rest_client.cc,
and rgw_rest_conn.cc to use the new operator for cleaner, more
consistent output.
Add unittest_rgw_http_client to test RGWEndpoint functionality.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Refactor endpoint tracking by adding original_url to RGWEndpoint struct
instead of maintaining a separate endpoint_orig member in RGWHTTPClient.
This simplifies the code by having each endpoint self-track its original
URL, which is needed for connection status lookups after URL modifications.
No functional changes intended.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Refactor RGWRESTConn to eliminate the separate endpoints_status map by
moving the connection status (std::atomic<ceph::real_time>) directly
into the ResolvedEndpoint struct. This reduces redundancy and simplifies
endpoint state management.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
If endpoint.connect_to is non-empty, populate and attach a curl slist to
CURLOPT_CONNECT_TO for this request.
Ensure the slist is freed with the request lifetime to avoid leaks across
retries/requests.
Fixes: https://tracker.ceph.com/issues/74677
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Add logic to select an IP for a given endpoint URL (RR over resolved addresses)
and build a host:port:ip:port mapping.
Store the mapping in the RGWEndpoint so the HTTP layer can apply it per request.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
When rgw_resolve_endpoints_into_all_addresses=true, parse each configured
endpoint URL, extract host/port, and resolve the host into all IP addresses.
Store resolution results (including a round-robin index) alongside the original
endpoint URL for later selection.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Introduce an advanced boolean config option (default: false) to enable resolving
multisite endpoint hostnames into all A/AAAA records.
When enabled, RGW can distribute outgoing inter-zone traffic across DNS-provided
backends even without an external load balancer.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
This commit is meant to be non-functional.
Replace the "URL as plain string" plumbing with an RGWEndpoint value type
that carries:
- the URL string, and
- optional per-request connect_to data for libcurl,
- it also encapsulates related functions/methods
This commit is intended to be non-functional: behavior should remain unchanged
until callers populate connect_to.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
With Protocols including NFSv3, Ganesha 9.x on Rocky 10 needs a running
portmapper to register NFSv3 on TCP. Start rpcbind from the container
entrypoint only when ganesha.conf enables v3; skip for NFSv4-only configs.
Fixes: https://tracker.ceph.com/issues/76981
Signed-off-by: Kobi Ginon <kginon@redhat.com>
`ceph cephadm osd activate` calls `deploy_osd_daemons_for_existing_osds`
with a synthetic DriveGroupSpec where service_id=''. Commit fbe3a053
introduced an unconditional osdspec_affinity filter:
if osd['tags']['ceph.osdspec_affinity'] != spec.service_id:
continue
The fix is to only enforce the affinity check when spec.service_id is
non-empty. An empty service_id means the caller is osd activate, which
should adopt any existing OSD regardless of its affinity tag.
Fixes: https://tracker.ceph.com/issues/76979
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Introduce a new read-only command, nvme-gw show-all, which
iterates over all configured pools and groups and runs
nvme-gw show for each, simplifying gateway inspection.
fixes https://tracker.ceph.com/issues/74854
Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>
test coverage for `rgw crypt sse algorithm: aes-256-gcm` was recently
added to the multisite suite, doubling the number of jobs from 2 to 4
instead of duplicating the multisite jobs for both cbc and gcm, select
a random algorithm for each of the 2 existing jobs
Signed-off-by: Casey Bodley <cbodley@redhat.com>
change the hander from EmptyResponder() to ErrorResponseHandler() to
emit the warning making use of the command's status channel.
Fixes: https://tracker.ceph.com/issues/61482
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
The previous patch only injected hostname into the metadata map for OSDs
that were placed under a CRUSH host bucket. For mds/mgr/mon daemons and
for OSDs absent from CRUSH, the original 'state->hostname = ...' line
was removed without a replacement, so update_metadata received a map
with no 'hostname' key. DaemonState::set_metadata only updates
state->hostname when the map carries it, so a daemon restarting on a
different host would be re-inserted in by_server under its stale
hostname.
Always seed m["hostname"] from the metadata-reported hostname, then
prefer the CRUSH host when available. Apply the same shape to the
new-daemon branch so set_metadata uniformly drives state->hostname,
removing the redundant direct assignment.
Reported-by: code review on PR #66757
Signed-off-by: senol.colak <senol.colak@sap.com>
test_container_engine mocks find_executable to keep command_check_host
off the real host, but command_check_host looks up 'lvcreate' (and
'systemctl') through find_program, which isn't mocked. So the test
actually needs lvcreate installed and only passes where it is.
Mock find_program too so the test doesn't depend on host binaries.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
await_idle_v() is wrapped in assert(), so with NDEBUG it is never
called and the test reads a default-constructed ack, racing the agent
thread. Use EXPECT_TRUE() like the rest of the file.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
This commit upgrades the rocksdb submodule from v7.9.2 to v7.10.2 to
address CVE-2022-23476, a security vulnerability in the Nokogiri gem
used by RocksDB's documentation build system.
CVE-2022-23476 Details:
- Affects: RocksDB versions < 7.10.2
- Issue: Unchecked return value from xmlTextReaderExpand in Nokogiri
- Fix commit: 6648dec0a3eee0a329af8342038ce099baa55122
- Fixed in: Nokogiri 1.13.10, included in RocksDB v7.10.2
The vulnerability was fixed in the upstream facebook/rocksdb repository
on December 8, 2022, and included in the v7.10.2 release. The v7.9.2
version currently used by Ceph does not contain this fix, as the 7.9.x
branch diverged before the security patch was applied.
Analysis confirmed that:
- Current Ceph RocksDB: v7.9.2 (commit 24ea35870fe9b3ba15285ec8746ba97ed5d67ff3)
- CVE fix present in v7.9.2: NO
- First version with fix: v7.10.2 (commit 9107c1059b4656513ae0d51e8976e4f69f59a9c3)
References:
- CVE-2022-23476: https://nvd.nist.gov/vuln/detail/CVE-2022-23476
- Nokogiri Advisory: GHSA-qv4q-mr5r-qprj
- Fix commit: facebook/rocksdb@6648dec
also, in this change, we adapt BinnedLRUCache to CacheItemHelper API.
rocksdb 7.10 reshaped `rocksdb::Cache`, so we have to update our
overrides accordingly to address the build failure. the check for
RocksDB version is dropped, as mainstream distros ship RocksDB >=
7.10.2 at the time of writing:
- alpine 3.23: 10.9.1
- fedora 44: 10.2.1
- debian stable: 9.10.0
- ubuntu nobel: 8.9.1
- ubuntu resolute: 9.11.2
Signed-off-by: Justin Caratzas <jcaratza@ibm.com>
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
op_ret is a member variable of RGWDeleteMultiObj shared between
concurrent callers of handle_individual_object(). because that coroutine
may suspend between writing to op_ret and later reading it back for
send_partial_response(), it's likely that other callers have modified it
in the meantime
store this error code in a local variable instead, so each coroutine has
its own consistent copy of it
Fixes: https://tracker.ceph.com/issues/76960
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Almost all callers pass a C string (in temporary Feature instances),
only `CompatSetHandler::handle()` passes a `std::string` reference.
Allocating a new `std::string` is useless overhead. This patch
reduces the binary size by 21 kB.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
In containerized deployments (e.g. Rook), OSD daemons report the
pod/container hostname. The CRUSH map records the physical host.
This mismatch causes 'ceph device ls' to show pod names instead of
physical hosts, breaking host-based device correlation.
Fix by deriving the CRUSH host bucket name for each OSD and substituting
it as the canonical hostname in all three metadata paths:
- MetadataUpdate::finish (async mon metadata fetch, primary path)
- Mgr::load_all_metadata OSD bulk-load (startup/failover)
- DaemonServer::fetch_missing_metadata (via MetadataUpdate)
The override is a no-op when the OSD is absent from CRUSH or when the
OSD map is not yet available.
Fixes: https://tracker.ceph.com/issues/73080
Signed-off-by: senol.colak <senol.colak@sap.com>
Append retcode and its string representation to the log line to facilitate the
diagnosis of sync issues.
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
The bucket-instance level sync (RGWDataSyncSingleEntryCR) excludes EBUSY and
EAGAIN from the sync error log as these are meant to be transient errors that
resolve on retry without an admin intervention.
Also, add EBUSY and EAGAIN to ignore_sync_error(), which is used by the
per-object level sync in RGWSyncObjectCR, to match the bucket-instance
level behavior.
Fixes: https://tracker.ceph.com/issues/76950
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
All parameters are C strings (either C literals, except for
MgrMonitor::update_from_paxos()) and there's no point in copying all
of them onto the heap during startup.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Add stub-level information for the documentation of ceph-exporter. Explicit configuration instructions are still missing from this documentation, and should in the future be supplied.
Signed-off-by: Zac Dover <zac.dover@clyso.com>
Expose HEAD Object operation metrics in the same format as existing RGW op metrics so operators can track HEAD request volume and latency at gateway, user, and bucket scope.
Previously, HEAD Object requests were silently counted under get_obj_ops because RGWGetObj handles both GET and HEAD. The HEAD-specific early-return path (get_data == false) had no counter instrumentation.
Fixes: ISCE-3042
Signed-off-by: Harsimran Singh <harsimransingh@ibm.com>
Found by Opus 4.6:
In get_obj() (lines 417-446), when obj_iter ==
list_results.objs.end() and the listing is truncated, the code calls
fetch() to get the next batch. fetch() sets obj_iter =
list_results.objs.begin(). But if the new fetch returns zero objects
(possible if objects were deleted between listing batches, or due to
filtering/race conditions), then begin() == end() and the code falls
through to line 437:
if (obj_iter->key.name == pre_obj.key.name) {
This dereferences end() — undefined behavior / crash. The
return-value check at line 445 (return obj_iter !=
list_results.objs.end()) comes too late; the dereference already
happened.
Fixes: https://tracker.ceph.com/issues/76790
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Assisted-by: Claude Code, Opus 4.6 1M
We were seeing failures in TestShellOpts again
and again due the (newline added within the output)
output structure change with the cmd2 releases.
Now, we directly extract the final output of 'set editor'
command and use it, instead of traversing on complete output
through hardcoded indexes.This fix is better than earlier
hardcoded splitting/striping via indexes.
Fixes: https://tracker.ceph.com/issues/71795
Signed-off-by: Neeraj Pratap Singh <Neeraj.Pratap.Singh1@ibm.com>
The test reused the "tentacle" jq filter while validating
"nvmeof_beacon_diff", causing the comparison to fail. Also fix
the umbrella feature validation and update the expected
persistent feature count.
Fixes: https://tracker.ceph.com/issues/76472
Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>
Push the -EBUSY early-return into init_multipart and the bucket
exists/create path so a 503 stops printing a bogus ERROR before the
outer retry catches it. Wrap abort_multipart in retry_on_busy too,
since a stuck 503 there leaves an orphan multipart on the remote.
retry_on_busy also logs when it gives up so a silent exhaustion
doesn't get lost.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Fixes: https://tracker.ceph.com/issues/62195
Extend cephadm list-networks (and the cephadm mgr module options that pass
flags through to it) so hosts can expose loopback and BGP-derived routes
when operators need them, while keeping the default conservative for
orchestrator network discovery.
- Add cephadm flags --allow-lo-routes and --allow-bgp-routes (after the
list-networks subcommand). Wire mgr options allow_lo_routes and
allow_bgp_routes to the remote cephadm invocation.
- IPv4: build from ``ip route ls`` (scope link + src). When allow_lo_routes
is true, merge extra lines from the same output via _parse_ipv4_lo_route
(lo /32 without src, host-scoped routes including dummy-style /32 from
76229). Always omit prefixes wholly inside 127.0.0.0/8. When
allow_lo_routes is false, skip any route whose device is lo at parse time.
When allow_bgp_routes is true, merge ``ip route ls proto bgp`` (ECMP
nexthops and lo BGP /32), again respecting allow_lo_routes for lo.
- IPv6: build from ``ip -6 route ls`` and ``ip -6 addr ls``; omit ::1/128.
When allow_lo_routes is false, skip routes on lo (including global /128 on
lo). When allow_bgp_routes is true, merge ``ip -6 route ls proto bgp`` in
the same way as IPv4, including lo BGP /128 when allow_lo_routes is true.
Signed-off-by: Kobi Ginon <kginon@redhat.com>
`cls_timeindex_entry` defines `encode()` and `decode()` inline in
`cls_timeindex_types.h`, but `dump()` and `generate_test_instances()` are
defined out of line in `cls_timeindex_types.cc`. That `.cc` was compiled
only into `cls_timeindex_client`, not into the `cls_timeindex objclass`
shared library. At `-O2` the optimizer inlines and dead-code-eliminates
these unused `dump()` paths, so `cls_timeindex.so` ends up with no
reference to the out-of-line symbols and loads fine. At `-O0` nothing is
inlined or eliminated: the header-inline `dump()` is emitted and leaves
an undefined reference to `cls_timeindex_entry::dump()` in
`cls_timeindex.so`, so `dlopen()` of the objclass fails at load time.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
We were aborting the lifecycle transition or restore on the first 503
SlowDown back from the remote. Wrap both paths in a small retry helper
that backs off and tries again on -EBUSY, using the caller's yield
context to sleep.
Tunable via rgw_cloud_tier_retry_limit (8), rgw_cloud_tier_retry_delay_ms
(500), and rgw_cloud_tier_retry_max_ms (30000). On the streamed restore
GET, RGWRadosPutObj::handle_headers rejects 5xx so the response body
doesn't get streamed into the put-processor before we know to retry.
This also affects the multi-zone fetch path that uses the same
put-processor; 503 there now returns -EBUSY cleanly instead of
streaming the error body.
Fixes: https://tracker.ceph.com/issues/76406
Co-authored-by: Kevin Lott <klott@backblaze.com>
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Since we can't map shards between generations, if we get a sync entry
for a future shard, synthesize an entry on the current generation for
all shards.
We use the bucket cache to keep track of when we've done this so we
don't give ourselves unbounded work when we get bursts of future
generation shards.
Fixes: https://tracker.ceph.com/issues/75786
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
The GenCache will keep track of per-bucket sync information, like
if we have had to synthesize a change-per-shard as a result of hitting
the wrong generation.
Templatize tests.
Fixes: https://tracker.ceph.com/issues/75786
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
NFS service specs should not allow placement.count_per_host.
Fixes: https://tracker.ceph.com/issues/76579
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Although not explicitly documented, the RGW ops log is generally
formatted with one entry per line. This makes it work well with log
shipping/ingestion services (many of which default to treating each line
as a separate entry) and in particular works well with log servers that
index based on the available JSON fields.
The current implementation separates the log call from the resulting
disk IO: Logs are written to a buffer and a separate thread flushes them
to disk (possibly in batches). The current code appends a newline only
at the end of the batch being flushed to disk and in many cases when
under load this means that several log entries are concatenated onto a
single line, which complicates attempts to process those logs.
This PR separates the addition of a new line from the flush to disk,
appending a newline after every log entry but still only flushing at the
end of the batch to avoid additional IO overhead.
Fixes: https://tracker.ceph.com/issues/76566
Signed-off-by: Jacques Heunis <jheunis@bloomberg.net>
Change default ingress mode from haproxy-standard to haproxy-protocol
to preserve client IP addresses for proper IP-level export restrictions
in NFS Ganesha.
Fixes: https://tracker.ceph.com/issues/74260
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>
Since we modified crushtool cli commands, we need to also update
its test with new flag: --show-retry-exhaustion
and also the modified --show-choose-tries option
Also added /src/script/run-cli-tests.sh to run the cram test
easily without having the config headache
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
source bucket deletion triggers a new bucket instance creation
at archive zone. record its mtime and display in bucket stats.
store it as an attr RGW_ATTR_ARCHIVE_INSTANCE_MTIME
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
Add script to test for CRUSH retry exhaustion in stretch mode with
2 datacenters. Tests unbiased stretch rules by running multiple
iterations of PG mappings and checking for collisions that exceed
the 50-try limit.
Also add --show-retry-exhaustion flag to crushtool to detect and
report when CRUSH mapping hits the maximum retry limit.
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
This PR wires the `force_delete_data` already existing flag in the
binary through cephadm’s daemon and service removal paths, so that
commands such as `ceph orch rm service` or equivalent daemon removal
can explicitly ask for data deletion instead of the default "move
under <fsid>/removed/" for daemons such as Prometheus, osd and mon.
Fixes: https://tracker.ceph.com/issues/74058
Signed-off-by: Kobi Ginon <kginon@redhat.com>
Currently tests wait for 10 seconds for clone progress bar to appear and
it usually takes 8 seconds to find it. In case of the host where test is
being run is slow, the progress bar can take more time to appear and the
test can fail unnecessarily. To avoid this, increase the waiting
duration.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
There's a race condition between cloning operation and clone progress
reporter thread to open the subvolume, former does so to copy/clone
files and latter to get details of the source and destination subvolume.
A low number of files in the snapshot gives clone progress reporter
thread less or no chance to fetch and process the details needed to
display the progress bar. This causes tests to fails when progress bar
isn't displayed in the output of "ceph status" output.
Increasing the number of files will give clone progress reporter more
time to display the progress bar and therefore these unnecessary test
failures wlll reduce/stop.
This failure wasn't seen earlier because Python code was being used to
copy files which was comparatively slow (compared to C++ code that was
merged to copy files as a part of fscrypt work) which allowed enough
time to clone progress reporter thread to process stats and display
clone progress bar.
Fixes: https://tracker.ceph.com/issues/74082
Signed-off-by: Rishabh Dave <ridave@redhat.com>
guard bluefs->expand_device for online-only
fixes offline expansion by reopening db from r/o to r/w
fixes int64_t/uint64_t type mismatch for old_size
Signed-off-by: Joshua Blanch <joshua.blanch@clyso.com>
Refactors device expansion to use a expand_devices()
that handles both online (admin socket) and offline (ceph-bluestore-tool)
use cases.
Signed-off-by: Joshua Blanch <joshua.blanch@clyso.com>
expands bluestore and bluefs while OSD is running.
Instead of recreating allocators, expands allocator and FM
in-place.
ceph tell osd.X bluestore bluefs-bdev-expand
Signed-off-by: Joshua Blanch <joshua.blanch@clyso.com>
Implements device expansion support for bitmap-based allocators
BitmapAllocator and HybridAllocator. Bitmap allocators uses allocated
fixed-size bit vectors (L0, L1, L2) and requires vector resizing
and metadata updates when underlying device expands.
Some different cases that are handled:
- padding reclamation when alignment boundaries change
- Sub-granularity expansions (L2 doesn't grow, only L2 bits update)
To be used such as expand() followed by init_add_free() to ensure
available space accounting happens once.
Signed-off-by: Joshua Blanch <joshua.blanch@clyso.com>
Setting maintenance mode show below warning,which appears incorrect
or ambiguous.
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes}have
{NOUP,NODOWN,NOIN,NOOUT} flags set
Change warning as shown bellow.
[WRN] OSD_FLAGS: 1 OSDs or CRUSH nodes/device-classes have
one or more of these flags set: NOUP, NODOWN, NOIN, NOOUT
Fixes: https://tracker.ceph.com/issues/71716
Signed-off-by: Vivek Maurya <vivek.maurya@ibm.com>
Part 3 of 6 to make backporting easier. Depends on part 1.
Use the the ref role for all remaining links in doc/radosgw/ with the
exception of config-ref.rst which will depend on changes to rgw.yaml.in.
The "external link definitions" syntax being removed is intended for
linking to external websites and not for intra-docs links. Validity of
ref links will be checked during the docs build process.
Add labels for links targets if necessary.
Remove unused external link definitions in the modified files.
Use confval instead of literal text for 2 configuration keys in
vault.rst.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
Extend `ceph mgr stat` to include basic information about standby
manager daemons in a new `standbys` field. This improves visibility
into standby managers without requiring `ceph mgr dump`.
Fixes: https://tracker.ceph.com/issues/75464
Signed-off-by: Indira Sawant <indira.sawant@ibm.com>
Specifying the quota or resize for a subvolume requires the value in bytes.
This value can now be accepted as <num><unit>. Also allows floating point numbers.
Fixes: https://tracker.ceph.com/issues/62673
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
Part 5 of 6 to make backporting easier. Depends on part 1.
Use the the ref role for all remaining links in doc/radosgw/ with the
exception of config-ref.rst which will depend on changes to rgw.yaml.in.
The "external link definitions" syntax being removed is intended for
linking to external websites and not for intra-docs links. Validity of
ref links will be checked during the docs build process.
Add labels for links targets if necessary.
Remove unused external link definitions in the modified files.
Use confval instead of literal text for 2 configuration keys in
vault.rst.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
2026-02-13 12:44:48 +07:00
1957 changed files with 161025 additions and 43665 deletions
body:`❌ **Audit Test Mode Failed**\n\nBranch \`${testBranch}\` was not found in \`${context.repo.owner}/${context.repo.repo}\`.\n\n[View workflow run](${runUrl})`
});
core.setOutput('run_audit', 'false');
return;
}
try {
core.info('[Router] Posting test mode activation comment...');
body:`🧪 **Audit Test Mode Activated** by @${actor}.\n\nExecuting audit using tooling checked out from branch \`${testBranch}\`. *This run will not affect required PR commit statuses or state labels.*\n\n[View workflow run](${runUrl})`
});
core.info('[Router] Successfully posted test mode activation comment.');
}catch (e) {
core.error(`[Router] Failed to post test mode comment:${e.message}`);
}
core.setOutput('checkout_ref', testBranch);
await triggerAuditRun(`Running audit using branch ${testBranch}...`, `Audit Test Mode (${testBranch})`);
}else {
core.error(`User @${actor} is not authorized to invoke test branches.`);
core.setOutput('run_audit', 'false');
}
}else {
core.info('[Router] Comment is not a recognized /audit command. Skipping.');
core.setOutput('run_audit', 'false');
}
core.info('[Router] Comment is not an audit command. Skipping.');
body:'⚠️ **Audit Override Removed**\n\nThis PR was reopened, so the previous `releng-audit-override` has been removed.'
body:`⚠️ **Audit Override Removed**\n\nNew commits were pushed to this PR, so the previous override has been removed.\n\n[View workflow run](${runUrl})`
* CephFS: The ``client_force_lazyio`` configuration option is now correctly marked
as not supporting runtime updates. Previously, the configuration schema indicated
this option could be changed at runtime, but changes had no effect on opened file
handlers in ceph-fuse or libcephfs clients because there is no logic to propagate
changes to open file handles.
* RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based
on its OSDs, with OSDs shared across multiple roots contributing proportionally
less to each root's allocation.
* CephFS: CephFS snapshots metadata is now mutable. It is now possible to add,
update and remove existing key-value pairs that are part of a snapshot
metadata via libcephfs API ceph_do_snap_md_op().
* CephFS: MDS dmclock support for subvolume QoS. With this feature, MDS assigns QoS
to throttle client metadata requests (e.g., create, mkdir, lookup, and so on) to
subvolumes, where each subvolume QoS is shared among multiple client sessions at
that time.
* librados/neorados: The C++ APIs for executing Ceph Class (CLS) methods
have undergone a breaking change to enforce compile-time type safety, replacing
legacy string-based parameters with strongly-typed ClsMethod structs. C++
@ -17,6 +36,8 @@
- The `rgw default data log backing` option is removed and it is no longer
possible to create clusters with an omap based datalog.
- `radosgw-admin datalog type` will only accept `--log_type=fifo`.
* RGW: iam:RemoveClientIdFromOIDCProvider is now a recognized action for
policy, corrected from a typo of iam:RemoveCientIdFromOIDCProvider
* MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd``
have been increased from 100 and 250 to 200 and 500, respectively. These values
@ -47,6 +68,12 @@
If the default provider is required when also using custom providers,
it must be explicitly loaded in the configuration file or code (see https://github.com/openssl/openssl/blob/master/README-PROVIDERS.md).
* RGW: Fixed bucket notification events so the 'x_amz_request_id' in NotificationEvent now matches the 'x_amz_request_id' returned by the corresponding S3 operation.
* RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything.
- Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size.
* RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128.
Benchmarks showed that the lower thread count improves throughput and reduces latency
on NVMe and HDD based environments. Users with specific workload requirements can
tune this value; see also ``rgw_max_concurrent_requests``.
* DASHBOARD: A new Overview landing page provides an at-a-glance cluster summary
@ -165,6 +192,35 @@
``ceph fs snapshot mirror daemon status`` now shows the remote cluster's
monitor addresses and cluster ID for each configured peer, making it easier
to verify peer connectivity and troubleshoot mirroring issues.
@ -41,8 +41,7 @@ Make sure X (and, ideally, X+1) is defined:
Github Actions
~~~~~~~~~~~~~~
- [ ] .github/workflows/redmine-upkeep.yml add release branch to pull_request_target trigger
- [ ] .github/workflows/releng-audit.yml add release branch to pull_request_target trigger
- [ ] `.github/workflows/*.yml' add release branch to pull_request_target trigger
Scripts
~~~~~~~
@ -137,6 +136,8 @@ After dev freeze
- [ ] open the Branch Protection settings of the Ceph repo. Duplicate settings to the new release branch.
- [ ] create vX.3.0 annotated tag on ``main`` so upgrades from new release to main are not wrongly considered downgrades.
- [ ] remove ``doc/releases/*.rst``. This should leave behind ``doc/releases/releases.yml`` which is used for doc building purposes. See also commit 33d63c3 ("doc: remove release notes for release branch") for details.
- [ ] remove ``.github/workflows``. ``main`` is authoritative for workflows; avoid possible confusion/conflicts.
- [ ] remove ``.github/pull_request_template.md``. Backports do not use it.
- [ ] cherry-pick 8cf9ad62949516666ad0f2c0bb7726ef68e4d666 ("doc: add releases links to toc"). There will be trivial conflicts.
- [ ] add redirect for new major release at `RTD <https://readthedocs.org/dashboard/ceph/redirects/>`_.
- [ ] add release name to redmine (using https://tracker.ceph.com/custom_fields/16/edit)
The ceph-release-pipeline parent job has three stages.
If your build fails during the "create ceph release tag" stage, troubleshoot the child ceph-tag job and re-run the parent ceph-release-pipeline job with the same parameters.
----
If your build fails during the "package build" stage, troubleshoot the child ceph-dev-pipeline job. If only one variant failed to build (e.g., centos9 arm64), start a new ceph-release-pipeline job specifying::
DISTROS=centos9
ARCHS=arm64
TAG=false <-- VERY IMPORTANT
This will leave the version commit and previously-created tag intact in ceph-releases.git. You will want the subsequent ceph-dev-pipeline job to reuse that SHA/tag.
Once all of your variants are successfully built, you will have to manually run the ceph-tag job. For example,::
BRANCH=tentacle
TAG=true
TAG_PHASE=push
VERSION=20.2.0
RELEASE_TYPE=STABLE
Then proceed with the normal release process.
----
If your build fails during the "push ceph release tag" stage, troubleshoot the child ceph-tag job and re-run **just** the ceph-tag job again manually. Do not re-run ceph-release-pipeline.
3. Release Notes
================
@ -178,18 +205,57 @@ See `the Ceph Tracker wiki page that explains how to write the release notes <ht
Reverses changes done by the new NCB code (either through ceph restart or when running allocmap command) and restores RocksDB B Column-Family (allocator-map).
:command:`recovery-compare`
Runs legacy onode recovery and multithread onode recovery. Prints timings and compares results.
Update the client compatibility mode for an SMB cluster. This setting controls whether client-specific SMB features and optimizations are enabled.
The client compatibility mode determines how the Samba server is configured to optimize for specific client types:
- ``default``: Standard SMB behavior without client-specific optimizations. This is the default mode and provides broad compatibility with all SMB clients.
- ``macos``: Enable macOS-specific features including the Samba's fruit VFS module for proper handling of macOS metadata and optimized settings for macOS clients.
When ``macos`` mode is enabled, the following features are automatically configured:
- Fruit VFS module for proper handling of macOS-specific file attributes
- Streams_xattr VFS module for extended attribute support
The ``macos`` compatibility mode is recommended when the primary clients accessing the SMB shares are macOS systems. Otherwise default mode is recommended.
Remove Cluster
++++++++++++++
@ -321,6 +382,51 @@ Create a read-only share at a custom path in the CephFS volume:
indicator that the user is aware that using this option can easily break
things in ways that the Ceph team can not help with. This special key will
automatically be removed from the list of options passed to Samba.
client_compat
Optional. One of ``default`` or ``macos``. Controls client-specific SMB
features and optimizations. The ``default`` mode provides standard SMB
behavior with broad compatibility. The ``macos`` mode enables macOS-specific
features including Samba's fruit VFS module for proper handling of macOS and
optimized settings for macOS clients. If unspecified, ``default`` is assumed.
..warning::
Setting the ``clustering`` option allows an administrator to choose exactly
@ -977,6 +1089,20 @@ The following is an example of a cluster configured for standalone operation:
hosts:
- node6.mycluster.sink.test
The following is an example of a cluster optimized for macOS clients:
..code-block:: yaml
resource_type: ceph.smb.cluster
cluster_id: macshare
auth_mode: user
client_compat: macos
user_group_settings:
- source_type: resource
ref: ug1
placement:
count: 1
An example cluster resource with intent to remove:
..code-block:: yaml
@ -1018,7 +1144,8 @@ max_connections
connections to a specific share. The default value is 0 and it indicates
that there is no limit on the number of connections
cephfs
Required object. Fields:
Object. Configures CephFS-backed storage for the share. Either a ``cephfs``
or ``rgw`` object must be specified, but not both. Fields:
volume
Required string. Name of the cephfs volume to use
@ -1088,6 +1215,22 @@ cephfs
name
String. A value indicating what FSCrypt key to fetch. The specific
value of the name depends on the scope being used.
rgw
Object. Configures RADOS Gateway (RGW) backed storage for the share. Either
a ``cephfs`` or ``rgw`` object must be specified, but not both. This allows
S3-compatible object storage to be accessed via the SMB protocol. Fields:
bucket
Required string. The name of the RGW bucket to be shared.
user_id
Optional string. The RGW user ID that owns the bucket. If not provided,
the system will automatically determine the bucket owner and fetch the
necessary credentials.
credential_ref
Optional string. The ``rgw_credential_id`` value of a
``ceph.smb.rgw.credential`` resource that contains RGW access and
secret key values needed to use the given bucket.
restrict_access
Optional boolean, defaulting to false. If true the share will only permit
access by users explicitly listed in ``login_control``.
@ -1135,7 +1278,32 @@ custom_smb_share_options
things in ways that the Ceph team can not help with. This special key will
automatically be removed from the list of options passed to Samba.
The following is an example of a share with QoS settings including burst
The following is an example of an RGW-backed share with minimal configuration
(credentials auto-fetched):
..code-block:: yaml
resource_type: ceph.smb.share
cluster_id: tango
share_id: s3share
name: "S3 Storage"
rgw:
bucket: my-bucket
Another example of an RGW-backed share with explicit user_id:
..code-block:: yaml
resource_type: ceph.smb.share
cluster_id: tango
share_id: s3share
name: "S3 Storage"
rgw:
bucket: my-bucket
user_id: s3user
The following is an example of a CephFS share with QoS settings including burst
multipliers and human-readable bandwidth limits:
..code-block:: yaml
@ -1285,6 +1453,43 @@ Example:
groups: []
RGW Credential Resource
------------------------
An RGW credential resource stores RADOS Gateway (RGW) access credentials that can be used by RGW-backed shares to authenticate with the object storage system.
A RGW credential resource supports the following fields:
resource_type
A literal string ``ceph.smb.rgw.credential``
rgw_credential_id
A short string identifying the RGW credential resource.
intent
One of ``present`` or ``removed``. If not provided, ``present`` is assumed.
If ``removed`` all following fields are optional
user_id
Required string. The RGW user ID that owns the credentials
access_key_id
Required string. The RGW access key for authentication
secret_access_key
Required string. The RGW secret key for authentication
linked_to_cluster:
Optional. A string containing a cluster ID. If set, the resource may only
be used with the linked cluster and will automatically be removed when the
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.