\(POOL_FULL\) only matches the "Health check failed/update" cluster-log
format (code wrapped in parens). It misses the "Health check cleared"
format, where the code is not parenthesized (see src/mon/Monitor.cc),
so a pool clearing its full state mid-upgrade-test was not ignorelisted.
Drop the escaped parens so the pattern matches both forms, consistent
with other bare entries already in this file (OSD_ROOT_DOWN,
MDS_INSUFFICIENT_STANDBY).
Fixes: https://tracker.ceph.com/issues/78149
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
The dashboard backend API suite occasionally fails in CI on the job's
7200 second timeout alone, with every test passing. Suite runtime
varies with builder load: identical runs of the same 311 tests took
anywhere from about 5400 to over 7000 seconds, and teardown needs
roughly another 180 seconds on top. On the runs that landed at the
high end, only 60 to 80 seconds of margin were left before the
timeout fired. Cutting setup overhead gives the suite that margin
back.
DashboardTestCase.setUpClass() restarted every mgr twice: once in
MgrTestCase.setup_mgrs(), and again right after in _assign_ports(),
whose only job is to set the per-mgr dashboard port, which has to
happen while the mgrs are down.
Let subclasses declare their module ports in a MODULE_PORTS class
attribute instead. setup_mgrs() assigns them while the daemons are
already stopped, so the single restart it performs picks them up and
the second stop/fail/restart cycle disappears.
The assignment loop moves to _assign_module_ports(). _assign_ports()
keeps its old stop/assign/restart contract for callers that reassign
ports outside class setup (test_prometheus, test_dashboard,
test_module_selftest) and delegates to the shared helper.
With 66 dashboard test classes per API suite run this removes one
full mgr bounce per class, roughly 10 to 15 minutes of wall clock in
the CI runs examined.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
qa/workunits/smb: work around test failures by pinning smbprotocol version
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Work around new tests failures (in the TestHostsAccessToggle1 tests)
where the error/exception handling has apparently changed by just
pinning the version (by excluding the new v1.17.0 and newer versions).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
UI has an async validator which calls the GET bucket API to make sure
the bucket name doesn't exist, but the proxy
was not properly handling the http_status_codes which results in raising
a massive traceback in logs whenever you type things in the bucket name
field. So handling that gracefully by capturing the proper status codes
for both RequestException and DashboardException
BEFORE
```
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in _call_
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get
result = self.proxy(daemon_name, 'GET', 'bucket', {'bucket': bucket})
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 213, in proxy
raise DashboardException(e, http_status_code=http_status_code, component='rgw')
dashboard.exceptions.DashboardException: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000f14e08c1af0d5615-006'
b'71f54a7-3bc6-default","HostId":"3bc6-default-default"}')
2024-10-28T09:08:55.990+0000 7f89e045b640 0 [dashboard INFO request] [::ffff:10.74.18.122:51853] [GET] [500] [0.007s] [admin] [200.0B] /api/rgw/bucket/bucket-das
```
AFTER
```
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.services.exception] Dashboard Exception: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000c029a81e5d154844-006'
b'a4e183c-14251-default","HostId":"14251-default-default"}')
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.tools] [::ffff:192.168.100.1:60408] [GET] [404] [0.078s] [admin] [200.0B] /api/rgw/bucket/testsa
```
Fixes: https://tracker.ceph.com/issues/78038
Signed-off-by: Nizamudeen A <nia@redhat.com>
As we do not yet scrub scheduling in Crimson, the
OSDs do not perform deep scrubs. last_deep_scrub_stamp thus
stays at epoch 0 for all PGs. The monitor flags these as overdue
immediately, causing spurious test failures — particularly in tests
that restart OSDs (e.g. crimson_fio_restart), where PGs reaching
active+clean state trigger the health check.
Fixes: https://tracker.ceph.com/issues/77929
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
setup_ca_signed_keys() hardcoded `systemctl restart sshd`, which fails
on Debian/Ubuntu where the SSH server unit is named `ssh.service`
rather than `sshd.service`. Try `ssh` first, fall back to `sshd` if
that fails, grouped so the fallback only fires on restart failure and
doesn't mask a failed config write.
Fixes: https://tracker.ceph.com/issues/77991
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
This used to be the case before commit d4b977afdc ("qa/distros:
rename centos_latest.yaml to rpm_latest.yaml") and subsequent changes
to enable Rocky 10. When paired with valgrind, python_api_tests* jobs
fail persistently on Ubuntu and Rocky, see [1]. Switch back until that
is resolved (or for as long as centos_9.stream remains in the mix).
[1] https://tracker.ceph.com/issues/74864
Fixes: https://tracker.ceph.com/issues/77982
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Currently client.admin is passed for client_name and that doesn't
exercise client_name handling much as client.admin is the default.
When deploying multiple clusters the ceph task distributes keyrings
with only the key for the initial monitor and client.admin key. Other
keys (e.g. client.0) are present only on their respective clusters, so
client.0's key for cluster2 needs to be obtained on cluster1 explicitly
with "ceph auth get".
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
In TestFSCryptVolumes, make sure to use mount class to
compare directory trees and to read/write files.
Fixes: https://tracker.ceph.com/issues/74110
Signed-off-by: Christopher Hoffman <choffman@redhat.com>
* refs/pull/66558/head:
qa/cephfs: minor fix in comment
qa/cephfs: give more time to tests in test_clone_stats.py
qa/cephfs: increase number of files to cloned in test_clone_stats.py
volumes/stats_util: improve log messages
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Without a facet marker, teuthology concatenated all three yamls into
one job, running plain, CBC, and GCM workunits back-to-back under
whichever rgw override won.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Extract the decrypt/decompress/recompress/re-encrypt filter pipeline
into a shared RGWRecompressDPF base class. RGWTransitionDPF is added
for lifecycle transitions and RGWCopyObjDPF is rewritten on top of
the same base, both as thin subclasses providing key retrieval and
compression config.
Add rgw_prepare_decrypt_object() for decrypting from stored attrs
without an HTTP request context (handles SSE-KMS/-GCM, SSE-S3/-GCM,
RGW-AUTO/-GCM), and rgw_prepare_reencrypt_object() which regenerates
the GCM salt before re-deriving the per-object key — avoids AEAD
nonce reuse when LC re-encrypts modified plaintext.
Includes a QA workunit that drives transitions through three
storage classes with different compression configs and verifies
data integrity, including with SSE-KMS encrypted objects.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Skip influx selftests when the Python influxdb client is not installed.
Remove the ineffective testhost workaround and ignore the expected
MGR_INFLUX_NO_SERVER warning when the module runs without a server.
Fixes: https://tracker.ceph.com/issues/77250
Signed-off-by: Kobi Ginon <kginon@redhat.com>
python 3.12 warns on unrecognized escape sequences in string literals.
fuse_mount's admin-socket pyscript and mount's nft payload carry regex
and shell escapes (\., \d, \;) that are meant literally, so make those
strings raw. filesystem's get_mds_addr docstring had a stray \/ in the
example address, which is just a slash, so drop the backslash.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
module_selftest fails on a cluster log warning that is not ignorelisted:
cluster [WRN] Health check failed: Failed to list/create InfluxDB database (MGR_INFLUX_DB_LIST_FAILED)
module_selftest is the only job that enables the influx module, and no suite
provisions an InfluxDB server, so once enabled the module cannot reach a backend
and raises a health warning: MGR_INFLUX_NO_SERVER when no hostname is set,
MGR_INFLUX_DB_LIST_FAILED from serve() once one is. test_influx sets a hostname,
so it hits the latter.
This surfaced only after the test image started shipping the influxdb python
module. Until then can_run() returned false, the module never enabled, and the
sole log line was "influxdb python module not found", which is already
ignorelisted. self_test() cannot run without enabling the module, so ignorelist
the warning it raises.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
rbd-mirror: Remove old non-primary demoted image snapshots on the local cluster
Reviewed-by: VinayBhaskar-V <vvarada@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
When set\get\rm config values with leading or trailing spaces,
those spaces need to be removed before we are trying to get\set\rm
those keys.
add trim code before each approch and also for exist keys in config.
Fixes: https://tracker.ceph.com/issues/77598
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
When an image is demoted on primary cluster and later promoted again,
a non-primary demoted image snapshot is created on the peer (secondary)
cluster. Each such promote/demote cycle on same cluster results in an
additional non-primary demoted snapshot being created on peer.
As a result, repeated promote/demote cycles on the same cluster can
lead to accumulation of stale non-primary demoted snapshots on the
secondary cluster. These snapshots are not removed immediately because
cleanup is only triggered when peer (secondary) is promoted and then
demoted.
The proposed changes ensures that such snapshots are proactively identified
and removed, preventing unbounded buildup. Also added test coverage to
verify cleanup behavior.
Fixes: https://tracker.ceph.com/issues/76155
Signed-off-by: Miki Patel <miki.patel132@gmail.com>
Test that multipart upload part heads are tracked under the
rgw.multipart category in radosgw-admin bucket stats, separate from
completed objects (rgw.main).
- Incomplete upload: rgw.main=0, rgw.multipart=X, rgw.multimeta=1
- CompleteMultipart: parts cleaned up, assembled object in rgw.main
- AbortMultipart: parts cleaned up, rgw.multipart returns to 0
- ListObjects returns 0 during incomplete upload
Also add quota enforcement testcase for multipart part category
- Verify that multipart upload parts tracked under RGWObjCategory::MultiPart
still count against both bucket and user quotas.
Tests: https://tracker.ceph.com/issues/77509
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Getting it from our own service avoids exposing us to
supply chain attacks in our QA infrastructure.
Fixes: https://tracker.ceph.com/issues/76927
Signed-off-by: Laura Flores <lflores@ibm.com>
The python binaries on some CI images and dev boxes ship stripped, so the
allocator frames in an interpreter-shutdown leak come through unsymbolised
(/usr/bin/python3.13+0x...) and the function-name matches above cannot apply.
leak:python3.10 already handled this for 3.10, and a stale comment claimed 3.12
does not leak.
Add leak:python3.12 and leak:python3.13, mirroring the 3.10 entry, so the
interpreter globals are suppressed whatever CPython the build uses.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
this task was manually editing /etc/resolv.conf to configure dns names.
on ubuntu 24, these changes were getting clobbered by NetworkManager or
systemd service restarts
instead, manage dnsmasq depending on host distro. rpm-based distros use
NetworkManager's dnsmasq plugin, while deb-based distros use
systemd-resolved to point at the local dnsmasq
Fixes: https://tracker.ceph.com/issues/76961
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Track per-OSD PG rebuild duration in prepare_stats_for_publish()
(primary OSD only). Three new counters are added to the
recoverystate_perf collection:
- pg_rebuild_duration: LONGRUNAVG time counter (sum+count pair);
This counter internally maintains 'avgcount' that tracks the
cumulative number of rebuild events.
- pg_rebuild_max_secs: maximum rebuild duration observed in seconds
- pg_rebuild_min_secs: minimum rebuild duration observed in seconds
The logic uses a per-PG in-memory latch (rebuild_start_time) to
capture the redundancy failure entry point. When a PG is first observed
to be in a vulnerable state (PG_STATE_DEGRADED,
PG_STATE_UNDERSIZED, or num_objects_misplaced/degraded > 0), it
latches info.stats.last_change as the start time, provided
last_change > last_clean, which ensures only genuine new failures
after the prior clean interval are tracked.
On recovery, the rebuild duration is computed as (now - rebuild_start_time)
and is only recorded if delta num_objects_recovered > 0 or the PG had
confirmed redundancy loss at latch time, filtering out spurious state
transitions. The latch is cleared after each recorded event.
The latch is also cleared in clear_primary_state() so that an interval
change or role transition (primary -> replica) does not carry a stale
start time or baseline recovered count into a future interval.
The new last_degraded is intentionally not used here to retain compatibility
with older Ceph branches where the field doesn't exist and requires encoding
changes. A future simplification can replace the latch with a direct
(last_clean - last_degraded) calculation once last_degraded is consistently
available.
This interim solution is a close approximation of the PG rebuild time.
These counters are scraped per-OSD by ceph-exporter and exposed to
Prometheus, enabling durability score calculations over user-defined
time windows.
Other Changes:
1. Add unit tests to TestPeeringState.cc that exercise the latch logic.
2. Add a standalone integration test to verify that the rebuild perf counters
are incremented on the primary OSD after a recovery event.
Fixes: https://tracker.ceph.com/issues/77493
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
unittest-seastore runs the seastar reactor on a separate thread
(SeastarRunner) and stops it at exit without draining its pending
tasks, so a few cached extents those tasks still held are leaked at
shutdown. Suppress them by the three seastore subsystems they come from.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Several unittests fail LeakSanitizer on still-reachable allocations that
belong to third-party libraries, not to Ceph. These are boost.thread's
main-thread TLS, OpenSSL's one-time init (the ForkDeathTest children
_exit() before it is freed), and the cipher, DRBG and error-stack state
that OpenSSL and libcryptsetup keep behind the librbd encryption and
migration unittests. None get freed without OPENSSL_cleanup(), so suppress
them by their allocation entry points.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
With test venvs on system site-packages, mypy picks up the system
prettytable (3.4.0+, typed). It flags mgr's add_row(tuple) against the
list[Any] signature (src/mypy.ini) and qa's float_format = str against
the dict[str, str] property (qa/mypy.ini). Skip follow_imports in both.
Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Ignore FS_DEGRADED in libcephfs_interface_tests and MON_DOWN in
rbd_python_api_tests to avoid false failures from transient cluster health
messages.
Signed-off-by: Adarsha Dinda <adarshadinda@Adarshas-MacBook-Pro.local>
Previously, obsolete primary and demoted primary snapshots on the
secondary cluster were not cleaned up immediately after relocation.
Instead, old primary snapshots remained until a subsequent promote
operation triggered their cleanup, while old demoted primary snapshots
persisted until a later demote operation removed them.
Adding changes for proactive cleanup of obsolete primary and demoted
primary snapshots that are no longer required after relocation.
Also adding test coverage to validate the cleanup behavior.
Fixes: https://tracker.ceph.com/issues/76154
Signed-off-by: Miki Patel <miki.patel132@gmail.com>
Use norecover to pause recovery ops while inspecting reservation
state. Without it, recovery can complete before the admin-socket
dump is captured, causing intermittent test failures
Fixes: https://tracker.ceph.com/issues/77113
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
Verify counter dump registration, current-sync gauges while syncing
(full/delta), last-sync and summary counters after idle sync, and
extend mirror stats and remote-snap failure tests for per-directory
snaps_* and dir_state.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Verify that reloading the mirroring module and removing a directory
does not leave a ghost replayer entry that keeps syncing snapshots.
Fixes: https://tracker.ceph.com/issues/77398
Signed-off-by: Kotresh HR <khiremat@redhat.com>
This reverts commit fd20467981.
kmip tests are failing again on ubuntu 24 because PyKMIP doesn't support
python 3.12. we'll be removing ubuntu 22 from main, so can't just pin the
test to that distro in the meantime
we're expecing the nvmeof team to add python 3.12 support in our
ceph/PyKMIP fork, and can reenable kmip testing once that happens
Fixes: https://tracker.ceph.com/issues/76995
Signed-off-by: Casey Bodley <cbodley@redhat.com>
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Fixes: https://tracker.ceph.com/issues/77272
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
* refs/pull/66726/head:
doc: Update documentation to reflect new functionality
test: Add integration tests for EC Omap operations and recovery
osd: Hook up omap operations in EC pools
osd: Allow for recovery of OMAP header and entries in EC pools
doc: Write design document to explain the reasoning behind implementing this feature
osd: Introduce functions required for EC OMAP support
osd: Add ECOmapJournal class and relocate OmapUpdateType enum class
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Verify that rpcclient lookupsids resolves domain user SIDs correctly
inside the smbd container, preventing regressions on /run bind mount
permissions that break smbd to winbindd communication.
Fixes: https://tracker.ceph.com/issues/77120
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
Assisted-by: Bob
Used for writing tests following the pattern of existing tests.
Fixes: https://tracker.ceph.com/issues/74188
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
