The test uses add-repo --release 17.2.6 to verify version-string repo
handling, but debian-17.2.6 only has focal and bullseye suites and
jammy packages weren't built until 17.2.7. This causes apt-get update
to fail with a 404 on ubuntu_22.04 nodes.
see: https://download.ceph.com/debian-17.2.6/dists/
This fix bumps the version to 17.2.7 which includes a jammy suite.
Fixes: https://tracker.ceph.com/issues/77130
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
repurpose ceph_test_neorados_completions to verify that completions are
delivered to the handler's associated executor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
this 34.1.0 version fails to pip install under python 3.12 when testing
ubuntu 24.04
i chose new version 45.0.0 because it corresponds to this commit:
> Use stable constraint in tox to release new tag for 2025.2
which matches the stable/2025.2 tag we use for keystone
Fixes: https://tracker.ceph.com/issues/76997
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Introduce a 'last_degraded' timestamp to the pg_stat_t structure to track
the initial point of redundancy loss. This field, used in conjunction
with 'last_clean', allows the manager to calculate a cluster-wide
durability score by measuring the duration of vulnerability windows.
Changes:
1) Add last_degraded (utime_t) to pg_stat_t in osd_types.h.
2) Increment pg_stat_t encoding version to 31. The decode logic
defaults last_degraded to last_clean for backward compatibility
during rolling upgrades.
3) Update operator==, dump(), and generate_test_instances() to
support ceph-dencoder testing and JSON output.
4) Implement latching logic in PeeringState::prepare_stats_for_publish():
- A PG is considered vulnerable if in DEGRADED or UNDERSIZED state.
- last_degraded is set to 'now' only if it is <= last_clean,
effectively latching the timestamp to the start of the failure
event until the PG next becomes clean.
5) Standalone tests to verify:
- The last_degraded timestamp latching logic.
- Verify last_degraded timestamp is modified when OSDs are marked 'out' for
draining purposes in which case PGs are marked undersized.
6) Release note the addition of 'last_degraded' field to PG stats.
Fixes: https://tracker.ceph.com/issues/76604
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Enhance mgr cache - from cache based on ttl to cache based on changes
Reviewed-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>
EC Direct Reads
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
This patch removes the old TTLCache implementation and introduces
a new generic MgrMapCache driven by a runtime toggle:
- Add `mgr_map_cache_enabled` config option in global.yaml
- Swap out `ttl_cache` for `api_cache` (MgrMapCache) in ActivePyModules
- Update cacheable_get_python() and get_python() to use LFUโbased api_cache
- add new get_mutable parameter to the get api call to get a copy.
- Invalidate api_cache on notify_all events
- Remove all TTLCache headers, sources, and tests
- Include MgrMapCache.cc in CMakeLists and update BaseMgrModule bindings
- Improve logging around cache hits, misses, and state changes
- ActivePyModules
* Remove unused update_cache_metrics()
* Log cache hits/misses inline and only insert into cache when
enabled+cacheable (with proper Py_INCREF)
* Switch get_python() to use PyFormatterRO for cacheable routes, PyFormatter otherwise
- MgrMapCache/LFUCache
* Add can_read_cache()/can_write_cache() helpers and use const& for key parameters
* Guard perf counter increments and improve debug logging
- PyFormatter
* Add PyFormatterRO subclass that freezes dicts/lists into read-only
proxies on the fly
- Python mgr_module
* Simplify get() to return raw result
This change ensures immutable JSON output on cache hits and tightens up cache logic.
mgr/cli: add cache flush command with proper status reporting
Allow operators to invalidate individual mgr Python caches at runtime
without restarting the manager. Introduces a new CLI command:
$ ceph mgr cli cache flush <map-name>
which returns success or a clear error if the named cache entry doesnโt
exist or isnโt cacheable. This makes it easy to drop stale cached maps
(e.g. osd_map, mon_map) on demand.
Fixes: https://tracker.ceph.com/issues/72447
Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
mgr: add new unit tests for MgrMapCache
- Guard against null perfโcounter before calling inc(), preventing crashes
- Add โfooโ to allowed_keys list (for test coverage)
- Rename and refocus the CMake test target from TTLCache to MgrMapCache
- Introduce test_mgrmapcache.cc with LFUCache tests.
- Remove the obsolete test_ttlcache.cc
Fixes: https://tracker.ceph.com/issues/72447
Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
mgr/test_cache: add new tests
adding new unit-test for mgrcache
Fixes: https://tracker.ceph.com/issues/72447
Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
Notably, only include these the `dnf install` commands if the distro is
not overriden by some other mechanism (like cephfs kernel overrides).
This is only a problem for tentacle presently as the k-stock kernel will
override with centos9.
Fixes: https://tracker.ceph.com/issues/77037
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
rbd-target-api validates that the gateway hostname supplied by gwcli
matches the container's own socket.getfqdn(). Running the same call on
the host can return a different value when the host and container resolve
names differently (e.g. on Rocky 10), causing gateway creation to fail
with HTTP 400 and all subsequent gwcli configuration to break silently.
Query the FQDN from inside the iSCSI container directly so the value is
always consistent with what rbd-target-api expects. This also removes the
"run twice" workaround, which was compensating for host-side DNS
warm-up flakiness rather than addressing the underlying mismatch.
Fixes: https://tracker.ceph.com/issues/74577
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Enable autoscale mode for pools which is default off for teuthology
Increase mon_target_pg_per_osd so pools scale up by enough
Signed-off-by: Eric Zhang <emzhang@ibm.com>
The current method fetches files from git.ceph.com, which is unreliable
and sometimes causes the file to contain HTML output instead of the C++ code.
Fetching from GitHub is a more reliable way to get the C++ code.
Fixes: https://tracker.ceph.com/issues/68669
Signed-off-by: Laura Flores <lflores@ibm.com>
test coverage for `rgw crypt sse algorithm: aes-256-gcm` was recently
added to the multisite suite, doubling the number of jobs from 2 to 4
instead of duplicating the multisite jobs for both cbc and gcm, select
a random algorithm for each of the 2 existing jobs
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Add DEPRECATED_EC_PLUGIN to the list of health warnings to
ignore in the thrash-erasure-code-* tests that use deprecated
plugins or techniques. It is expected that this warning will
be raised.
Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Add "have only 1 nvmeof gateway" to ignorelist.
NVMEOF_SINGLE_GATEWAY is already part of ignorelist
but tests sometimes fail on "have only 1 nvmeof gateway".
Thrasher or scalability tests can trigger this but there
are enough asserts to ensure all expected gateways are
up, we can safely ignore this healthcheck warning.
Fixes: https://tracker.ceph.com/issues/75913
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>
The safe_while retry loop does not catch exceptions, so a
CommandCrashedError from `nvme list` bypasses it entirely. Catch
CommandCrashedError and continue the retry loop instead.
Fixes: https://tracker.ceph.com/issues/76984
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Previously the test_rados_tool.sh test was dependant on flag ordering. This mean if you added a new flag after full_quota (such as split_reads or ec_optimizations), this could break the teuthology test if we try to test with these flags on. We prevent this by changing this condition to use json to ensure we are no longer depend on the order of the flags which the default command line output gives.
This also adds a check to ensure the pool name matches what we are working on, to ensure we don't get false-positives if we happened to have other pools.
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>
So we no longer need to install system-wide.
Avoids errors like on Ubuntu 24.04:
2026-05-24T13:48:19.681 DEBUG:teuthology.orchestra.run.trial043:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt
2026-05-24T13:48:19.861 INFO:teuthology.orchestra.run.trial043.stderr:error: externally-managed-environment
2026-05-24T13:48:19.861 INFO:teuthology.orchestra.run.trial043.stderr:
2026-05-24T13:48:19.861 INFO:teuthology.orchestra.run.trial043.stderr:ร This environment is externally managed
2026-05-24T13:48:19.861 INFO:teuthology.orchestra.run.trial043.stderr:โฐโ> To install Python packages system-wide, try apt install
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: python3-xyz, where xyz is the package you are trying to
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: install.
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr:
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: If you wish to install a non-Debian-packaged Python package,
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: create a virtual environment using python3 -m venv path/to/venv.
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: sure you have python3-full installed.
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr:
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: If you wish to install a non-Debian packaged Python application,
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: it may be easiest to use pipx install xyz, which will manage a
2026-05-24T13:48:19.862 INFO:teuthology.orchestra.run.trial043.stderr: virtual environment for you. Make sure you have pipx installed.
2026-05-24T13:48:19.863 INFO:teuthology.orchestra.run.trial043.stderr:
2026-05-24T13:48:19.863 INFO:teuthology.orchestra.run.trial043.stderr: See /usr/share/doc/python3.12/README.venv for more information.
2026-05-24T13:48:19.863 INFO:teuthology.orchestra.run.trial043.stderr:
2026-05-24T13:48:19.863 INFO:teuthology.orchestra.run.trial043.stderr:note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
2026-05-24T13:48:19.863 INFO:teuthology.orchestra.run.trial043.stderr:hint: See PEP 668 for the detailed specification.
2026-05-24T13:48:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1
2026-05-24T13:48:19.883 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/run_tasks.py", line 112, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/task/__init__.py", line 122, in __enter__
self.setup()
File "/home/teuthworker/src/github.com_ceph_ceph-c_1bc3c25246d3a6fbc360dc78d9b4b51200743391/qa/tasks/cbt.py", line 173, in setup
self.install_dependencies()
File "/home/teuthworker/src/github.com_ceph_ceph-c_1bc3c25246d3a6fbc360dc78d9b4b51200743391/qa/tasks/cbt.py", line 112, in install_dependencies
self.first_mon.run(args=pip_install_cmd)
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/orchestra/remote.py", line 596, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/orchestra/run.py", line 461, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_3686f8793d626abcf5a0018da0a50786e41fed9d/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 1: 'python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt'
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Put the release name in the yaml name so it's easy to read from the job
description. "ubuntu_latest" means different things depending on the
Ceph release.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Compared to older distros, this one complains if
you include `-q hostnqn` in the nvme connect command,
saying "Failed to write to /dev/nvme-fabrics: Invalid argument".
Removing that argument gets passed that error and
doesn't seem to have any downsides
Signed-off-by: Adam King <adking@redhat.com>
Now that the ceph-mgr plugins are being separated into essential and
non-essential packages (always-on vs. optional), cephfs qa suite
requires the optional packages for ceph-mgr plugins which are not
always-on, but are being tested with fs suite. The good thing is, we
can install _all_ optional plugins using ceph-mgr-modules-standard
package instead of installing cherry-picked packages.
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Set rgw sts key and enable rgw s3 auth use sts, both needed by
test_bucket_logging_requester_assumed_role. Mirrors the existing
settings in qa/suites/rgw/verify/overrides.yaml.
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
Add support for mutual TLS (mTLS) client certificate authentication
when publishing bucket notifications to Kafka brokers. RGW can now
present a client certificate and private key to authenticate with
brokers that require ssl.client.auth=required.
Changes:
- Add ssl-certificate-location, ssl-key-location, and ssl-key-password
topic attributes for configuring client certificates
- Validate that ssl_certificate and ssl_key are provided together
- Include ssl_key_password in connection identity (hash/equality)
- Add kafka-security.sh script for generating broker and client TLS certs
- Add mTLS test (test_notification_kafka_security_ssl_mtls) using
use_mtls=True flag on the existing SSL security path
- Update RGW notifications documentation with mTLS parameters
Fixes: http://tracker.ceph.com/issues/67427
Signed-off-by: Jan Radon <jan.fabian.radon@sap.com>
The thrashosds task is occasionally restarting OSDs and mon/mgr log
warnings are wrongly flagging this as a problem.
Fixes: https://tracker.ceph.com/issues/76747
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
this 'keytool' invocation was moved from qa/tasks/s3tests_java.py to
qa/tasks/rgw.py so that it would also cover the java checksum tests
but that means it runs for any rgw job with https enabled, even if it
doesn't install or use any java stuff. the 'keytool' command itself
comes from jdk packages which aren't installed by default
ignore errors from this command so that subsuites can use https without
installing java
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Add an aes facet to the rgw/crypt and rgw/multisite suites so
teuthology runs them with both the default cipher (CBC) and with
rgw_crypt_sse_algorithm set to aes-256-gcm.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
We were seeing failures in TestShellOpts again
and again due the (newline added within the output)
output structure change with the cmd2 releases.
Now, we directly extract the final output of 'set editor'
command and use it, instead of traversing on complete output
through hardcoded indexes.This fix is better than earlier
hardcoded splitting/striping via indexes.
Fixes: https://tracker.ceph.com/issues/71795
Signed-off-by: Neeraj Pratap Singh <Neeraj.Pratap.Singh1@ibm.com>
i would prefer to run the s3control test coverage in rgw/verify, but it
depends on rgw_dns_name configuration and support for wildcard dns which
breaks most of the other rgw/verify test cases
Signed-off-by: Casey Bodley <cbodley@redhat.com>
We want to reduce the number of EC plugins and techniques we support
in order to focus dev and test effort on the ones that are most
useful.
We are deprecating the following plugins and techniques in Umbrella,
and dropping support for them in the V release:
* shec
* clay
* all non-reed_sol_van jerasure techniques
This commit adds a health checker to print a warning message if the cluster
is using any of the deprecated plugins/techniques and instructs the user
to migrate objects to a different pool.
Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
The test reused the "tentacle" jq filter while validating
"nvmeof_beacon_diff", causing the comparison to fail. Also fix
the umbrella feature validation and update the expected
persistent feature count.
Fixes: https://tracker.ceph.com/issues/76472
Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>
This test is there to aid in debugging coredumps creation and collection.
It always fails (as it intentionally leaves the coredump in place for collection).
Disabling it in the suite to avoid noise, but leaving the test in place for manual
runs when needed.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Trigger a crash on a Crimson OSD via the admin socket 'assert'
command and verify the OSD goes down and a coredump is produced.
Exercises the debug_asok_assert_abort path added in the companion
commit.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Originally resolved in 9fa163df8f but the
changes were lost when upgrade suites were refreshed for the new release.
Fixes: https://tracker.ceph.com/issues/76599
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
crimson/osd,qa: support OSD memory size in the OSD and in QA suites
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>
* refs/pull/68128/head:
qa: Fix checksum calculation on empty directories
qa: Add mirror test for snapshot with only dir
tools/cephfs_mirror: Fix sync hang
Reviewed-by: Venky Shankar <vshankar@redhat.com>
qa/cephadm: start upgrade tests from tentacle instead of reef on main
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
After adding mds client into singletone bluestore and increas debug
levels and bluestore_allocator=stupid client is slow enough that MDS
evicts it after ~302 seconds
Fixes: https://tracker.ceph.com/issues/76497
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
we can simply ignore this warning that pops up temporary on the logs,
since do check for active+clean in ceph.healthy
Fixes: https://tracker.ceph.com/issues/72424
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
This commit removes centos9 from crimson's supported distros. This is in
line with the wider ceph moving on to rocky10 from centos9. We have
established that crimson is compatible with rocky10. More details can be
found in this tracker: https://tracker.ceph.com/issues/75823.
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Add --logging-type flag to run the Python bucket logging test suite
in either Standard or Journal mode. The same tests run against both
logging types with no changes to test logic or assertions.
- Add --logging-type pytest CLI option (Standard default, Journal opt-in)
- Detect boto3 LoggingType extension availability at session startup
- Thread logging_type through helpers and test functions
- Add teuthology task YAML for Journal mode suite runs
- Install service-2.sdk-extras.json in the teuthology task when
logging_type is Journal (s3tests cleans it up after its own run,
so the file isn't available by the time our Journal job runs)
- Document Journal mode local usage in the test suite README
Signed-off-by: Shree Jejurikar <shree.jejurikar@gmail.com>
Add cloud_target_by_bucket and cloud_target_by_bucket_prefix options
to rgw_cloudtier.py and s3tests.py. Create new test suite to run
target_by_bucket-specific s3-tests.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
used in testing. This translates into more segments, which helps
in preventing test failures due to insufficient free segments for mounting.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
The objectstore tool tests restart the OSDs without allowing enough
time for GC to run, which can lead to no-OOL-segments conditions on restart. This
adds a gc_before_restart option to the test config, which when set
to true will run crimson-objectstore-tool --op gc on each OSD
before restarting them.
Fixes: https://tracker.ceph.com/issues/73101
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Implements real-time per-bucket lifecycle (LC) monitoring via performance
counters exposed through the admin socket.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Add a crimson workunit that validates pg query and list_unfound
including offset behavior, and wire it into the singleton suite.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
Add a dedicated crimson-rados singleton suite yaml to run the
osd/pg-subcommands.sh standalone test.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
This adds test script for pg subcommands like query, log, scrub,
deep-scrub, list-unfound. It compares the output between classic and
crimson.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
When building with ASan, it reports leaks in five tests that exercise
CPython 3.10 either as an external process (bin/ceph) or embedded in
unittests:
#36 smoke.sh (timeout, via bin/ceph)
#205 unittest_mgr_pyformatter (libpython3.10.so embedded)
#206 unittest_mgr_pyutil (libpython3.10.so embedded)
#210 mgr-dashboard-smoke.sh (via bin/ceph)
#225 safe-to-destroy.sh (via bin/ceph)
A representative stack from #210:
Direct leak of 25846 byte(s) in 28 object(s) allocated from:
#0 malloc
#1 /usr/bin/python3.10+0x16d7be
Direct leak of 20456 byte(s) in 29 object(s):
#1 PyDict_Copy /usr/bin/python3.10+0x16ae06
Direct leak of 8456 byte(s) in 14 object(s):
#1 _PyObject_GC_NewVar /usr/bin/python3.10+0x14fc57
The unittests show the same shape against /lib/.../libpython3.10.so.
These are CPython 3.10 runtime artefacts, not Ceph bugs. In CPython
3.10, Py_FinalizeEx() leaves a set of interpreter-internal allocations
(module namespace dict copies, GC-tracked variable-size containers,
type-method caches, interned strings) heap-allocated until process
exit; the OS reclaims them. PEP 683 (Immortal Objects, accepted for
Python 3.12) extends pylifecycle.c::finalize_modules() to deallocate
these remaining objects during runtime shutdown:
"During runtime shutdown, the strategy will be to first let the
runtime try to do its best effort of deallocating these instances
normally. Most of the module deallocation will now be handled by
pylifecycle.c:finalize_modules() where we clean up the remaining
modules as best as we can."
-- PEP 683, https://peps.python.org/pep-0683/
The qa/lsan.supp file has carried the empirical note "python3.12
doesn't leak anything" since the file was introduced (commit
8c099a5340, "asan: add qa/lsan.supp for leak sanitizer
suppressions", Mar 2023). An empirical reproduction with a minimal
Py_Initialize()/Py_FinalizeEx() program built against Python 3.13
under -fsanitize=address reports zero leaks; the same minimal test
against Python 3.10 (CI environment) reports the leaks shown above.
Add three suppressions to qa/lsan.supp's existing 3.9-3.11 block:
- leak:^PyDict_Copy and leak:^_PyObject_GC_NewVar match the two
exported CPython functions visible in the leak stacks.
- leak:python3.10 substring-matches the module path in the stack
frame, catching unsymbolised offsets in both the /usr/bin/python3.10
binary and the libpython3.10.so.1.0 shared object. Mirrors the
existing leak:libsqlite3.so pattern earlier in the file.
The whole block (including the existing PyMem_Malloc entry above) can
be removed once CI runs on Python >= 3.12.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Currently tests wait for 10 seconds for clone progress bar to appear and
it usually takes 8 seconds to find it. In case of the host where test is
being run is slow, the progress bar can take more time to appear and the
test can fail unnecessarily. To avoid this, increase the waiting
duration.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
There's a race condition between cloning operation and clone progress
reporter thread to open the subvolume, former does so to copy/clone
files and latter to get details of the source and destination subvolume.
A low number of files in the snapshot gives clone progress reporter
thread less or no chance to fetch and process the details needed to
display the progress bar. This causes tests to fails when progress bar
isn't displayed in the output of "ceph status" output.
Increasing the number of files will give clone progress reporter more
time to display the progress bar and therefore these unnecessary test
failures wlll reduce/stop.
This failure wasn't seen earlier because Python code was being used to
copy files which was comparatively slow (compared to C++ code that was
merged to copy files as a part of fscrypt work) which allowed enough
time to clone progress reporter thread to process stats and display
clone progress bar.
Fixes: https://tracker.ceph.com/issues/74082
Signed-off-by: Rishabh Dave <ridave@redhat.com>
follow up on commit f3c938f503
```
2026-04-22T10:16:21.350 DEBUG:teuthology.orchestra.run.trial118:> sudo ceph-bluestore-tool zap-device --dev /dev/nvme3n1 --yes-i-really-really-mean-it
2026-04-22T10:16:21.358 INFO:teuthology.orchestra.run.trial118.stderr:sudo: ceph-bluestore-tool: command not found
```
ceph-bluestore-tool is not available on the host and must be run from within a container.
Instead, use `ceph-volume lvm zap` which internally calls `ceph-bluestore-tool zap-device`.
Fixes: https://tracker.ceph.com/issues/76238
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Latest versions of OS comes with fuse3 installed and we are checking for
only fuse in the get_package_version check in the test case. Checking just
for the package string fuse will return None if only fuse3 is installed.
Adding check for both fuse and fuse3 packages now and also handling NoneType
return in _is_flockable() if the package is actually missing.
Fixes: https://tracker.ceph.com/issues/75959
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Modify calc_final_pg_target test cases to reflect optimization algorithm
Add test cases that ensure cost is respected to ensure pools get rounded
closely to target
Fixes: https://tracker.ceph.com/issues/73418
Signed-off-by: Eric Zhang <emzhang@ibm.com>
some commands during setup expect the zone to exist already, so run
'radosgw-admin user list' to make sure a default zone/zonegroup are
created. avoid duplicating this in several subtasks by moving this to
its own subtask that runs when a realm is not configured
Signed-off-by: Casey Bodley <cbodley@redhat.com>
This type of warning is typical during tests where we are thrashing the
OSDs.
Fixes: https://tracker.ceph.com/issues/73147
Signed-off-by: Laura Flores <lflores@ibm.com>
Currently it's set on the intermediary clone instead of the parent.
As a result the setting is effective only for reads that terminate at
the intermediary clone -- reads that go all the way to the parent may
end up being handled as not sparse depending on their size.
Fixes: https://tracker.ceph.com/issues/76101
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>
Conflicts:
qa/valgrind.supp
The qa/valgrind.supp file had a conflict with an extra curly brace at the EOF as a result of changes adding CPython suppressions in the unittest-mgr branch, as well as additions coming form main.
Resolved by keeping both: the CPython suppressions for the mgr unit tests and all the rocky10 suppressions from main (rocky10_gcc14_mismatch_delete_map_erase
and rocky10 still reachable rocksdb leak), and removing the extra curly brace after merging.
qa/valgrind: generalize suppressions for gcc-14 MismatchedFree
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
PDSH is not available in EPEL10 repos. Install the el9 RPMs directly
via yum, which are compatible with el10.
Fixes: https://tracker.ceph.com/issues/75877
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
Summary:
Fix misleading failure reasons reported as `"โฆ in cluster log"` when
no such log entry actually exists.
The cephadm task currently treats `grep` errors from the cluster log
scan as if they were actual log matches. This can produce bogus
failure summaries when `ceph.log` is missing, especially after early
failures such as image pull or bootstrap problems.
Problem:
first_in_ceph_log() currently:
- returns stdout if a match is found
- otherwise returns stderr
The caller then treats any non-None value as a real cluster log hit and formats it as:
"<value>" in cluster log
That means an error like:
grep: /var/log/ceph/<fsid>/ceph.log: No such file or directory
can be misreported as if it came from the cluster log.
This change makes cluster log scanning robust and accurate by:
- checking whether /var/log/ceph/<fsid>/ceph.log exists before scanning
- using check_status=False for the grep pipeline
- treating only stdout as a real log match
- treating stderr as a scan error instead of log content
- avoiding overwrite of a more accurate pre-existing failure_reason
- reporting scan failures separately as cluster log scan failed
Fixes: https://tracker.ceph.com/issues/76051
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
* refs/pull/66294/head:
qa: enforce centos9 for test
qa: rename distro
qa/suites/fs/bugs: use centos9 for squid upgrade test
qa: remove unused variables
qa: use centos9 for fs suites using k-testing
qa: update fs suite to rocky10
qa: skip dashboard install due to dependency noise
qa: only setup nat rules during bridge creation
qa: correct wording of comment
qa: use nft instead iptables
qa: use py3 builtin ipaddress module
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Test that put-bucket-acl and put-object-acl work both before and after
migrating a user to an account. After migration, the bucket/object ACL
owner is still the old user id, but the requester authenticates as the
account id.
Signed-off-by: Krunal Chheda <kchheda3@bloomberg.net>
Update yaml file to include TestCorruptedSubvolumes in qa tests.
Fixes: https://tracker.ceph.com/issues/76001
Signed-off-by: Jos Collin <jcollin@redhat.com>
The kernel mount overrides for the distro have no effect if they are
applied before `supported-random-distro`.
Fixes:
2026-04-13T19:06:13.603 INFO:teuthology.task.pexec:sudo dnf remove nvme-cli -y
2026-04-13T19:06:13.603 INFO:teuthology.task.pexec:sudo dnf install nvmetcli nvme-cli -y
2026-04-13T19:06:13.626 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial005.front.sepia.ceph.com
2026-04-13T19:06:13.627 INFO:teuthology.task.pexec:sudo dnf remove nvme-cli -y
2026-04-13T19:06:13.627 INFO:teuthology.task.pexec:sudo dnf install nvmetcli nvme-cli -y
2026-04-13T19:06:13.652 INFO:teuthology.orchestra.run.trial148.stderr:sudo: dnf: command not found
2026-04-13T19:06:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1
2026-04-13T19:06:13.654 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/task/pexec.py", line 149, in task
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/task/pexec.py", line 62, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 485, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on trial148 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
which was done because these dnf commands were pulled from rocky10.yaml from the kclient overrides but ubuntu_latest was used for the random distro.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
A better approach would be to include centos9 OR rocky10 for
distribution choice. Then we can just filter out rocky10 when we're
testing the `testing` kernel but keep rocky10 coverage for other
testing.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
2025-11-18T19:46:46.226 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stderr Error ENOTSUP: Module 'alerts' is not enabled/loaded (required by command 'dashboard set-ssl-certificate'): use `ceph mgr module enable alerts` to enable it
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Currently the code recreates these NAT rules for every mount. This only
needs to be done once by the first mount.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
rocky.10 does not support iptables with MASQUERADE targets. (Or maybe it
does with more prodding but it's easier to just switch to nft.)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
mgr/DaemonServer: Limit search for OSDs to upgrade within the crush bucket
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
qa/cephadm: ignore transient CEPHADM_FAILED_DAEMON in smoke-singlehost
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
