Now that the ceph-mgr plugins are being separated into essential and
non-essential packages (always-on vs. optional), cephfs qa suite
requires the optional packages for ceph-mgr plugins which are not
always-on, but are being tested with fs suite. The good thing is, we
can install _all_ optional plugins using ceph-mgr-modules-standard
package instead of installing cherry-picked packages.
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Set rgw sts key and enable rgw s3 auth use sts, both needed by
test_bucket_logging_requester_assumed_role. Mirrors the existing
settings in qa/suites/rgw/verify/overrides.yaml.
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
Add support for mutual TLS (mTLS) client certificate authentication
when publishing bucket notifications to Kafka brokers. RGW can now
present a client certificate and private key to authenticate with
brokers that require ssl.client.auth=required.
Changes:
- Add ssl-certificate-location, ssl-key-location, and ssl-key-password
topic attributes for configuring client certificates
- Validate that ssl_certificate and ssl_key are provided together
- Include ssl_key_password in connection identity (hash/equality)
- Add kafka-security.sh script for generating broker and client TLS certs
- Add mTLS test (test_notification_kafka_security_ssl_mtls) using
use_mtls=True flag on the existing SSL security path
- Update RGW notifications documentation with mTLS parameters
Fixes: http://tracker.ceph.com/issues/67427
Signed-off-by: Jan Radon <jan.fabian.radon@sap.com>
The thrashosds task is occasionally restarting OSDs and mon/mgr log
warnings are wrongly flagging this as a problem.
Fixes: https://tracker.ceph.com/issues/76747
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
this 'keytool' invocation was moved from qa/tasks/s3tests_java.py to
qa/tasks/rgw.py so that it would also cover the java checksum tests
but that means it runs for any rgw job with https enabled, even if it
doesn't install or use any java stuff. the 'keytool' command itself
comes from jdk packages which aren't installed by default
ignore errors from this command so that subsuites can use https without
installing java
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Add an aes facet to the rgw/crypt and rgw/multisite suites so
teuthology runs them with both the default cipher (CBC) and with
rgw_crypt_sse_algorithm set to aes-256-gcm.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
We were seeing failures in TestShellOpts again
and again due the (newline added within the output)
output structure change with the cmd2 releases.
Now, we directly extract the final output of 'set editor'
command and use it, instead of traversing on complete output
through hardcoded indexes.This fix is better than earlier
hardcoded splitting/striping via indexes.
Fixes: https://tracker.ceph.com/issues/71795
Signed-off-by: Neeraj Pratap Singh <Neeraj.Pratap.Singh1@ibm.com>
i would prefer to run the s3control test coverage in rgw/verify, but it
depends on rgw_dns_name configuration and support for wildcard dns which
breaks most of the other rgw/verify test cases
Signed-off-by: Casey Bodley <cbodley@redhat.com>
We want to reduce the number of EC plugins and techniques we support
in order to focus dev and test effort on the ones that are most
useful.
We are deprecating the following plugins and techniques in Umbrella,
and dropping support for them in the V release:
* shec
* clay
* all non-reed_sol_van jerasure techniques
This commit adds a health checker to print a warning message if the cluster
is using any of the deprecated plugins/techniques and instructs the user
to migrate objects to a different pool.
Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
The test reused the "tentacle" jq filter while validating
"nvmeof_beacon_diff", causing the comparison to fail. Also fix
the umbrella feature validation and update the expected
persistent feature count.
Fixes: https://tracker.ceph.com/issues/76472
Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>
This test is there to aid in debugging coredumps creation and collection.
It always fails (as it intentionally leaves the coredump in place for collection).
Disabling it in the suite to avoid noise, but leaving the test in place for manual
runs when needed.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Trigger a crash on a Crimson OSD via the admin socket 'assert'
command and verify the OSD goes down and a coredump is produced.
Exercises the debug_asok_assert_abort path added in the companion
commit.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Originally resolved in 9fa163df8f but the
changes were lost when upgrade suites were refreshed for the new release.
Fixes: https://tracker.ceph.com/issues/76599
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
crimson/osd,qa: support OSD memory size in the OSD and in QA suites
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>
* refs/pull/68128/head:
qa: Fix checksum calculation on empty directories
qa: Add mirror test for snapshot with only dir
tools/cephfs_mirror: Fix sync hang
Reviewed-by: Venky Shankar <vshankar@redhat.com>
qa/cephadm: start upgrade tests from tentacle instead of reef on main
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
After adding mds client into singletone bluestore and increas debug
levels and bluestore_allocator=stupid client is slow enough that MDS
evicts it after ~302 seconds
Fixes: https://tracker.ceph.com/issues/76497
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
we can simply ignore this warning that pops up temporary on the logs,
since do check for active+clean in ceph.healthy
Fixes: https://tracker.ceph.com/issues/72424
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
This commit removes centos9 from crimson's supported distros. This is in
line with the wider ceph moving on to rocky10 from centos9. We have
established that crimson is compatible with rocky10. More details can be
found in this tracker: https://tracker.ceph.com/issues/75823.
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Add --logging-type flag to run the Python bucket logging test suite
in either Standard or Journal mode. The same tests run against both
logging types with no changes to test logic or assertions.
- Add --logging-type pytest CLI option (Standard default, Journal opt-in)
- Detect boto3 LoggingType extension availability at session startup
- Thread logging_type through helpers and test functions
- Add teuthology task YAML for Journal mode suite runs
- Install service-2.sdk-extras.json in the teuthology task when
logging_type is Journal (s3tests cleans it up after its own run,
so the file isn't available by the time our Journal job runs)
- Document Journal mode local usage in the test suite README
Signed-off-by: Shree Jejurikar <shree.jejurikar@gmail.com>
Add cloud_target_by_bucket and cloud_target_by_bucket_prefix options
to rgw_cloudtier.py and s3tests.py. Create new test suite to run
target_by_bucket-specific s3-tests.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
used in testing. This translates into more segments, which helps
in preventing test failures due to insufficient free segments for mounting.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
The objectstore tool tests restart the OSDs without allowing enough
time for GC to run, which can lead to no-OOL-segments conditions on restart. This
adds a gc_before_restart option to the test config, which when set
to true will run crimson-objectstore-tool --op gc on each OSD
before restarting them.
Fixes: https://tracker.ceph.com/issues/73101
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Implements real-time per-bucket lifecycle (LC) monitoring via performance
counters exposed through the admin socket.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Add a crimson workunit that validates pg query and list_unfound
including offset behavior, and wire it into the singleton suite.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
Add a dedicated crimson-rados singleton suite yaml to run the
osd/pg-subcommands.sh standalone test.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
This adds test script for pg subcommands like query, log, scrub,
deep-scrub, list-unfound. It compares the output between classic and
crimson.
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
When building with ASan, it reports leaks in five tests that exercise
CPython 3.10 either as an external process (bin/ceph) or embedded in
unittests:
#36 smoke.sh (timeout, via bin/ceph)
#205 unittest_mgr_pyformatter (libpython3.10.so embedded)
#206 unittest_mgr_pyutil (libpython3.10.so embedded)
#210 mgr-dashboard-smoke.sh (via bin/ceph)
#225 safe-to-destroy.sh (via bin/ceph)
A representative stack from #210:
Direct leak of 25846 byte(s) in 28 object(s) allocated from:
#0 malloc
#1 /usr/bin/python3.10+0x16d7be
Direct leak of 20456 byte(s) in 29 object(s):
#1 PyDict_Copy /usr/bin/python3.10+0x16ae06
Direct leak of 8456 byte(s) in 14 object(s):
#1 _PyObject_GC_NewVar /usr/bin/python3.10+0x14fc57
The unittests show the same shape against /lib/.../libpython3.10.so.
These are CPython 3.10 runtime artefacts, not Ceph bugs. In CPython
3.10, Py_FinalizeEx() leaves a set of interpreter-internal allocations
(module namespace dict copies, GC-tracked variable-size containers,
type-method caches, interned strings) heap-allocated until process
exit; the OS reclaims them. PEP 683 (Immortal Objects, accepted for
Python 3.12) extends pylifecycle.c::finalize_modules() to deallocate
these remaining objects during runtime shutdown:
"During runtime shutdown, the strategy will be to first let the
runtime try to do its best effort of deallocating these instances
normally. Most of the module deallocation will now be handled by
pylifecycle.c:finalize_modules() where we clean up the remaining
modules as best as we can."
-- PEP 683, https://peps.python.org/pep-0683/
The qa/lsan.supp file has carried the empirical note "python3.12
doesn't leak anything" since the file was introduced (commit
8c099a5340, "asan: add qa/lsan.supp for leak sanitizer
suppressions", Mar 2023). An empirical reproduction with a minimal
Py_Initialize()/Py_FinalizeEx() program built against Python 3.13
under -fsanitize=address reports zero leaks; the same minimal test
against Python 3.10 (CI environment) reports the leaks shown above.
Add three suppressions to qa/lsan.supp's existing 3.9-3.11 block:
- leak:^PyDict_Copy and leak:^_PyObject_GC_NewVar match the two
exported CPython functions visible in the leak stacks.
- leak:python3.10 substring-matches the module path in the stack
frame, catching unsymbolised offsets in both the /usr/bin/python3.10
binary and the libpython3.10.so.1.0 shared object. Mirrors the
existing leak:libsqlite3.so pattern earlier in the file.
The whole block (including the existing PyMem_Malloc entry above) can
be removed once CI runs on Python >= 3.12.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Currently tests wait for 10 seconds for clone progress bar to appear and
it usually takes 8 seconds to find it. In case of the host where test is
being run is slow, the progress bar can take more time to appear and the
test can fail unnecessarily. To avoid this, increase the waiting
duration.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
There's a race condition between cloning operation and clone progress
reporter thread to open the subvolume, former does so to copy/clone
files and latter to get details of the source and destination subvolume.
A low number of files in the snapshot gives clone progress reporter
thread less or no chance to fetch and process the details needed to
display the progress bar. This causes tests to fails when progress bar
isn't displayed in the output of "ceph status" output.
Increasing the number of files will give clone progress reporter more
time to display the progress bar and therefore these unnecessary test
failures wlll reduce/stop.
This failure wasn't seen earlier because Python code was being used to
copy files which was comparatively slow (compared to C++ code that was
merged to copy files as a part of fscrypt work) which allowed enough
time to clone progress reporter thread to process stats and display
clone progress bar.
Fixes: https://tracker.ceph.com/issues/74082
Signed-off-by: Rishabh Dave <ridave@redhat.com>
follow up on commit f3c938f503
```
2026-04-22T10:16:21.350 DEBUG:teuthology.orchestra.run.trial118:> sudo ceph-bluestore-tool zap-device --dev /dev/nvme3n1 --yes-i-really-really-mean-it
2026-04-22T10:16:21.358 INFO:teuthology.orchestra.run.trial118.stderr:sudo: ceph-bluestore-tool: command not found
```
ceph-bluestore-tool is not available on the host and must be run from within a container.
Instead, use `ceph-volume lvm zap` which internally calls `ceph-bluestore-tool zap-device`.
Fixes: https://tracker.ceph.com/issues/76238
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Latest versions of OS comes with fuse3 installed and we are checking for
only fuse in the get_package_version check in the test case. Checking just
for the package string fuse will return None if only fuse3 is installed.
Adding check for both fuse and fuse3 packages now and also handling NoneType
return in _is_flockable() if the package is actually missing.
Fixes: https://tracker.ceph.com/issues/75959
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Modify calc_final_pg_target test cases to reflect optimization algorithm
Add test cases that ensure cost is respected to ensure pools get rounded
closely to target
Fixes: https://tracker.ceph.com/issues/73418
Signed-off-by: Eric Zhang <emzhang@ibm.com>
some commands during setup expect the zone to exist already, so run
'radosgw-admin user list' to make sure a default zone/zonegroup are
created. avoid duplicating this in several subtasks by moving this to
its own subtask that runs when a realm is not configured
Signed-off-by: Casey Bodley <cbodley@redhat.com>
This type of warning is typical during tests where we are thrashing the
OSDs.
Fixes: https://tracker.ceph.com/issues/73147
Signed-off-by: Laura Flores <lflores@ibm.com>
Currently it's set on the intermediary clone instead of the parent.
As a result the setting is effective only for reads that terminate at
the intermediary clone -- reads that go all the way to the parent may
end up being handled as not sparse depending on their size.
Fixes: https://tracker.ceph.com/issues/76101
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>
Conflicts:
qa/valgrind.supp
The qa/valgrind.supp file had a conflict with an extra curly brace at the EOF as a result of changes adding CPython suppressions in the unittest-mgr branch, as well as additions coming form main.
Resolved by keeping both: the CPython suppressions for the mgr unit tests and all the rocky10 suppressions from main (rocky10_gcc14_mismatch_delete_map_erase
and rocky10 still reachable rocksdb leak), and removing the extra curly brace after merging.
qa/valgrind: generalize suppressions for gcc-14 MismatchedFree
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
PDSH is not available in EPEL10 repos. Install the el9 RPMs directly
via yum, which are compatible with el10.
Fixes: https://tracker.ceph.com/issues/75877
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
Summary:
Fix misleading failure reasons reported as `"โฆ in cluster log"` when
no such log entry actually exists.
The cephadm task currently treats `grep` errors from the cluster log
scan as if they were actual log matches. This can produce bogus
failure summaries when `ceph.log` is missing, especially after early
failures such as image pull or bootstrap problems.
Problem:
first_in_ceph_log() currently:
- returns stdout if a match is found
- otherwise returns stderr
The caller then treats any non-None value as a real cluster log hit and formats it as:
"<value>" in cluster log
That means an error like:
grep: /var/log/ceph/<fsid>/ceph.log: No such file or directory
can be misreported as if it came from the cluster log.
This change makes cluster log scanning robust and accurate by:
- checking whether /var/log/ceph/<fsid>/ceph.log exists before scanning
- using check_status=False for the grep pipeline
- treating only stdout as a real log match
- treating stderr as a scan error instead of log content
- avoiding overwrite of a more accurate pre-existing failure_reason
- reporting scan failures separately as cluster log scan failed
Fixes: https://tracker.ceph.com/issues/76051
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
* refs/pull/66294/head:
qa: enforce centos9 for test
qa: rename distro
qa/suites/fs/bugs: use centos9 for squid upgrade test
qa: remove unused variables
qa: use centos9 for fs suites using k-testing
qa: update fs suite to rocky10
qa: skip dashboard install due to dependency noise
qa: only setup nat rules during bridge creation
qa: correct wording of comment
qa: use nft instead iptables
qa: use py3 builtin ipaddress module
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Test that put-bucket-acl and put-object-acl work both before and after
migrating a user to an account. After migration, the bucket/object ACL
owner is still the old user id, but the requester authenticates as the
account id.
Signed-off-by: Krunal Chheda <kchheda3@bloomberg.net>
Update yaml file to include TestCorruptedSubvolumes in qa tests.
Fixes: https://tracker.ceph.com/issues/76001
Signed-off-by: Jos Collin <jcollin@redhat.com>
The kernel mount overrides for the distro have no effect if they are
applied before `supported-random-distro`.
Fixes:
2026-04-13T19:06:13.603 INFO:teuthology.task.pexec:sudo dnf remove nvme-cli -y
2026-04-13T19:06:13.603 INFO:teuthology.task.pexec:sudo dnf install nvmetcli nvme-cli -y
2026-04-13T19:06:13.626 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial005.front.sepia.ceph.com
2026-04-13T19:06:13.627 INFO:teuthology.task.pexec:sudo dnf remove nvme-cli -y
2026-04-13T19:06:13.627 INFO:teuthology.task.pexec:sudo dnf install nvmetcli nvme-cli -y
2026-04-13T19:06:13.652 INFO:teuthology.orchestra.run.trial148.stderr:sudo: dnf: command not found
2026-04-13T19:06:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1
2026-04-13T19:06:13.654 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/task/pexec.py", line 149, in task
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/task/pexec.py", line 62, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 485, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_426ec63bc4a39bba882efb593125294667afc593/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on trial148 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
which was done because these dnf commands were pulled from rocky10.yaml from the kclient overrides but ubuntu_latest was used for the random distro.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
A better approach would be to include centos9 OR rocky10 for
distribution choice. Then we can just filter out rocky10 when we're
testing the `testing` kernel but keep rocky10 coverage for other
testing.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
2025-11-18T19:46:46.226 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stderr Error ENOTSUP: Module 'alerts' is not enabled/loaded (required by command 'dashboard set-ssl-certificate'): use `ceph mgr module enable alerts` to enable it
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Currently the code recreates these NAT rules for every mount. This only
needs to be done once by the first mount.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
rocky.10 does not support iptables with MASQUERADE targets. (Or maybe it
does with more prodding but it's easier to just switch to nft.)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
mgr/DaemonServer: Limit search for OSDs to upgrade within the crush bucket
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
qa/cephadm: ignore transient CEPHADM_FAILED_DAEMON in smoke-singlehost
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
admin_socket tests can still fail on non-default branches
due to git.ceph.com wget timeouts. Use raw URLs instead.
Fixes: https://tracker.ceph.com/issues/75969
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
The encoder test tries to download reef and squid packages, but
these do not support rocky10. So, we should remove that distro
from the supported distros- only distros that all the involved
releases have in common should be tested.
Rocky10 will only be backported as far as tentacle.
Fixes: https://tracker.ceph.com/issues/75763
Signed-off-by: Laura Flores <lflores@ibm.com>
The feedback module hard-coded tracker.ceph.com as the Ceph issue
tracker hostname. This meant the dashboard test
test_issue_tracker_create_with_invalid_key actually performed an
HTTPS POST against the live production tracker on every CI run.
Whenever tracker.ceph.com returned an unexpected response (5xx,
gateway error, transient failure), the test would fail on
completely unrelated pull requests.
Add a runtime module option 'mgr/feedback/tracker_url' that
controls which host the CephTrackerClient talks to. The default
keeps the existing behaviour (tracker.ceph.com). The dashboard
test now sets this option to an unreachable hostname in setUpClass
so create_issue fails fast with a ConnectionError instead of
depending on tracker.ceph.com being reachable.
Combined with the prior commit's error-handling fix, the test now
deterministically verifies that any tracker failure is surfaced as
HTTP 400, without requiring network access.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
the rgw suite is showing a handful of these false positives, and i
really don't want to micromanage suppressions for every little thing
that shows up here
so i chose to generalize the existing suppressions from
https://tracker.ceph.com/issues/74604 so that a single suppression would
cover this entire class of MismatchedFree warnings
Fixes: https://tracker.ceph.com/issues/75945
Signed-off-by: Casey Bodley <cbodley@redhat.com>
test_cephadm.sh hardcodes IMAGE_DEFAULT to ceph:main, which breaks
every stable branch whenever main is renamed to a new release. The
mismatch check in cephadm correctly rejects the container because its
release name doesn't match cephadm's own release. This has recurred on
every release transition (squidโtentacle, quincyโreef) without a fix.
Instead of always pulling ceph:main, derive IMAGE_DEFAULT from the
installed cephadm's version output. On stable builds (release type
"stable"), use ceph:<release> so the container matches cephadm. On dev
builds (main branch), fall back to ceph:main as before. The IMAGE_DEFAULT
env var can still be set externally to override.
Fixes: https://tracker.ceph.com/issues/75821
Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
The crash and status mgr modules return positive errno values (e.g.
errno.EINVAL) on error. However, the ceph CLI (src/ceph.in) only
treats negative return codes as errors:
if ret < 0:
final_ret = -ret
A positive return code falls through and the CLI exits with 0, masking
the error. This causes commands like "ceph crash post" and "ceph crash
info <bad-id>" to return success (rc=0) despite failing, with the
error message only appearing on stderr.
This was observed in the wild and worked around in a77b47eeeb, which added a
stderr != "" check to ceph-crash's post_crash() as a mitigation.
Fix the root cause by negating the errno values in both modules,
consistent with the HandleCommandResult convention
(retval: "E.g. 0 or -errno.EINVAL") and the 154 other call sites
across mgr modules that already use negative errno.
Also add error-path tests for the crash module to cover the three
affected commands (info, post, archive).
See-also: a77b47eeeb
Fixes: https://tracker.ceph.com/issues/75937
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Adapting the orch apply cmd to use the new pool/group arguments
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Fixes: https://tracker.ceph.com/issues/75934
Bootstrap fails on v20.2.0 upgrade because of
cephadm binary and ceph image version mismatch.
This fixes following problem in bootstrap by using
tentacle cephadm binary:
```
Error: Container release tentacle != cephadm release umbrella; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)
```
Fixes: https://tracker.ceph.com/issues/75912
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>
This is normal to see during upgrades as we mark OSDs down to
upgrade them.
Fixes: https://tracker.ceph.com/issues/75415
Signed-off-by: Laura Flores <lflores@ibm.com>
Increase op_delay to pggrow to avoid rapid PG splits.
Excessive splitting with a low reactor count can leave many PGs in
snaptrim, causing tests to hit the (short) snap trimming timeout.
Crimson's pggrow keeps the OSDs clean thorugout the entire test,
which is against do_thrash expectations.
Increasing op_delay would reduce do_thrash "actions" back to a normal rate.
Fixes: https://tracker.ceph.com/issues/75273
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Under Centos 9 the Java 8 version is recognized by the substring
"java-1.8" rather than "java-8". So the grep has been modified to
accept either.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit a49d4446e4)
This adds sts qa tests of rgw to crimson-rados suite
Fixes: https://tracker.ceph.com/issues/74027
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
This adds singleton tests of rgw to crimson. In classical OSD both
ubuntu and centos build are supported. However the crimson OSD only
provide centos build so cannot symlink all the files directly.
Instead of linking entire subdirectories the crimson suite uses
selective symlinks for the required yaml files.
Fixes: https://tracker.ceph.com/issues/73947
Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
The $() notation not only calls the function, it also creates subshell,
which is a separate process. Additionally in debug mode all the content
of the function get_asok_path is printed out in the logs each time
whenever it is called, for example:
...stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path
...stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=
...stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']'
...stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir
...stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']'
...stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.48301
...stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.48301/$cluster-$name.asok'
Instead of calling get_asok_path each time we need to define osd.0,
etc. asok file path, we just predefine corresponding variables.
It not only avoids extra resource usage, but also removes a lot of
noise from the logs.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>
Previously s3tests_java.py set JAVA_HOME using the `alternatives`
command. That had issues in that `alternatives` is not present on all
Ubuntu systems, and some installations of Java don't update
alternatives. So instead we look for a "java-8" jvm in /usr/lib/jvm/
and set JAVA_HOME to the first one we find.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
When graph walks is complete, it will actually be possible link to
supported without repeating the links. The matrix implementation doesn't
support this.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
The behavior of the 'ok-to-upgrade' command is now more deterministic with
respect to the parameters passed.
To achieve the above, the commit implements the following changes:
1. The 'ok-to-upgrade' command is modified to operate strictly on the OSDs
within the CRUSH bucket and, if possible, meet the '--max' criteria when
specified. When --max <num> is provided, the command returns up to <num>
OSD IDs from the specified CRUSH bucket that can be safely stopped for
simultaneous upgrade. This is useful when only a subset of OSDs within
the bucket needs to be upgraded for performance or other reasons.
2. Modifies the standalone tests to reflect the above change.
3. Modifies the relevant documentation to reflect the change in behavior.
Fixes: https://tracker.ceph.com/issues/75681
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
The upgrade from tentacle builds were failing because the system was attempting
to install packages whose names were changed in Umbrella.
We use the same workaround previously used for squid.
Fixes: https://tracker.ceph.com/issues/75730
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
This commit ensure that we only pass --objectstore argument to
cephadm's add/apply OSD command only when the value is not the
default value, bluestore.
This is done to ensure older ceph releases, like Squid and Tentacle
do not fail, as --objectstore argument was only added in Umbrella.
Fixes: https://tracker.ceph.com/issues/75731
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
egrep and fgrep are deprecated and emit warnings on modern systems:
egrep: warning: egrep is obsolescent; using grep -E
Replace all occurrences of egrep with grep -E and fgrep with grep -F
across shell scripts, Python files, and YAML configs. These are
POSIX-defined equivalences with identical behavior.
Fixes: https://tracker.ceph.com/issues/75776
Signed-off-by: Lumir Sliva <lumir.sliva@firma.seznam.cz>
All the qa tasks in the umbrella release checklist.
This includes removing X-3 releases and adding X-1 releases
(where appropriate).
Also updated release-checklist.rst
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
All the qa tasks in the umbrella release checklist
Signed-off-by: Chris Harris <harriscr@uk.ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
# Conflicts:
# qa/suites/upgrade/telemetry-upgrade/tentacle-x/1-tasks.yaml
# qa/workunits/test_telemetry_tentacle.sh
# qa/workunits/test_telemetry_tentacle_x.sh
avoid relying on "ubuntu_latest" and "rpm_latest" symlinks, which change
over time on main. be explicit about the distro versions supported by
the initial release
Signed-off-by: Casey Bodley <cbodley@redhat.com>
it's currently broken with newer python on rocky 10 and ubuntu 24
(tracked in https://tracker.ceph.com/issues/72500) and doesn't provide
interesting test coverage outside of rgw/upgrade
Signed-off-by: Casey Bodley <cbodley@redhat.com>
It's aggravating to have "$" in a file name. It's not necessary since it
already contains "$" in the directory itself.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
https://github.com/ceph/ceph/pull/66055 replaced centos_latest with
rpm_latest but didn't add centos_9 back. we want to test both
Signed-off-by: Casey Bodley <cbodley@redhat.com>
ceph_osds() assumes all teuthology scratch devices are clean
and directly usable by:
> ceph orch apply osd --all-available-devices --method raw
However, in practice some devices may retain stale BlueStore metadata
(or other data) from previous runs. cephadm correctly skips such
devices, leading to fewer OSDs than expected and causing the test to
timeout waiting for an exact OSD count.
This change adds a pre-deployment cleanup step for `raw-osds` that:
- zaps any existing BlueStore metadata (`ceph-bluestore-tool zap-device`)
- removes filesystem signatures (`wipefs --all`)
- clears initial disk data (`dd`)
This ensures all scratch devices are truly available for OSD
deployment and prevents mismatches between expected and actual OSD
counts.
Fixes: https://tracker.ceph.com/issues/75218
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
* refs/pull/63859/head:
qa/workunits/mgr: account for nvmeof module being "always-on"
mgr, qa: clarify module checks in DaemonServer
mgr, qa: add `pending_modules` to asock command
mgr, common, qa, doc: issue health error after max expiration is exceeded
mgr: ensure that all modules have started before advertising active mgr
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Rocky Linux 10 logs SELinux AVCs for systemd BPF operations during container startup due to incomplete SELinux policy coverage. These AVCs occur in permissive mode, are reproducible without Ceph, and do not indicate functional failure. Tests should ignore this specific AVC class while continuing to fail on enforced denials.
Signed-off-by: David Galloway <david.galloway@ibm.com>
