* refs/pull/70212/head:
qa/cephfs-mirror: Fix tests for ISO sync_time_stamp
doc/cephfs-mirroring: document ISO-8601 sync timestamps
cephfs_mirror: format sync timestamps as ISO-8601 local time
tools/cephfs-mirror: report epoch time in last snap sync time
Reviewed-by: Karthik U S <karthik.u.s1@ibm.com>
mgr/volumes: accept unit with subvolume resize
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Christopher Hoffman <choffman@redhat.com>
On the systemd-resolved (debian) path, dnsmasq has only address= records
and no upstream, so queries it can't answer locally are forwarded via
/etc/resolv.conf which points back at systemd-resolved, which routes
the cname domains back to dnsmasq. The resulting loop burns a 5s
resolver retry per round trip which manifests itself in rgw s3tests
on Ubuntu running too slow hitting the job timeout.
Set no_resolv on that path so dnsmasq answers authoritatively and never
forwards. The rpm/NetworkManager path is unchanged, as dnsmasq there is
the system resolver and must forward upstream.
Fixes: https://tracker.ceph.com/issues/78572
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
librdkafka submodule
Vendors librdkafka v2.12.1 as a submodule (default) and teaches the
notification teuthology task to set up Kafka 4.x KRaft brokers. The
distro-shipped librdkafka 1.6.1 cannot SCRAM-auth against Kafka 4.x
brokers (#75900); the bundled version fixes that. Build with
WITH_SYSTEM_RDKAFKA=ON to opt back into the distro library (floor 2.6.1).
follwinf changes needed with teh submodule:
* build kafka_stub only in case of system librdkafka
* use the same header location in submodule and system
* supress compilation warning to avoid -WError issues
2.12.1 matches Fedora(44)'s current librdkafka-devel, so the vendored copy
tracks a version that distro packagers have already vetted rather than
whichever release happens to be latest upstream.
The teuthology version matrix picks one of {3.9.2, 4.1, 4.2, 4.3} per run.
3.9.2 covers the last Zookeeper-based release for the migration cohort.
4.1/4.2/4.3 track Apache's current three supported minor lines and are
resolved to the latest patch at test time via dlcdn.apache.org/kafka/, so
new patches within a supported minor need no PR update. A new minor line
(4.4) requires renaming one YAML.
* new java version (17) is needed for kafka 4.2 and up (we wil use it on
all version)
* kerberos task need to run before the kafka install task
After pulling this branch, fetch the new submodule:
git submodule update --init src/librdkafka
Fixes: https://tracker.ceph.com/issues/77336
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com>
Block-manager users were denied access to the NVMe/TCP page because
host listing APIs require hosts read permission.
Fixes: https://tracker.ceph.com/issues/77952
Signed-off-by: Abhishek Desai <adesai@redhat.com>
This adds a Kerberos setup task, tells the Kafka task to emit GSSAPI listeners and broker principals, and updates notification tests to pass Kerberos credentials through to the bucket-notification suite. It enables the RGW Kafka notification coverage to run against GSSAPI-secured brokers.
Signed-off-by: sujay-d07 <sujaydongre07@gmail.com>
* refs/pull/68813/head:
tools/cephfs: make journal pointer/header failure explicit
doc/cephfs: add note explainin the new batching technique
tools/cephfs: document --max-rss usage in first-damage.py
qa: test cephfs-journal-recovery event recover_dentries summary
tools/cephfs: flush every journal event as soon as it is scanned
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
smb: add a teuthology test that exercises the gRPC over TCP interface of smb remote-control
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Rabinarayan Panigrahi <rapanigr@redhat.com>
Ensure that the testdir (parent dir of generated tls certs)
is made available to the smb test configuration used by
the workunit tests.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Update template.py jinja2 templating, to add support for extracting the
content of tls certs, generated by the ssl_keys task.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The dashboard backend API suite occasionally fails in CI on the job's
7200 second timeout alone, with every test passing. Suite runtime
varies with builder load: identical runs of the same 311 tests took
anywhere from about 5400 to over 7000 seconds, and teardown needs
roughly another 180 seconds on top. On the runs that landed at the
high end, only 60 to 80 seconds of margin were left before the
timeout fired. Cutting setup overhead gives the suite that margin
back.
DashboardTestCase.setUpClass() restarted every mgr twice: once in
MgrTestCase.setup_mgrs(), and again right after in _assign_ports(),
whose only job is to set the per-mgr dashboard port, which has to
happen while the mgrs are down.
Let subclasses declare their module ports in a MODULE_PORTS class
attribute instead. setup_mgrs() assigns them while the daemons are
already stopped, so the single restart it performs picks them up and
the second stop/fail/restart cycle disappears.
The assignment loop moves to _assign_module_ports(). _assign_ports()
keeps its old stop/assign/restart contract for callers that reassign
ports outside class setup (test_prometheus, test_dashboard,
test_module_selftest) and delegates to the shared helper.
With 66 dashboard test classes per API suite run this removes one
full mgr bounce per class, roughly 10 to 15 minutes of wall clock in
the CI runs examined.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
UI has an async validator which calls the GET bucket API to make sure
the bucket name doesn't exist, but the proxy
was not properly handling the http_status_codes which results in raising
a massive traceback in logs whenever you type things in the bucket name
field. So handling that gracefully by capturing the proper status codes
for both RequestException and DashboardException
BEFORE
```
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in _call_
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get
result = self.proxy(daemon_name, 'GET', 'bucket', {'bucket': bucket})
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 213, in proxy
raise DashboardException(e, http_status_code=http_status_code, component='rgw')
dashboard.exceptions.DashboardException: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000f14e08c1af0d5615-006'
b'71f54a7-3bc6-default","HostId":"3bc6-default-default"}')
2024-10-28T09:08:55.990+0000 7f89e045b640 0 [dashboard INFO request] [::ffff:10.74.18.122:51853] [GET] [500] [0.007s] [admin] [200.0B] /api/rgw/bucket/bucket-das
```
AFTER
```
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.services.exception] Dashboard Exception: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000c029a81e5d154844-006'
b'a4e183c-14251-default","HostId":"14251-default-default"}')
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.tools] [::ffff:192.168.100.1:60408] [GET] [404] [0.078s] [admin] [200.0B] /api/rgw/bucket/testsa
```
Fixes: https://tracker.ceph.com/issues/78038
Signed-off-by: Nizamudeen A <nia@redhat.com>
setup_ca_signed_keys() hardcoded `systemctl restart sshd`, which fails
on Debian/Ubuntu where the SSH server unit is named `ssh.service`
rather than `sshd.service`. Try `ssh` first, fall back to `sshd` if
that fails, grouped so the fallback only fires on restart failure and
doesn't mask a failed config write.
Fixes: https://tracker.ceph.com/issues/77991
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454
Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
In TestFSCryptVolumes, make sure to use mount class to
compare directory trees and to read/write files.
Fixes: https://tracker.ceph.com/issues/74110
Signed-off-by: Christopher Hoffman <choffman@redhat.com>
* refs/pull/66558/head:
qa/cephfs: minor fix in comment
qa/cephfs: give more time to tests in test_clone_stats.py
qa/cephfs: increase number of files to cloned in test_clone_stats.py
volumes/stats_util: improve log messages
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Skip influx selftests when the Python influxdb client is not installed.
Remove the ineffective testhost workaround and ignore the expected
MGR_INFLUX_NO_SERVER warning when the module runs without a server.
Fixes: https://tracker.ceph.com/issues/77250
Signed-off-by: Kobi Ginon <kginon@redhat.com>
python 3.12 warns on unrecognized escape sequences in string literals.
fuse_mount's admin-socket pyscript and mount's nft payload carry regex
and shell escapes (\., \d, \;) that are meant literally, so make those
strings raw. filesystem's get_mds_addr docstring had a stray \/ in the
example address, which is just a slash, so drop the backslash.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
Getting it from our own service avoids exposing us to
supply chain attacks in our QA infrastructure.
Fixes: https://tracker.ceph.com/issues/76927
Signed-off-by: Laura Flores <lflores@ibm.com>
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Fixes: https://tracker.ceph.com/issues/76686
Signed-off-by: Kotresh HR <khiremat@redhat.com>
this task was manually editing /etc/resolv.conf to configure dns names.
on ubuntu 24, these changes were getting clobbered by NetworkManager or
systemd service restarts
instead, manage dnsmasq depending on host distro. rpm-based distros use
NetworkManager's dnsmasq plugin, while deb-based distros use
systemd-resolved to point at the local dnsmasq
Fixes: https://tracker.ceph.com/issues/76961
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Verify counter dump registration, current-sync gauges while syncing
(full/delta), last-sync and summary counters after idle sync, and
extend mirror stats and remote-snap failure tests for per-directory
snaps_* and dir_state.
Fixes: https://tracker.ceph.com/issues/73457
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Verify that reloading the mirroring module and removing a directory
does not leave a ghost replayer entry that keeps syncing snapshots.
Fixes: https://tracker.ceph.com/issues/77398
Signed-off-by: Kotresh HR <khiremat@redhat.com>
