Commit Graph

735 Commits

Author SHA1 Message Date
Kefu Chai
80b9067286 blk,build: drop BlueStore PMEM support
BlueStore's PMEMDevice backend targets byte-addressable persistent
memory exposed as a DAX device, in practice Intel Optane DC persistent
memory. Intel Optane has been discontinued since 2021 and there is no
mainstream replacement, so the backend has effectively no hardware left
to run on. It was always experimental and has seen no functional change
since the DML/DSA offload landed in 2022; every commit to it since has
been a formatting sweep.

Keeping it builds in a chain of dependencies that nothing else needs:
libpmem, libndctl and libdaxctl, plus dml for the DSA offload that only
works on a Sapphire Rapids CPU. Remove the backend and everything tied
to it.

RBD persistent write-back cache (WITH_RBD_RWL) still uses PMDK's
libpmemobj, so the pmdk submodule, WITH_SYSTEM_PMDK, Buildpmdk and
Findpmdk stay. WITH_SYSTEM_PMDK now depends on WITH_RBD_RWL alone, and
Buildpmdk no longer builds ndctl support.

- remove the WITH_BLUESTORE_PMEM option, HAVE_BLUESTORE_PMEM and
  HAVE_LIBDML
- delete src/blk/pmem/PMEMDevice.{cc,h} and the pmem block_device_t
- drop the pmem value from the bdev_type option
- remove the --bluestore-pmem vstart flag
- drop the libpmem, libndctl and libdaxctl build deps from debian and
  rpm, keeping libpmemobj for RWL
- delete the Finddml, Findndctl and Finddaxctl cmake modules
- remove the DSA/DML and ndctl documentation

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-07-05 08:37:21 +08:00
Ronen Friedman
e552441619 build: add Mold linker compatibility
Mold handles several linker features differently from ld.bfd:

- --exclude-libs,ALL hides .symver-aliased symbols (e.g. rados_*)
  even when the version script lists them as global. Detect Mold
  (via USING_MOLD_LINKER) and disable --exclude-libs; version
  scripts already control symbol visibility.

- Duplicate symbols across static archives are treated as errors.
  Merge crimson-alien-common into crimson-alienstore as a single
  archive. For crimson-osd, link crimson-alienstore before
  crimson-common with --allow-multiple-definition so the alien
  thread's non-crimson definitions win (Mold picks first definition).

- Python/Cython extension builds (via Distutils.cmake) invoked the
  system default linker. Pass -fuse-ld= via MOLD_FUSE_LD_FLAG so
  extensions link with the configured linker.

- Add rados_* and _rados_* to librados.map global section for Mold
  compatibility with .symver aliases.

Co-authored-by: Mark Kogan <mkogan@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2026-06-28 14:53:54 +00:00
Kefu Chai
b16654474e
Merge pull request #69745 from sunyuechi/wip-fio-riscv-upstream
cmake: upgrade to upstream fio

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-06-27 06:33:09 +08:00
Kefu Chai
6a8a0a51b2
Merge pull request #69660 from tchaikov/wip-cmake-boost-with-asan
cmake/boost: build libboost_context with BOOST_USE_ASAN

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2026-06-26 22:00:46 +08:00
Sun Yuechi
216d468a1a cmake: build fio from upstream 3.42
fio-3.42 is the latest stable release. We switched to the ceph/fio fork in
10baab3fc8 to pick up a clang build fix; that
fix has since landed upstream, so switch back. This also gives riscv64
proper arch support (arch-riscv64.h, added in fio-3.36), which the fork's
fio-3.27-cxx lacks.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-06-26 15:17:17 +08:00
Kefu Chai
a1afaa6529 cmake/boost: build libboost_context with BOOST_USE_ASAN
The ceph-api job failed under ASan: radosgw-admin aborted at startup with a
heap-buffer-overflow in boost.context's fiber resume().

  ==1155842==ERROR: AddressSanitizer: heap-buffer-overflow ... READ of size 8
    #0 boost::context::detail::fiber_activation_record::resume() fiber_ucontext.hpp:153
    #10 RGWSI_Notify::do_start(optional_yield, DoutPrefixProvider const*) svc_notify.cc:261
  0x... is located 8 bytes after 1048-byte region allocated by
    boost::context::detail::fiber_activation_record_initializer()

fiber_activation_record carries three extra members (fake_stack, stack_bottom,
stack_size) only when BOOST_USE_ASAN is defined.  Under WITH_ASAN we build
boost.context with context-impl=ucontext and give consumers BOOST_USE_ASAN and
BOOST_USE_UCONTEXT through Boost::context's INTERFACE_COMPILE_DEFINITIONS, but
we never pass BOOST_USE_ASAN to b2.  So libboost_context, which compiles
fiber_activation_record_initializer() and allocates the record, uses the
smaller no-ASan layout, while consumers that include the header (here rgw's
RGWSI_Notify::do_start via boost::asio::spawn) compile resume() with the larger
layout and read stack_bottom/stack_size past the allocation.

Pass define=BOOST_USE_ASAN to b2 so the library and its consumers agree on the
struct layout.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-06-24 06:44:18 +08:00
Kefu Chai
2c9b73894b cmake: factor the ASan/LSan test options into cache variables
add_ceph_test() spelled out the suppression-file paths and sanitizer flags
inline.  bin/ceph needs the same options, so lift them into CEPH_ASAN_OPTIONS
and CEPH_LSAN_OPTIONS and have add_ceph_test() consume those.  The environment
the tests run with is unchanged.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-06-23 23:02:05 +08:00
Kefu Chai
ca44481cdf
Merge pull request #69622 from sunyuechi/boost
cmake: define BOOST_USE_UCONTEXT tree-wide under ASan

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2026-06-23 07:52:23 +08:00
Kefu Chai
002d35cbcc
Merge pull request #69601 from sunyuechi/wip-test-gtest-parallel-cache
cmake: make GTEST_PARALLEL_COMMAND visible to all test directories

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-06-22 18:54:32 +08:00
Sun Yuechi
5166ec6f06 cmake: define BOOST_USE_UCONTEXT tree-wide under ASan
Under WITH_ASAN Boost.Context is built ucontext-only, so consumers that
include its headers without linking Boost::context (e.g. libosd) were
still built for the fcontext backend and broke the link:

    mold: error: undefined symbol: boost::context::detail::make_fcontext

Define the backend tree-wide so every consumer agrees on it.

riscv64's ASan runtime mis-handles makecontext/swapcontext, so the
ucontext fiber backend reports false-positive heap-buffer-overflows on
fiber switch that can't be suppressed. So exclude it.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-06-22 13:26:12 +08:00
Sun Yuechi
5d82294f3f cmake/boost: load context Jamfile before passing context-impl to b2
With WITH_ASAN, b2 runs as `b2 context-impl=ucontext headers stage` for the
build step and `b2 context-impl=ucontext install` for the install step. The
`context-impl` feature is declared in libs/context/build/Jamfile.v2, which
neither the headers/stage nor the install targets load, so b2 aborts with:

    error: unknown feature "<context-impl>"

Name the context project as a target in both commands so its Jamfile (and the
feature) loads before the build request is expanded.

This works around https://github.com/boostorg/context/issues/297, fixed
upstream in
12ac945158
and first released in Boost 1.88; drop it once the bundled Boost is bumped to
1.88 or newer.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-06-20 15:41:11 +08:00
Sun Yuechi
0705c6d60b cmake: make GTEST_PARALLEL_COMMAND visible to all test directories
GTEST_PARALLEL_COMMAND was set as an ordinary variable inside the
`if(NOT TARGET gtest-parallel_ext)` guard, so it only existed in the
first directory that include()s AddCephTest (src/common/options). Later
includes skip the guarded block and leave it empty, so PARALLEL
unittests under src/test silently ran serially.

Promote it to CACHE INTERNAL so it is visible across all directories
regardless of include order.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-06-19 23:09:22 +08:00
Kefu Chai
6cc2fc7a45
Merge pull request #69165 from sunyuechi/wip-addcephtest-catch2-imported-target
cmake/AddCephTest: use namespaced Catch2 imported targets

Reviewed-by: Jesse F. Williamson <jfw@ibm.com>
2026-06-09 07:37:28 +08:00
Sun Yuechi
6af1a85946 cmake: add WITH_SYSTEM_SPDK to link a system-installed SPDK
By default ceph builds the bundled src/spdk fork via BuildSPDK. Add a
WITH_SYSTEM_SPDK option that instead locates a distro-provided SPDK
through a new Findspdk.cmake (pkg-config based, modelled on
Finddpdk.cmake), exposing the same spdk::spdk target.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-05-31 08:33:11 +08:00
Sun Yuechi
66a775ae90 cmake/AddCephTest: use namespaced Catch2 imported targets
AddCephTest.cmake links the bare target names Catch2 / Catch2WithMain.
With WITH_SYSTEM_CATCH2=ON, CPM resolves Catch2 via find_package(),
which only exports the namespaced IMPORTED targets Catch2::Catch2 /
Catch2::Catch2WithMain. CMake then treats the bare names as plain
library names and the link fails with -lCatch2WithMain, since the
physical library is named libCatch2Main (OUTPUT_NAME "Catch2Main").

Use the namespaced names. Catch2 exports them as ALIASes in the bundled
(CPM subproject) build too, so the default path keeps working.

Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
2026-05-29 18:30:10 +08:00
Kefu Chai
d6d88f70dd
Merge pull request #69068 from tchaikov/wip-bump-arrow-submodule
rgw: bump Apache Arrow submodule from 17.0.0 to 19.0.1

Reviewed-by: Justin Caratzas <jcaratza@ibm.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2026-05-27 08:05:25 +08:00
Kefu Chai
bb3f03d0f8
Merge pull request #68258 from tchaikov/wip-with-system-jerasure
cmake: support building with system jerasure and gf-complete

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2026-05-26 17:53:02 +08:00
Kefu Chai
355872641a rgw: bump Apache Arrow submodule from 17.0.0 to 19.0.1
When WITH_SYSTEM_ARROW is false, Ceph builds Arrow from the bundled
src/apache submodule. Our CI uses ubuntu:jammy as the base image, which
does not package libarrow-dev, so the bundled path is always taken there.

Arrow 17.0.0 vendors a copy of Thrift whose download URLs are no longer
reachable, breaking CI builds that try to fetch them at configure time.

Bump arrow submodule to 19.0.1, the latest Arrow release that:
- builds successfully on ubuntu:jammy, and
- requires only CMake 3.22 (the version shipped by ubuntu:jammy)

See also

CMake version shipped by ubuntu:jammy
- https://packages.ubuntu.com/jammy/cmake

arrow releases' CMake support
- maint-19.0.1: 272715f6df/cpp/CMakeLists.txt (L18)
- maint-20.0.0: 3ad0370a04/cpp/CMakeLists.txt (L18)

arrow enabled minmalloc by default
-
b907c5dadb

Because arrow uses the bundled mialloc library be default, we need
to disable it in the same commit bumping up the submodule.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-05-24 18:57:04 +08:00
Maodi Ma
e941e89318 common: enable AVX512+VPCLMULQDQ for crc32c performance on x86
- Add crc32_iscsi_by16_10 in src/isa-l into candidates for ceph_crc32c
- Add hardware capability check for AVX512 instr before register
- Add NASM feature check to ensure compatibility and to enable
  AS_FEATURE_LEVEL in crc32_iscsi_by16_10.asm

Signed-off-by: Maodi Ma <mamaodi@hygon.cn>
2026-05-21 17:30:13 +08:00
Kefu Chai
625f6baeb0 cmake/BuildISAL: build and install library targets only
Skip building the igzip executables; Ceph only needs libisal.la.
This should speed up the build a little bit, as we don't build the
executables previous built with "make"

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-05-17 20:13:00 +08:00
John Mulligan
288f8f081a
Merge pull request #68401 from phlogistonjohn/jjm-pypkg
build: Update python packaging for src/python-common

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-05-11 13:10:50 -04:00
Jamie Pryde
91e97bf3d1
Merge pull request #68711 from jamiepryde/isal-arm-build-typo
cmake: Fix ISA-L build on arm
2026-05-08 14:52:54 +01:00
John Mulligan
42ddb5eafd cmake/modules: add PythonPackage.cmake
Time is marching on and the state of the art with python packaging has
not stood still. In Python 3.12, distutils has been removed after being
deprecated for a couple of versions. According to the Python Packaging
User Guide [1]: "However, `python setup.py` and the use of `setup.py` as a
command line tool are deprecated."

Currently, ceph provides a decent sized and growing library of python
code in `src/python-common/ceph`. It currently relies on `setup.py` and
the deprecated `python setup.py install` command. This change aims to be
the first step in moving toward a more contemporary approach so that we
don't get caught late when the older approaches really stop working.

Because ceph's primary diver of "build stuff" is CMake, there was an
existing `cmake/modules/Distutils.cmake` that invokes a `python setup.py
install` command. Rather than risk breaking older distros we add a new
`cmake/modules/PythonPackage.cmake` file that uses the PEP 517/518
[2][3] style

of packaging. I could not find some existing CMake support for this
so unfortunately I had to write this.

The approach taken is loosely based on what the rpm build process does.
It invokes pip's wheel subcommand to build a wheel (during the build
phase) and then uses pip to install the wheel to install the content
to the system.

A future commit will add conditional support for using this approach
in src/python-common.

[1] https://packaging.python.org/en/latest/discussions/setup-py-deprecated/
[2] https://peps.python.org/pep-0517/
[3] https://peps.python.org/pep-0518/

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2026-05-06 09:32:25 -04:00
Kefu Chai
03bed028a9
Merge pull request #68404 from ifed01/wip-fix-isal-build
cmake/isal-l: explicitly configure libdir to avoid wrong libisal.a

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-05-06 20:47:30 +08:00
Jamie Pryde
5bf5bc69dd cmake: Fix ISA-L build on arm
A typo in CFLAGS means we're passing an empty string to configure_cmd.
We are then overwriting the build environment CFLAGS with our empty string CFLAGS,
which can result in build failures in certain environments, as seen in the tracker.
This fix gets any build environment CFLAGS and appends the other flags
we want to use when building ISA-L 2.32.0

Fixes: https://tracker.ceph.com/issues/76389

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
2026-05-06 11:58:06 +01:00
Jamie Pryde
dfd1fa7d5e
Merge pull request #67749 from jamiepryde/update-isal-2.32.0
isa: Update ISA-L from 2.30.0 to 2.32.0
2026-04-17 14:07:36 +01:00
Kefu Chai
b13c74637b
Merge pull request #68154 from leiwen2025/rv64-crc32c
src/common: optimize crc32c using zbc extension for riscv64

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-04-16 14:55:48 +08:00
Igor Fedotov
679f93818a cmake/isal-l: explicitly configure libdir to avoid wrong libisal.a
location.

This is to fix the following libec_isa.si build error :

FAILED: lib/libec_isa.so
: && /usr/bin/g++-14 -fPIC -O3 -DNDEBUG
-Wl,--dependency-file=src/erasure-code/isa/CMakeFiles/ec_isa.dir/link.d
-shared -Wl,-soname,libec_isa.so -o lib/libec_isa.so
src/erasure-code/CMakeFiles/erasure_code_objs.dir/ErasureCode.cc.o
src/erasure-code/isa/CMakeFiles/ec_isa.dir/ErasureCodeIsa.cc.o
src/erasure-code/isa/CMakeFiles/ec_isa.dir/ErasureCodeIsaTableCache.cc.o
src/erasure-code/isa/CMakeFiles/ec_isa.dir/ErasureCodePluginIsa.cc.o
src/isa-l/install/lib/libisal.a  -ldl  /usr/lib64/librt.a  -lresolv
-Wl,--as-needed -latomic && :
/usr/lib64/gcc/x86_64-suse-linux/14/../../../../x86_64-suse-linux/bin/ld:
cannot find src/isa-l/install/lib/libisal.a: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

It looks like [under some circumstances?] the building procedure puts
resulting .a under src/isa-l/install/lib64 path which causes lookup
error .so linkage.

This patch enforces lib64 usage.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
2026-04-16 00:35:13 +03:00
Kefu Chai
e35375b673 cmake: add WITH_SYSTEM_JERASURE option for system jerasure and gf-complete
Add a cmake option to build ceph's erasure code plugins (ec_jerasure
and ec_shec) against system-provided jerasure and gf-complete libraries
instead of the vendored copies.

This allows downstream distributions to build with their own packaged
versions of these libraries. For instance, Debian has shipped
libjerasure-dev and libgf-complete-dev as build dependencies for ceph
since 2021, but the build always used the vendored copies because there
was no cmake option to use the system ones.

Using system libraries also lets distributions benefit from any
optimizations or fixes applied to their packages without having to
carry patches against ceph's vendored copies. For example, the vendored
gf-complete requires a downstream -O1 workaround to avoid emitting
SSE 4.1 instructions that crash on older CPUs. But with system libraries,
that becomes the library package's responsibility.

The option is OFF by default, preserving the current behavior of
building from vendored sources.

A new FindJerasure.cmake module locates the system jerasure and
gf-complete headers and libraries, and creates the Jerasure::jerasure
imported target with the same target name that the vendored build now
provides (see previous commit).

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-04-08 18:42:19 +08:00
WenLei
444b8c92be src/common: optimize crc32c using zbc extension for riscv64
Signed-off-by: WenLei <lei.wen2@zte.com.cn>
2026-04-07 17:25:48 +08:00
Alex Ainscow
dba51e2ad3 Updates for review
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
2026-03-30 12:53:15 +01:00
lvshuo2016
01dc12ad56 common,arch,cmake: add RISC-V crc32c support
This adds hardware-accelerated crc32c support for the RISC-V
architecture. It includes the feature implementation, necessary
CMake configuration, and plumbing in src/arch/riscv.c to correctly
detect and select the optimized instructions.

Signed-off-by: lvshuo2016 <lv.shuo@sanechips.com.cn>
2026-03-17 10:11:04 +08:00
Jamie Pryde
6930c42600 cmake: Add support for SVE instructions in ISA-L on aarch64
ISA-L 2.31.0 introduced new EC functions that use SVE instructions on aarch64.
This means that the assembler must support SVE instructions
to build for aarch64. This commit passes `-Wa,-march=armv8-a+sve` to the ISA-L
configure step so the external assembler can process the new .S and .c files.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
2026-03-12 15:05:26 +00:00
Kefu Chai
34cf65d12e
Merge pull request #67081 from adamemerson/wip-gcc16-clang21
Fixes for GCC 16 and Clang 21

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-03-05 17:32:49 +08:00
Aishwarya Mathuria
f29e98fb5b cmake: explicitly support and enable vptr sanitizer
As of Clang 21, -fsanitize=undefined no longer implies vptr.
This adds vptr as an explicit component and provides the necessary compile options to satisfy Seastar's build requirements.
See: c1060ea7d4 for more details

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
2026-03-02 11:24:44 +00:00
Matan Breizman
cb9e1dca58 cmake/modules/Builduring.cmake: add URING::uring as alias
Allow seastar submodule which expects URING:uring target name
to be imported.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2026-02-12 10:26:16 +00:00
Kefu Chai
91de0e1933 cmake: migrate Python module installation from setup.py to pip
Replace 'setup.py install' with 'pip install --use-pep517' to fix
Cython compilation failures and eliminate deprecation warnings.

Problem Statement:
The build process for Cython modules involves preprocessing .pyx files
(e.g., generating rbd_processed.pyx from rbd.pyx) and then cythonizing
with specific compiler_directives. The previous approach using separate
'setup.py build' and 'setup.py install' commands caused this failure:

```
Error compiling Cython file:
------------------------------------------------------------
...
        """
        name = cstr(name, 'name')
        cdef:
            rados_ioctx_t _ioctx = convert_ioctx(ioctx)
            char *_name = name
            librbd_progress_fn_t _prog_cb = &no_op_progress_callback
                                            ^
------------------------------------------------------------

rbd_processed.pyx:781:44: Cannot assign type 'int (*)(uint64_t, uint64_t, void *) except? -1' to 'librbd_progress_fn_t'. Exception values are incompatible. Suggest adding 'noexcept' to type 'int (uint64_t, uint64_t, void *) except? -1'.
```

This occurs because:
1. 'setup.py build build_ext' successfully preprocesses and cythonizes
   with compiler_directives from setup.py's cythonize() call
2. 'setup.py install' internally triggers a rebuild that:
   - Regenerates the preprocessed .pyx files
   - Re-runs cythonize() through Cython.Distutils.build_ext
   - Does NOT apply the compiler_directives from setup.py
   - Fails on the regenerated files missing required directives

New Options Explained:

`--use-pep517`:
  Addresses deprecation warning:
  ```
    DEPRECATION: Building 'rados' using the legacy setup.py bdist_wheel
    mechanism, which will be removed in a future version. pip 25.3 will
    enforce this behaviour change.
  ```
  Uses the modern PEP 517 build backend which:
  - Performs a single build pass with all compiler_directives applied
  - Prevents the implicit rebuild that caused CompileError
  - Future-proofs against pip 25.3+ which will require this

`--no-build-isolation`:
  Ensures that environment variables set by CMake are respected:
  - CC, LDSHARED (compiler toolchain)
  - CPPFLAGS, LDFLAGS (compilation flags)
  - CYTHON_BUILD_DIR, CEPH_LIBDIR (build paths)

  Without this flag, pip would create an isolated build environment
  that ignores these critical build settings.

`--no-deps`:
  Prevents pip from attempting to install Python dependencies listed
  in setup.py's install_requires. All dependencies are managed by
  CMake and the distribution's package manager, not pip.

`--ignore-installed`:
  Addresses installation error when DESTDIR is set:
  ```
    ERROR: Could not install packages due to an OSError: [Errno 13]
    Permission denied: '/usr/lib/python3/dist-packages/rados-2.0.0.egg-info'

    OSError: [Errno 18] Invalid cross-device link:
    '/usr/lib/python3/dist-packages/rados-2.0.0.egg-info' -> '/tmp/pip-uninstall-...'
  ```
  This error occurs because pip detects an existing system installation
  and tries to uninstall it before installing to DESTDIR. With
  --ignore-installed, pip skips the uninstall step and directly installs
  to the DESTDIR staging directory, which is the correct behavior for
  packaging.

Removed Options:

`--install-layout=deb`:
  This Debian-specific patch to 'setup.py install' is no longer needed.
  Modern pip automatically detects the distribution and uses the correct
  layout (dist-packages on Debian, site-packages on RPM distros).

`--single-version-externally-managed`:
  This option was specific to 'setup.py install' to prevent egg
  installation. With pip, this is handled automatically.

`--record /dev/null`:
  No longer needed as pip manages installation records internally.

`egg_info --egg-base`:
  Not needed with pip as metadata is generated automatically during
  the build process.

Not added option:
`--root-user-action=ignore`: not added
  In this change, we installing a python module using pip with
  `fakeroot` before packaging it. But pip warned:
  ```
  Error: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behavior with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  ```
  But we use fakeroot on purpose, this option could have been added  to
  silence this warning. But it is not available in all supported pip
  versions. see
  2e1112a814

New environmental variable:
`DEB_PYTHON_INSTALL_LAYOUT=deb` is conditionally applied when packaging
for debian-derivative distributions. As pip does not support
`--install-layout` option. Since debian patches pip so it installs Python
modules into /usr/local/lib instead of /usr/lib where debian dh_install
helper looks for the content to be packaged, so we have to enforce the
debian layout using the environmental variable.

Working Directory Change:

Changed from `CMAKE_CURRENT_SOURCE_DIR` to `CMAKE_CURRENT_BINARY_DIR` to
keep pip's temporary files and logs in the build directory rather than
polluting the source tree.

Additional Dependencies:

Since the build process uses pip and creates a wheel distribution,
we need to add `pip` and `wheel` Python modules as build dependencies.

Python moduels packaging:

- with `--use-pep517`, pip creates .dist-info directoires as per PEP-517
instead of .egg-info, so we need to package the new metadata directory.

Future Improvements

We considered implementing a custom `build_templates` command or using
setuptools' `sub_commands` mechanism to avoid regenerating `*_processed.pyx`
files on every build (tracking dependencies via file modification times or
hash-based checks). However, to keep `setup.py` simple and maintainable,
we've deferred this optimization for future work. The current solution
using `pip install --use-pep517` ensures correct builds without additional
complexity.

This solution works correctly for both Debian and RPM packaging workflows,
both of which use DESTDIR-based staged installations.

Fixes: 719b749846
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-01-29 08:45:50 +08:00
Adam C. Emerson
b1c6f95a57
cmake: Remove FindPython3.cmake
This is bundled with modern cmake and we no longer use Ubuntu Focal as
our minimum, so we can stop carrying this.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2026-01-27 11:57:28 -05:00
Kefu Chai
fadc2b1cc2 cmake: fix undefined PY_LDFLAGS in distutils_install_cython_module
The distutils_install_cython_module() function was using ${PY_LDFLAGS}
without defining it, causing the linker to fail with:

  /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld:
  cannot find -lrados: No such file or directory

This bug was introduced in commit d22734f6cb which changed:
  set(ENV{LDFLAGS} "-L${CMAKE_LIBRARY_OUTPUT_DIRECTORY}")
to:
  set(ENV{LDFLAGS} "${PY_LDFLAGS}")

However, PY_LDFLAGS was only defined in distutils_add_cython_module(),
not in distutils_install_cython_module(). This meant that during the
install phase, LDFLAGS was set to an empty string, and the linker
couldn't find librados.so and other Ceph libraries in the build
directory.

The bug was exposed by commit 719b749846
which changed rados.pxd to use C preprocessor conditionals (#ifdef
BUILD_DOC) instead of Cython's compile-time IF statements. This meant
the build now required proper linking during the install phase.

Fix by defining PY_LDFLAGS in distutils_install_cython_module():

  set(PY_LDFLAGS "${CMAKE_SHARED_LINKER_FLAGS} -L${CMAKE_LIBRARY_OUTPUT_DIRECTORY}")

This keeps CMAKE_SHARED_LINKER_FLAGS as a space-separated string and
appends the library directory flag, avoiding issues with semicolon
conversion.

Fixes: d22734f6cb
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2026-01-22 13:50:22 +08:00
Adam C. Emerson
1dfe804c84
build: Disable FindBoost for Boost's included cmake config
Boost has included this since 1.70 and CMake has deprecated the
non-config version since 3.30.

See also
https://cmake.org/cmake/help/latest/policy/CMP0167.html#policy:CMP0167

We enable CMP0167 (The `FindBoost` module is removed.) to force cmake
to use the installed Boost configuration files rather than its own
detection.

We also enable CMP0144 (`find_package()` uses upper-case
`<PACKAGENAME>_ROOT` variables.) to ensue that the `BOOST_ROOT`
parameter continues to function in the config-style `find_package`.

`BuildBoost.cmake` is updated to add the `Boost::headers` interface
target to match configured system boost (retaining the Boost::boost
alias).

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2026-01-14 23:54:08 -05:00
Kefu Chai
ced012c566
Merge pull request #66786 from sungjoon-koh/wip-rocksdb-fix-liburing-libs
cmake: fix liburing path passed to rocksdb

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
2026-01-09 08:17:46 +08:00
sungjoon_koh
6dae09de6a cmake: fix liburing path passed to rocksdb
Signed-off-by: sungjoon_koh <sungjoon_koh@linecorp.com>
2026-01-05 15:09:00 +09:00
T K Chandra Hasan
b7a7d99dfb
cmake: While building fio headers, reference to macro BITS_PER_LONG defined by the fio's build is not being used in our CMake based system for plugins.
Fixes: https://tracker.ceph.com/issues/74182
Signed-off-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
2025-12-17 14:06:05 +05:30
Seena Fallah
4468c4ef00 cmake: skip boost dependency on ALIAS executable targets
The current add_executable override in Boost does not support alias
targets. Although Ceph currently has no alias targets that are
affected by this limitation, addressing this issue now will benefit
future developments and personal projects.
This change enhances the robustness of the override logic, ensuring
compatibility with alias targets moving forward.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
2025-11-19 22:11:08 +01:00
Casey Bodley
58fe0aad9a
Merge pull request #65916 from edwinzrodriguez/ceph-cmake4
cmake: add cmake 4 support

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2025-10-21 11:07:42 -04:00
Kefu Chai
3aef9fc059
Merge pull request #65408 from tchaikov/wip-suppress-asan-warnings
qa/{asan,lsan}.supp: suppress Python module and libstdc++ false positives

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2025-10-21 08:18:45 +08:00
Edwin Rodriguez
802195ec97 cmake: add cmake 4 support
This change improves the CMAKE variable detection logic in do_cmake.sh
to allow users to override the CMAKE binary used for building.

Changes:
- Add conditional check to only set CMAKE if not already set
- Add detection for cmake version 4.x and later (prioritized)
- Maintain backward compatibility with cmake3 fallback
- Allow users to specify custom CMAKE path via environment variable
- Pass CMAKE_POLICY_VERSION_MINIMUM if set to submodule build steps for
    older modules

This enables users to use a specific cmake binary by setting the CMAKE
environment variable before running the script:
  CMAKE=/custom/path/to/cmake ./do_cmake.sh

Additionally, the script now automatically detects and uses cmake 4.x+
when available, falling back to cmake3 or cmake as needed. This supports
building Ceph with newer cmake versions while maintaining compatibility
with existing build environments.

Fixes: https://tracker.ceph.com/issues/73523
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
2025-10-20 17:02:57 -04:00
Casey Bodley
9d92dbb2cb cmake: BuildArrow.cmake uses bundled thrift if system version < 0.17
the bump to arrow 17.0.0 broke the ubuntu jammy builds with:

In file included from /usr/include/thrift/transport/TTransport.h:25,
                 from /usr/include/thrift/protocol/TProtocol.h:28,
                 from /usr/include/thrift/TBase.h:24,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_types.h:14,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_constants.h:10,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_constants.cpp:7:
/usr/include/thrift/transport/TTransportException.h:23:10: fatal error: boost/numeric/conversion/cast.hpp: No such file or directory
   23 | #include <boost/numeric/conversion/cast.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

when comparing the gcc command line with arrow-15.0.0, the following argument
is no longer present:
> -isystem /build/ceph-20.3.0-3402-gb2db4947/obj-x86_64-linux-gnu/boost/include

arrow 17.0.0 seems to assume that thrift doesn't depend on boost anymore. a
comment in https://github.com/apache/arrow/issues/32266 claims that
> we don't need Boost with system Thrift 0.17.0 or later

but our jammy builds are stuck with libthrift-0.16.0. to reenable jammy builds,
instruct Arrow's cmake to use its bundled thrift dependency if our system thrift
version is < 0.17.0

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2025-10-15 17:08:55 -04:00
Casey Bodley
dd868a9e53 cmake: Findthrift.cmake parses version string
when thrift headers are found, parse the version string into a
cmake variable and pass it to find_package_handle_standard_args

this will be needed by BuildArrow.cmake for some conditional logic

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2025-10-15 17:06:43 -04:00
Kefu Chai
2defde30e4 cmake/BuildArrow: Use AUTO mode for xsimd dependency detection
Arrow requires xsimd >= 9.0.1 according to arrow/cpp/thirdparty/versions.txt.
Previously, we unconditionally set -Dxsimd_SOURCE=BUNDLED, forcing the use
of Arrow's vendored xsimd regardless of system package availability.

This commit changes to -Dxsimd_SOURCE=AUTO, which allows Arrow's
resolve_dependency mechanism to automatically:
1. Try to find system xsimd package
2. Check if version >= 9.0.1
3. Use system version if found and sufficient
4. Fall back to bundled version otherwise

This reduces build time and dependencies on systems with sufficient xsimd,
while maintaining compatibility with older distributions.

Distribution availability:
- Ubuntu Noble (24.04): libxsimd-dev 12.1.1 (✓ will use system)
- Ubuntu Jammy (22.04): libxsimd-dev 7.6.0 (✗ will use bundled)
- Debian Trixie (13): libxsimd-dev 13.2.0 (✓ will use system)
- CentOS Stream 9: xsimd-devel 7.4.9 (✗ will use bundled)

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
2025-10-15 15:49:09 +08:00