gian/ceph-mirror
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2026-08-02 07:03:18 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
93ff14d091
ceph-mirror/doc/rados/operations/erasure-code.rst
Anthony D'Atri 93c3f67031
Merge pull request #67779 from anthonyeleven/egregrioustypo 
doc/rados/operations: Fix typos in erasure-code.rst
2026-07-04 10:15:30 -04:00

577 lines
19 KiB
ReStructuredText
Raw Blame History

.. _ecpool:
==============
Erasure code
==============
By default, Ceph :ref:`rados_pools` are created with the type "replicated". In
replicated-type pools, every object is copied to multiple disks. This
multiple copying is the method of data protection known as "replication".
By contrast, `erasure-coded <https://en.wikipedia.org/wiki/Erasure_code>`_
pools use a method of data protection that is different from replication. In
erasure coding, data is broken into fragments of two kinds: data blocks and
parity blocks. If a drive fails or becomes corrupted, the parity blocks are
used to rebuild the data. At scale, erasure coding saves space relative to
replication.
In this documentation, data blocks are referred to as "data chunks"
and parity blocks are referred to as "coding chunks".
Erasure codes are also called "forward error correction codes". The
first forward error correction code was developed in 1950 by Richard
Hamming at Bell Laboratories.
Creating a sample erasure-coded pool
------------------------------------
The simplest erasure-coded pool is similar to `RAID5
<https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5>`_ and
requires at least three hosts:
.. prompt:: bash $
ceph osd pool create ecpool erasure
::
pool 'ecpool' created
.. prompt:: bash $
echo ABCDEFGHI | rados --pool ecpool put NYAN -
rados --pool ecpool get NYAN -
::
ABCDEFGHI
Erasure-code profiles
---------------------
The default erasure-code profile can sustain the overlapping loss of two OSDs
without losing data. This erasure-code profile is equivalent to a replicated
pool of size three, but with different storage requirements: instead of
requiring 3TB to store 1TB, it requires only 2TB to store 1TB. The default
profile can be displayed with this command:
.. prompt:: bash $
ceph osd erasure-code-profile get default
::
k=2
m=2
plugin=isa
crush-failure-domain=host
technique=reed_sol_van
.. note::
The profile just displayed is for the *default* erasure-coded pool, not the
*simplest* erasure-coded pool. These two pools are not the same:
The default erasure-coded pool has two data chunks (K) and two coding chunks
(M). The profile of the default erasure-coded pool is "k=2 m=2".
The simplest erasure-coded pool has two data chunks (K) and one coding chunk
(M). The profile of the simplest erasure-coded pool is "k=2 m=1".
Choosing the right profile is important because the profile cannot be modified
after the pool is created. If you find that you need an erasure-coded pool with
a profile different than the one you have created, you must create a new pool
with a different (and presumably more carefully considered) profile. When the
new pool is created, all objects from the wrongly configured pool must be moved
to the newly created pool. There is no way to alter the profile of a pool after
the pool has been created.
However, you can change the *crush-failure-domain* without creating a new pool,
by changing the CRUSH rule of the pool using
``ceph osd pool set <pool-name> crush_rule <rule-name>``
as shown in :ref:`device_classes`
(you should ensure that the new CRUSH rule is identical to the old rule
in every way except the failure domain!).
Once you do that, ``ceph osd pool ls detail`` will still show as ``erasure``
profile the profile that you used for initial creation (which refers to the
old *crush-failure-domain*), but the CRUSH rule in effect will be the new
one, as the *crush-failure-domain* in the profile is only used during
initial creation of the pool.
The most important parameters of the profile are *K*, *M*, and
*crush-failure-domain* because they define the storage overhead and
the data durability. For example, if the desired architecture must
sustain the loss of two racks with a storage overhead of 67%,
the following profile can be defined:
.. prompt:: bash $
ceph osd erasure-code-profile set myprofile \
k=3 \
m=2 \
crush-failure-domain=rack
ceph osd pool create ecpool erasure myprofile
echo ABCDEFGHI | rados --pool ecpool put NYAN -
rados --pool ecpool get NYAN -
::
ABCDEFGHI
The *NYAN* object will be divided in three (*K=3*) and two additional
*chunks* will be created (*M=2*). The value of *M* defines how many
OSDs can be lost simultaneously without losing any data. The
*crush-failure-domain=rack* will create a CRUSH rule that ensures
no two *chunks* are stored in the same rack.
.. ditaa::
+-------------------+
name | NYAN |
+-------------------+
content | ABCDEFGHI |
+--------+----------+
|
|
v
+------+------+
+---------------+ encode(3,2) +-----------+
| +--+--+---+---+ |
| | | | |
| +-------+ | +-----+ |
| | | | |
+--v---+ +--v---+ +--v---+ +--v---+ +--v---+
name | NYAN | | NYAN | | NYAN | | NYAN | | NYAN |
+------+ +------+ +------+ +------+ +------+
shard | 1 | | 2 | | 3 | | 4 | | 5 |
+------+ +------+ +------+ +------+ +------+
content | ABC | | DEF | | GHI | | YXY | | QGC |
+--+---+ +--+---+ +--+---+ +--+---+ +--+---+
| | | | |
| | v | |
| | +--+---+ | |
| | | OSD1 | | |
| | +------+ | |
| | | |
| | +------+ | |
| +------>| OSD2 | | |
| +------+ | |
| | |
| +------+ | |
| | OSD3 |<----+ |
| +------+ |
| |
| +------+ |
| | OSD4 |<--------------+
| +------+
|
| +------+
+----------------->| OSD5 |
+------+
More information can be found in the :ref:`erasure-code-profiles`
documentation.
Erasure Coding with Overwrites
------------------------------
By default, erasure-coded pools work only with operations that
perform full RADOS object writes, for example, RGW.
Since Luminous, partial writes for an erasure-coded pool may be
enabled with a per-pool setting. This lets RBD, CephFS, and librados store
data in an erasure-coded pool:
.. prompt:: bash $
ceph osd pool set ec_pool allow_ec_overwrites true
This can be enabled only on a pool residing on BlueStore OSDs, since
BlueStore's checksumming is used during deep scrubs to detect bitrot
and other corruption. Using Filestore with EC overwrites is not only
unsafe, but it also results in lower performance compared to BlueStore.
Moreover, Filestore is deprecated and any Filestore OSDs in your cluster
should be migrated to BlueStore.
There is no downside to enabling EC overwrites, so it is best practice to
routinely do so.
RBD and CephFS are not fully supported in erasure-coded pools, instead
you must instruct them and their clients to store their data in an EC
pool and their metadata in a replicated pool. For RBD, this means using the
erasure-coded pool as the ``--data-pool`` during image creation:
.. prompt:: bash $
rbd create --size 1G --data-pool ec_pool replicated_pool/image_name
For CephFS, an erasure-coded pool can be set as the default data pool during
file system creation or via :ref:`file-layouts`.
.. _rados_ops_erasure_coding_optimizations:
Erasure Coding Optimizations
----------------------------
Since Tentacle, an erasure-coded pool may have optimizations enabled
with a per-pool setting. This improves performance for smaller I/Os and
eliminates padding, which can significantly reduce space amplification
and wasted capacity:
.. prompt:: bash $
ceph osd pool set ec_pool allow_ec_optimizations true
The optimizations will make an erasure code pool more suitable for use
with RBD or CephFS. For RGW workloads that have large objects that are read and
written sequentially there will be little benefit from these optimizations; but
RGW workloads with lots of very small objects or small random access reads will
see performance and capacity benefits.
This flag may be enabled for existing pools, and can be configured
to default for new pools using the central configuration option
:confval:`osd_pool_default_flag_ec_optimizations`. Once the flag has been
enabled for a pool it cannot be disabled because it changes how new data is
stored.
The flag cannot be set unless all the Monitors and OSDs have been
upgraded to Tentacle or later. Optimizations can be enabled and used without
upgrading gateways and clients.
Optimizations are currently only supported with the Jerasure and ISA-L plugins
when using the ``reed_sol_van`` technique (these are the old and current
defaults and are the most widely used plugins and technique). Attempting to
set the flag for a pool using an unsupported combination of plugin and
technique is blocked with an error message.
The default stripe unit is 4K which works well for standard EC pools.
For the majority of I/O workloads it is recommended to increase the stripe
unit to at least 16K when using optimizations. Performance testing
shows that 16K is the best choice for general purpose I/O workloads. Increasing
this value will significantly improve small read performance but will slightly
reduce the performance of small sequential writes. For I/O workloads that are
predominately reads, larger values up to 256KB will further improve read
performance but will further reduce the performance of small sequential writes.
Values larger than 256KB are unlikely to have any performance benefit. The
stripe unit is a pool create-time option that can be set in the erasure code
profile or by setting the central configuration option
:confval:`osd_pool_erasure_code_stripe_unit`. The stripe unit cannot be changed
after the pool has been created, so if enabling optimizations for an existing
pool you will not get the full benefit of the optimizations.
Without optimizations enabled, the choice of ``k+m`` in the erasure code profile
affects performance. The higher the values of ``k`` and ``m`` the lower the
performance will be. With optimizations enabled there is only a very slight
reduction in performance as ``k`` increases so this makes using a higher value
of ``k`` more viable. Increasing ``m`` still impacts write performance,
especially for small writes, so for block and file workloads a value of ``m``
no larger than 3 is recommended.
Erasure-coded pool overhead
---------------------------
The overhead factor (space amplification) of an erasure-coded pool
is ``(k+m) / k``. For a ``4+2`` profile, the overhead is
thus 1.5, which means that 1.5 GiB of underlying storage is used to store
1 GiB of user data. Contrast with default replication with ``size=3``, with
which the space amplification factor is ``3.0``. Do not mistake erasure coding for a free
lunch: there is a significant performance tradeoff, especially when using HDDs
and when performing cluster recovery or backfill.
Below is a table showing the overhead factors for various values of ``k`` and ``m``.
As ``k`` increases above ``4``, the incremental space amplification gain
experiences diminishing returns but the performance impact grows proportionally.
We recommend that you do not choose a profile with ``k`` > ``4` or ``m`` > 2 unless
and until you fully understand the ramifications, including the number of
failure domains your cluster topology presents. If you choose ``m=1``,
expect data unavailability during maintenance and data loss when component
failures overlap. Profiles with ``m=1`` are strongly discouraged for
production data.
Deployments that must remain active and avoid data loss when larger numbers
of overlapping component failure must be survived may favor a value of ``m`` > 2.
Note that such profiles may result in lower space efficiency and lessened performance, especially
during backfill and recovery.
If you are certain that you wish to use erasure coding for one or more pools but
are not certain which profile to use, select ``4+2`` or ``6+3``. You will realize
double the usable space compared to replication with ``size=3`` with relatively
tolerable write and recovery performance impact.
.. note:: For a discussion of how durability varies across EC profiles, consult
`this Cephalocon presentation <https://www.youtube.com/watch?v=0y8WMu-hMBQ>`_ .
.. note:: Most erasure-coded pool deployments require at least ``k+m`` CRUSH failure
domains, which in most cases means racks or hosts. There are
operational advantages to planning EC profiles and cluster topology
so that there are at least ``k+m+1`` failure domains. In most cases
a value of ``k`` > 8 is discouraged.
.. note:: CephFS and RGW deployments with a significant proportion
of very small user files/objects may wish to plan carefully as
erasure-coded data pools can result in considerable additional space
amplification. Both CephFS and RGW support multiple data pools
with different media, performance, and data protection strategies,
which can enable efficient and effective deployments. An RGW
deployment might for example provision a modest complement of
TLC SSDs used by replicated index and default bucket data pools,
and a larger complement of erasure-coded QLC SSDs or HDDs to which
larger and colder objects are directed via storage class, placement
target, or Lua scripting.
.. list-table:: Erasure coding overhead
:widths: 4 4 4 4 4 4 4 4 4 4 4 4
:header-rows: 1
:stub-columns: 1
* -
- m=1
- m=2
- m=3
- m=4
- m=5
- m=6
- m=7
- m=8
- m=9
- m=10
- m=11
* - k=1
- 2.00
- 3.00
- 4.00
- 5.00
- 6.00
- 7.00
- 8.00
- 9.00
- 10.00
- 11.00
- 12.00
* - k=2
- 1.50
- 2.00
- 2.50
- 3.00
- 3.50
- 4.00
- 4.50
- 5.00
- 5.50
- 6.00
- 6.50
* - k=3
- 1.33
- 1.67
- 2.00
- 2.33
- 2.67
- 3.00
- 3.33
- 3.67
- 4.00
- 4.33
- 4.67
* - k=4
- 1.25
- 1.50
- 1.75
- 2.00
- 2.25
- 2.50
- 2.75
- 3.00
- 3.25
- 3.50
- 3.75
* - k=5
- 1.20
- 1.40
- 1.60
- 1.80
- 2.00
- 2.20
- 2.40
- 2.60
- 2.80
- 3.00
- 3.20
* - k=6
- 1.16
- 1.33
- 1.50
- 1.66
- 1.83
- 2.00
- 2.17
- 2.33
- 2.50
- 2.66
- 2.83
* - k=7
- 1.14
- 1.29
- 1.43
- 1.58
- 1.71
- 1.86
- 2.00
- 2.14
- 2.29
- 2.43
- 2.58
* - k=8
- 1.13
- 1.25
- 1.38
- 1.50
- 1.63
- 1.75
- 1.88
- 2.00
- 2.13
- 2.25
- 2.38
* - k=9
- 1.11
- 1.22
- 1.33
- 1.44
- 1.56
- 1.67
- 1.78
- 1.88
- 2.00
- 2.11
- 2.22
* - k=10
- 1.10
- 1.20
- 1.30
- 1.40
- 1.50
- 1.60
- 1.70
- 1.80
- 1.90
- 2.00
- 2.10
* - k=11
- 1.09
- 1.18
- 1.27
- 1.36
- 1.45
- 1.54
- 1.63
- 1.72
- 1.82
- 1.91
- 2.00
* - k=12
- 1.08
- 1.17
- 1.25
- 1.33
- 1.42
- 1.50
- 1.58
- 1.67
- 1.75
- 1.83
- 1.92
* - k=20
- 1.05
- 1.10
- 1.15
- 1.20
- 1.25
- 1.30
- 1.35
- 1.40
- 1.45
- 1.50
- 1.55
Erasure-coded pools and cache tiering
-------------------------------------
.. note:: Cache tiering was deprecated in Reef. We strongly advise not deploying new cache tiers, and working to remove them from existing deployments.
Erasure-coded pools require more resources than replicated pools and
lack some of the functionality supported by replicated pools (for example, omap
is not supported by default). To overcome these limitations, one can set up a
`cache tier <../cache-tiering>`_ before setting up the erasure-coded pool.
For example, if the pool *hot-storage* is made of fast storage, the following commands
will place the *hot-storage* pool as a tier of *ecpool* in *writeback*
mode:
.. prompt:: bash $
ceph osd tier add ecpool hot-storage
ceph osd tier cache-mode hot-storage writeback
ceph osd tier set-overlay ecpool hot-storage
The result is that every write and read to the *ecpool* actually uses
the *hot-storage* pool and benefits from its flexibility and speed.
More information can be found in the `cache tiering
<../cache-tiering>`_ documentation. Note, however, that cache tiering
is deprecated and may be removed completely in a future release.
Erasure-coded pool recovery
---------------------------
If an erasure-coded pool loses any data shards, it must recover them from others.
This recovery involves reading from the remaining shards, reconstructing the data, and
writing new shards.
In Octopus and later releases, erasure-coded pools can recover as long as there are at least *K* shards
available. (With fewer than *K* shards, you have actually lost data!)
Prior to Octopus, erasure-coded pools required that at least ``min_size`` shards be
available, even if ``min_size`` was greater than ``K``. This was a conservative
decision made out of an abundance of caution when designing the new pool
mode. As a result, however, pools with lost OSDs but without complete data loss were
unable to recover and go active without manual intervention to temporarily change
the ``min_size`` setting.
We recommend that ``min_size`` be ``K+1`` or greater to prevent loss of writes and
loss of data.
Glossary
--------
*chunk*
When the encoding function is called, it returns chunks of the same size as each other. There are two
kinds of chunks: (1) *data chunks*, which can be concatenated to reconstruct the original object, and
(2) *coding chunks*, which can be used to rebuild a lost chunk.
*K*
The number of data chunks into which an object is divided. For example, if *K* = 2, then a 10KB object
is divided into two objects of 5KB each.
*M*
The number of coding chunks computed by the encoding function. *M* is equal to the number of OSDs that can
be missing from the cluster without the cluster suffering data loss. For example, if there are two coding
chunks, then two OSDs can be missing without data loss.
Table of contents
-----------------
.. toctree::
:maxdepth: 1
erasure-code-profile
erasure-code-jerasure
erasure-code-isa
erasure-code-lrc
erasure-code-shec
erasure-code-clay
Reference in New Issue View Git Blame Copy Permalink