mirror of
https://github.com/ceph/ceph
synced 2026-08-02 07:03:18 +00:00
Merge PR #66769 into tentacle
* refs/pull/66769/head: test/rgw/logging: run teuthology on erasure coded pool rgw/bucket-logging: support for EC pools rgw/logging: do not create empty temporary objects rgw/logging: deleteting the object holding the temp object name on cleanup rgw/logging: make sure source bucket is in the target's list rgw/logging: removed unused APIs from header rgw/logging: fix race condition when name update returns ECANCELED rgw/logging: add error message when log_record fails rgw/logging: rollover objects when conf changes rgw/logging: allow committing empty objects rgw/logging: verify http method exists rgw/logging: fix/remove/add bucket logging op names rgw/logging: refactor canonical_name() rgw/logging: fix canonical names rgw: RGWPostBucketLoggingOp uses yield context Reviewed-by: Anthony D Atri <anthony.datri@gmail.com> Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
This commit is contained in:
Patrick Donnelly
commit
fa5ecb2232
@ -4,6 +4,8 @@
|
||||
|
||||
>=20.0.0
|
||||
|
||||
* RGW: Bucket Logging suppports creating log buckets in EC pools.
|
||||
Implicit logging object commits are now performed asynchronously.
|
||||
* RADOS: leader monitor and stretch mode status are now included in the `ceph status` output.
|
||||
Related Tracker: https://tracker.ceph.com/issues/70406
|
||||
* RGW: The User Account feature introduced in Squid provides first-class support for
|
||||
|
||||
@ -306,6 +306,8 @@ values to their defaults or to a level suitable for normal operations.
|
||||
+--------------------------+-----------+--------------+
|
||||
| ``rgw notification`` | 1 | 5 |
|
||||
+--------------------------+-----------+--------------+
|
||||
| ``rgw bucket logging`` | 1 | 5 |
|
||||
+--------------------------+-----------+--------------+
|
||||
| ``javaclient`` | 1 | 5 |
|
||||
+--------------------------+-----------+--------------+
|
||||
| ``asok`` | 1 | 5 |
|
||||
|
||||
@ -54,6 +54,22 @@ them, regardless if enough time passed or if no more records are written to the
|
||||
object. Flushing will happen automatically when logging is disabled on a
|
||||
bucket, or its logging configuration is changed, or the bucket is deleted.
|
||||
|
||||
The process of adding a new log object to the log bucket is asynchronous when
|
||||
triggered by a log record being written. Consequently, the operation that
|
||||
generated the log is completed immediately and does not wait for the log object
|
||||
addition to finish; this addition occurs later.
|
||||
The log object is added to the log bucket immediately when flushed. Consequently,
|
||||
this action may result in temporary gaps in the log records within the bucket
|
||||
until any pending log objects are also added.
|
||||
|
||||
To check which log objects for a source bucket are currently pending addition to
|
||||
the log bucket, execute the following command:
|
||||
|
||||
.. prompt:: bash #
|
||||
|
||||
radosgw-admin bucket logging list --bucket <source bucket>
|
||||
|
||||
|
||||
Standard
|
||||
````````
|
||||
If the logging type is set to "Standard" (the default) the log records are
|
||||
|
||||
@ -645,7 +645,7 @@ Parameters
|
||||
Response Entities
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Response is XML encoded in the body of the request, in the following format:
|
||||
The response is XML encoded in the body of the request, in the following format:
|
||||
|
||||
::
|
||||
|
||||
@ -786,7 +786,7 @@ Parameters are XML encoded in the body of the request, in the following format:
|
||||
| | | between different source buckets writing log records to the same log bucket. | |
|
||||
+-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
|
||||
| ``LoggingType`` | String | The type of logging. Valid values are: | No |
|
||||
| | | ``Standard`` (default) all bucket operations are logged after being perfomed. | |
|
||||
| | | ``Standard`` (default) all bucket operations are logged after being performed. | |
|
||||
| | | The log record will contain all fields. | |
|
||||
| | | ``Journal`` only operations that modify and object are logged. | |
|
||||
| | | Will record the minimum subset of fields in the log record that is needed | |
|
||||
@ -796,6 +796,18 @@ Parameters are XML encoded in the body of the request, in the following format:
|
||||
| | | object added to the log bucket. Default is 3600 seconds (1 hour). | |
|
||||
+-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
|
||||
|
||||
Response Entities
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The response is XML encoded in the body of the request, only if a configuration change triggers flushing of the current logging object.
|
||||
In this case it will return the name of the flushed logging object in following format:
|
||||
|
||||
::
|
||||
|
||||
<PostBucketLoggingOutput xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<FlushedLoggingObject>string</FlushedLoggingObject>
|
||||
</PostBucketLoggingOutput>
|
||||
|
||||
|
||||
HTTP Response
|
||||
~~~~~~~~~~~~~
|
||||
@ -860,7 +872,7 @@ Syntax
|
||||
Response Entities
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Response header contains ``Last-Modified`` date/time of the logging configuration.
|
||||
The response header contains ``Last-Modified`` date/time of the logging configuration.
|
||||
Logging configuration is XML encoded in the body of the response, in the following format:
|
||||
|
||||
::
|
||||
@ -915,7 +927,7 @@ Flush Bucket Logging
|
||||
--------------------
|
||||
|
||||
Flushes logging object for a given source bucket (if not flushed, the logging objects are written lazily to the log bucket).
|
||||
Returns the name of the object that was flushed. An empty name will be returned if no object needs to be flushed.
|
||||
Returns the name of the object that was flushed. Flushing will happen even if the logging object is empty.
|
||||
|
||||
Syntax
|
||||
~~~~~~
|
||||
@ -927,7 +939,7 @@ Syntax
|
||||
Response Entities
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Response is XML encoded in the body of the request, in the following format:
|
||||
The response is XML encoded in the body of the request, in the following format:
|
||||
|
||||
::
|
||||
|
||||
|
||||
@ -24,6 +24,17 @@
|
||||
"output": {"shape": "PostBucketLoggingOutput"},
|
||||
"documentationUrl":"https://docs.ceph.com/docs/master/radosgw/s3/bucketops/#post-bucket-logging",
|
||||
"documentation":"<p>Flushes the logging objects of the buckets.</p>"
|
||||
},
|
||||
"PutBucketLogging":{
|
||||
"name":"PutBucketLogging",
|
||||
"http":{
|
||||
"method":"PUT",
|
||||
"requestUri":"/{Bucket}?logging"
|
||||
},
|
||||
"input":{"shape":"PutBucketLoggingRequest"},
|
||||
"output": {"shape": "PutBucketLoggingOutput"},
|
||||
"documentationUrl":"https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLogging.html",
|
||||
"documentation":"<p>Put bucket logging configuration on source bucket.</p>"
|
||||
},
|
||||
"GetUsageStats":{
|
||||
"name":"GetUsageStats",
|
||||
@ -402,6 +413,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"PutBucketLoggingOutput": {
|
||||
"type":"structure",
|
||||
"members":{
|
||||
"FlushedLoggingObject": {
|
||||
"shape":"FlushedLoggingObject",
|
||||
"documentation":"<p>Name of the pending logging object that was flushed.</p>"
|
||||
}
|
||||
}
|
||||
},
|
||||
"FlushedLoggingObject":{
|
||||
"type":"string"
|
||||
},
|
||||
|
||||
1
qa/suites/rgw/bucket-logging/pools/.qa
Symbolic link
1
qa/suites/rgw/bucket-logging/pools/.qa
Symbolic link
@ -0,0 +1 @@
|
||||
../.qa/
|
||||
3
qa/suites/rgw/bucket-logging/pools/erasure.yaml
Normal file
3
qa/suites/rgw/bucket-logging/pools/erasure.yaml
Normal file
@ -0,0 +1,3 @@
|
||||
overrides:
|
||||
rgw:
|
||||
ec-data-pool: true
|
||||
3
qa/suites/rgw/bucket-logging/pools/replicated.yaml
Normal file
3
qa/suites/rgw/bucket-logging/pools/replicated.yaml
Normal file
@ -0,0 +1,3 @@
|
||||
overrides:
|
||||
rgw:
|
||||
ec-data-pool: false
|
||||
@ -68,6 +68,7 @@ SUBSYS(rgw_flight, 1, 5)
|
||||
SUBSYS(rgw_lifecycle, 1, 5)
|
||||
SUBSYS(rgw_restore, 1, 5)
|
||||
SUBSYS(rgw_notification, 1, 5)
|
||||
SUBSYS(rgw_bucket_logging, 1, 5)
|
||||
SUBSYS(javaclient, 1, 5)
|
||||
SUBSYS(asok, 1, 5)
|
||||
SUBSYS(throttle, 1, 1)
|
||||
|
||||
@ -174,6 +174,7 @@ set(librgw_common_srcs
|
||||
driver/rados/groups.cc
|
||||
driver/rados/rgw_bucket.cc
|
||||
driver/rados/rgw_bucket_sync.cc
|
||||
driver/rados/rgw_bl_rados.cc
|
||||
driver/rados/rgw_cr_rados.cc
|
||||
driver/rados/rgw_cr_tools.cc
|
||||
driver/rados/rgw_d3n_datacache.cc
|
||||
|
||||
832
src/rgw/driver/rados/rgw_bl_rados.cc
Normal file
832
src/rgw/driver/rados/rgw_bl_rados.cc
Normal file
@ -0,0 +1,832 @@
|
||||
// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
|
||||
// vim: ts=8 sw=2 smarttab
|
||||
|
||||
#include "rgw_bucket_logging.h"
|
||||
#include "rgw_bl_rados.h"
|
||||
#include <chrono>
|
||||
#include <fmt/format.h>
|
||||
#include <future>
|
||||
#include <memory>
|
||||
#include <thread>
|
||||
#include <boost/algorithm/hex.hpp>
|
||||
#include <boost/asio/basic_waitable_timer.hpp>
|
||||
#include <boost/asio/executor_work_guard.hpp>
|
||||
#include <boost/asio/io_context.hpp>
|
||||
#include <boost/asio/spawn.hpp>
|
||||
#include <boost/context/protected_fixedsize_stack.hpp>
|
||||
#include "common/async/yield_context.h"
|
||||
#include "common/async/yield_waiter.h"
|
||||
#include "common/ceph_time.h"
|
||||
#include "common/dout.h"
|
||||
#include "cls/lock/cls_lock_client.h"
|
||||
#include "include/common_fwd.h"
|
||||
#include "include/function2.hpp"
|
||||
#include "librados/AioCompletionImpl.h"
|
||||
#include "services/svc_zone.h"
|
||||
#include "rgw_common.h"
|
||||
#include "rgw_perf_counters.h"
|
||||
#include "rgw_sal_rados.h"
|
||||
#include "rgw_zone_features.h"
|
||||
|
||||
|
||||
#define dout_subsys ceph_subsys_rgw_bucket_logging
|
||||
|
||||
namespace rgw::bucketlogging {
|
||||
|
||||
using commit_list_t = std::set<std::string>;
|
||||
|
||||
// use mmap/mprotect to allocate 128k coroutine stacks
|
||||
auto make_stack_allocator() {
|
||||
return boost::context::protected_fixedsize_stack{128*1024};
|
||||
}
|
||||
|
||||
const std::string COMMIT_LIST_OBJECT_NAME = "bucket_logging_global_commit_list";
|
||||
static const std::string TEMP_POOL_ATTR = "temp_logging_pool";
|
||||
|
||||
static std::string get_commit_list_name (const rgw::sal::Bucket* log_bucket,
|
||||
const std::string& prefix) {
|
||||
return fmt::format("{}/{}/{}/{}", log_bucket->get_tenant(),
|
||||
log_bucket->get_name(),
|
||||
log_bucket->get_bucket_id(), prefix);
|
||||
}
|
||||
|
||||
int add_commit_target_entry(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::RadosStore* store,
|
||||
const rgw::sal::Bucket* log_bucket,
|
||||
const std::string& prefix,
|
||||
const std::string& obj_name,
|
||||
const std::string& tail_obj_name,
|
||||
const rgw_pool& temp_data_pool,
|
||||
optional_yield y) {
|
||||
|
||||
auto& ioctx_logging = store->getRados()->get_logging_pool_ctx();
|
||||
std::string target_obj_name = get_commit_list_name (log_bucket, prefix);
|
||||
{
|
||||
librados::ObjectWriteOperation op;
|
||||
op.create(false);
|
||||
|
||||
bufferlist bl;
|
||||
bl.append(tail_obj_name);
|
||||
std::map<std::string, bufferlist> new_commit_list{{obj_name, bl}};
|
||||
op.omap_set(new_commit_list);
|
||||
|
||||
bufferlist pool_bl;
|
||||
encode(temp_data_pool, pool_bl);
|
||||
op.setxattr(TEMP_POOL_ATTR.c_str(), pool_bl);
|
||||
|
||||
auto ret = rgw_rados_operate(dpp, ioctx_logging, target_obj_name, std::move(op), y);
|
||||
if (ret < 0){
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to add logging object entry " << obj_name
|
||||
<< " to commit list object:" << target_obj_name
|
||||
<< ". ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
bufferlist empty_bl;
|
||||
std::map<std::string, bufferlist> new_commit_list{{target_obj_name, empty_bl}};
|
||||
|
||||
librados::ObjectWriteOperation op;
|
||||
op.create(false);
|
||||
op.omap_set(new_commit_list);
|
||||
|
||||
auto ret = rgw_rados_operate(dpp, ioctx_logging, COMMIT_LIST_OBJECT_NAME,
|
||||
std::move(op), y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to add entry " << target_obj_name
|
||||
<< " to global bucket logging list object: "
|
||||
<< COMMIT_LIST_OBJECT_NAME << ". ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: added entry " << target_obj_name
|
||||
<< " to global bucket logging object: "
|
||||
<< COMMIT_LIST_OBJECT_NAME << dendl;
|
||||
return 0;
|
||||
}
|
||||
|
||||
int list_pending_commit_objects(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::RadosStore* store,
|
||||
const rgw::sal::Bucket* log_bucket,
|
||||
const std::string& prefix,
|
||||
std::set<std::string>& entries,
|
||||
optional_yield y) {
|
||||
|
||||
auto& ioctx_logging = store->getRados()->get_logging_pool_ctx();
|
||||
std::string target_obj_name = get_commit_list_name (log_bucket, prefix);
|
||||
|
||||
entries.clear();
|
||||
|
||||
constexpr auto max_chunk = 1024U;
|
||||
std::string start_after;
|
||||
bool more = true;
|
||||
int rval;
|
||||
while (more) {
|
||||
librados::ObjectReadOperation op;
|
||||
std::set <std::string> entries_chunk;
|
||||
op.omap_get_keys2(start_after, max_chunk, &entries_chunk, &more, &rval);
|
||||
const auto ret = rgw_rados_operate(dpp, ioctx_logging, target_obj_name,
|
||||
std::move(op), nullptr, y);
|
||||
if (ret == -ENOENT) {
|
||||
// log commit list object was not created - nothing to do
|
||||
return 0;
|
||||
}
|
||||
if (ret < 0) {
|
||||
// TODO: do we need to check on rval as well as ret?
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to read logging commit list "
|
||||
<< target_obj_name << " ret: " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
entries.merge(entries_chunk);
|
||||
if (more) {
|
||||
start_after = *entries.rbegin();
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
class BucketLoggingManager : public DoutPrefixProvider {
|
||||
using Executor = boost::asio::io_context::executor_type;
|
||||
bool m_shutdown = false;
|
||||
static constexpr auto m_log_commits_update_period = std::chrono::milliseconds(10000); // 10s
|
||||
static constexpr auto m_log_commits_update_retry = std::chrono::milliseconds(1000); // 1s
|
||||
static constexpr auto m_log_commits_idle_sleep = std::chrono::milliseconds(5000); // 5s
|
||||
const utime_t m_failover_time = utime_t(m_log_commits_update_period*9); // 90s
|
||||
CephContext* const m_cct;
|
||||
static constexpr auto COOKIE_LEN = 16;
|
||||
const std::string m_lock_cookie;
|
||||
const std::string m_lock_name = "bl_mgr_lock";
|
||||
boost::asio::io_context m_io_context;
|
||||
boost::asio::executor_work_guard<Executor> m_work_guard;
|
||||
std::thread m_worker;
|
||||
const SiteConfig& m_site;
|
||||
rgw::sal::RadosStore* const m_rados_store;
|
||||
|
||||
private:
|
||||
|
||||
CephContext *get_cct() const override { return m_cct; }
|
||||
|
||||
unsigned get_subsys() const override { return dout_subsys; }
|
||||
|
||||
std::ostream& gen_prefix(std::ostream& out) const override {
|
||||
return out << "rgw bucket_logging: ";
|
||||
}
|
||||
|
||||
int parse_list_name(std::string list_obj_name, std::string &tenant_name,
|
||||
std::string &bucket_name, std::string &bucket_id,
|
||||
std::string &prefix) {
|
||||
// <tenant_name>/<bucket_name>/<bucket_id>/<prefix>
|
||||
size_t pstart = 0, pend = 0;
|
||||
pend =list_obj_name.find('/');
|
||||
if (pend == std::string::npos) {
|
||||
return -EINVAL;
|
||||
}
|
||||
tenant_name = list_obj_name.substr(pstart, pend);
|
||||
|
||||
pstart = pend + 1;
|
||||
pend =list_obj_name.find('/', pstart);
|
||||
if (pend == std::string::npos) {
|
||||
return -EINVAL;
|
||||
}
|
||||
bucket_name = list_obj_name.substr(pstart, pend - pstart);
|
||||
if (bucket_name.empty()) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
pstart = pend + 1;
|
||||
pend =list_obj_name.find('/', pstart);
|
||||
if (pend == std::string::npos) {
|
||||
return -EINVAL;
|
||||
}
|
||||
bucket_id = list_obj_name.substr(pstart, pend - pstart);
|
||||
if (bucket_id.empty()) {
|
||||
return -EINVAL;
|
||||
}
|
||||
prefix = list_obj_name.substr(pend + 1);
|
||||
|
||||
/*
|
||||
//TODO: Fix this
|
||||
std::regex pattern(R"(([^/]+)/([^/]+)/([^/]+)/(.*))");
|
||||
std::smatch matches;
|
||||
|
||||
if (std::regex_match(list_obj_name, matches, pattern)) {
|
||||
tenant_name = matches[1].str();
|
||||
bucket_name = matches[2].str();
|
||||
bucket_id = matches[3].str();
|
||||
prefix = matches[4].str();
|
||||
} else {
|
||||
return -EINVAL;
|
||||
}
|
||||
*/
|
||||
return 0;
|
||||
}
|
||||
|
||||
// read the list of commit lists from the global list object
|
||||
int read_global_logging_list(commit_list_t& commits, optional_yield y) {
|
||||
constexpr auto max_chunk = 1024U;
|
||||
std::string start_after;
|
||||
bool more = true;
|
||||
int rval;
|
||||
while (more) {
|
||||
librados::ObjectReadOperation op;
|
||||
commit_list_t commits_chunk;
|
||||
op.omap_get_keys2(start_after, max_chunk, &commits_chunk, &more, &rval);
|
||||
const auto ret = rgw_rados_operate(this, m_rados_store->getRados()->get_logging_pool_ctx(),
|
||||
COMMIT_LIST_OBJECT_NAME, std::move(op),
|
||||
nullptr, y);
|
||||
if (ret == -ENOENT) {
|
||||
// global commit list object was not created - nothing to do
|
||||
return 0;
|
||||
}
|
||||
if (ret < 0) {
|
||||
// TODO: do we need to check on rval as well as ret?
|
||||
ldpp_dout(this, 5) << "ERROR: failed to read bucket logging global commit list. error: "
|
||||
<< ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
commits.merge(commits_chunk);
|
||||
if (more) {
|
||||
start_after = *commits.rbegin();
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
class tokens_waiter {
|
||||
size_t pending_tokens = 0;
|
||||
DoutPrefixProvider* const dpp;
|
||||
ceph::async::yield_waiter<void> waiter;
|
||||
|
||||
public:
|
||||
class token{
|
||||
tokens_waiter* tw;
|
||||
public:
|
||||
token(const token& other) = delete;
|
||||
token(token&& other) : tw(other.tw) {
|
||||
other.tw = nullptr; // mark as moved
|
||||
}
|
||||
token& operator=(const token& other) = delete;
|
||||
token(tokens_waiter* _tw) : tw(_tw) {
|
||||
++tw->pending_tokens;
|
||||
}
|
||||
|
||||
~token() {
|
||||
if (!tw) {
|
||||
return; // already moved
|
||||
}
|
||||
--tw->pending_tokens;
|
||||
if (tw->pending_tokens == 0 && tw->waiter) {
|
||||
tw->waiter.complete(boost::system::error_code{});
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
tokens_waiter(DoutPrefixProvider* _dpp) : dpp(_dpp) {}
|
||||
tokens_waiter(const tokens_waiter& other) = delete;
|
||||
tokens_waiter& operator=(const tokens_waiter& other) = delete;
|
||||
|
||||
void async_wait(boost::asio::yield_context yield) {
|
||||
if (pending_tokens == 0) {
|
||||
return;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: tokens waiter is waiting on "
|
||||
<< pending_tokens << " tokens" << dendl;
|
||||
boost::system::error_code ec;
|
||||
waiter.async_wait(yield[ec]);
|
||||
ldpp_dout(dpp, 20) << "INFO: tokens waiter finished waiting for all tokens" << dendl;
|
||||
}
|
||||
};
|
||||
|
||||
// processing of a specific entry
|
||||
// return whether processing was successful (true) or not (false)
|
||||
int process_entry(
|
||||
const ConfigProxy& conf,
|
||||
const std::string& entry,
|
||||
const std::string& temp_obj_name,
|
||||
const std::string& tenant_name,
|
||||
const std::string& bucket_name,
|
||||
const std::string& bucket_id,
|
||||
const std::string& prefix,
|
||||
const rgw_pool& obj_pool,
|
||||
bool& bucket_deleted,
|
||||
boost::asio::yield_context yield) {
|
||||
|
||||
std::unique_ptr<rgw::sal::Bucket> log_bucket;
|
||||
rgw_bucket bucket = rgw_bucket{tenant_name, bucket_name, bucket_id};
|
||||
int ret = m_rados_store->load_bucket(this, bucket, &log_bucket, yield);
|
||||
if (ret < 0 && ret != -ENOENT) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to load bucket : "
|
||||
<< bucket_name << " id : " << bucket_id
|
||||
<< ", error: " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
if (ret == 0) {
|
||||
ldpp_dout(this, 20) << "INFO: loaded bucket : " << bucket_name
|
||||
<< ", id: "<< bucket_id << dendl;
|
||||
ret = log_bucket->commit_logging_object(entry, yield, this, prefix,
|
||||
nullptr, false);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to commit logging object : "
|
||||
<< entry << " , error: " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(this, 20) << "INFO: committed logging object : " << entry
|
||||
<< " to bucket " << bucket_name
|
||||
<< ", id: "<< bucket_id << dendl;
|
||||
return 0;
|
||||
}
|
||||
// The log bucket has been deleted. Clean up pending objects.
|
||||
// This is done because the bucket_deletion_cleanup only removes
|
||||
// the current temp log objects.
|
||||
bucket_deleted = true;
|
||||
ldpp_dout(this, 20) << "INFO: log bucket : " << bucket_name
|
||||
<< " no longer exists. Removing pending logging object "
|
||||
<< temp_obj_name << " from pool " << obj_pool
|
||||
<< dendl;
|
||||
ret = rgw_delete_system_obj(this, m_rados_store->svc()->sysobj, obj_pool,
|
||||
temp_obj_name, nullptr, yield);
|
||||
if (ret < 0 && ret != -ENOENT) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to delete pending logging object : "
|
||||
<< temp_obj_name << " in pool " << obj_pool
|
||||
<< " for deleted log bucket : " << bucket_name
|
||||
<< " . ret = "<< ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(this, 20) << "INFO: Deleted pending logging object "
|
||||
<< temp_obj_name << " from pool " << obj_pool
|
||||
<< " for deleted log bucket : " << bucket_name
|
||||
<< dendl;
|
||||
return 0;
|
||||
}
|
||||
|
||||
void async_sleep(boost::asio::yield_context yield, const std::chrono::milliseconds& duration) {
|
||||
using Clock = ceph::coarse_mono_clock;
|
||||
using Timer = boost::asio::basic_waitable_timer<Clock,
|
||||
boost::asio::wait_traits<Clock>, Executor>;
|
||||
Timer timer(m_io_context);
|
||||
timer.expires_after(duration);
|
||||
boost::system::error_code ec;
|
||||
timer.async_wait(yield[ec]);
|
||||
if (ec) {
|
||||
ldpp_dout(this, 10) << "ERROR: async_sleep failed with error: "
|
||||
<< ec.message() << dendl;
|
||||
}
|
||||
}
|
||||
|
||||
// unlock (lose ownership) list object
|
||||
int unlock_list_object(librados::IoCtx& rados_ioctx,
|
||||
const std::string& list_obj_name,
|
||||
boost::asio::yield_context yield) {
|
||||
|
||||
librados::ObjectWriteOperation op;
|
||||
op.assert_exists();
|
||||
rados::cls::lock::unlock(&op, m_lock_name, m_lock_cookie);
|
||||
const auto ret = rgw_rados_operate(this, rados_ioctx, list_obj_name,
|
||||
std::move(op), yield);
|
||||
if (ret == -ENOENT) {
|
||||
ldpp_dout(this, 20) << "INFO: log commit list: " << list_obj_name
|
||||
<< " was removed. Nothing to unlock." << dendl;
|
||||
return 0;
|
||||
}
|
||||
if (ret == -EBUSY) {
|
||||
ldpp_dout(this, 20) << "INFO: log commit list: " << list_obj_name
|
||||
<< " already owned by another RGW. No need to unlock."
|
||||
<< dendl;
|
||||
return 0;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
int remove_commit_list(librados::IoCtx& rados_ioctx,
|
||||
const std::string& commit_list_name,
|
||||
optional_yield y) {
|
||||
{
|
||||
librados::ObjectWriteOperation op;
|
||||
rados::cls::lock::assert_locked(&op, m_lock_name,
|
||||
ClsLockType::EXCLUSIVE,
|
||||
m_lock_cookie,
|
||||
"" /*no tag*/);
|
||||
op.remove();
|
||||
auto ret = rgw_rados_operate(this, rados_ioctx, commit_list_name, std::move(op), y);
|
||||
if (ret < 0 && ret != -ENOENT) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to remove bucket logging commit list :"
|
||||
<< commit_list_name << ". ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
librados::ObjectWriteOperation op;
|
||||
op.omap_rm_keys({commit_list_name});
|
||||
auto ret = rgw_rados_operate(this, rados_ioctx, COMMIT_LIST_OBJECT_NAME,
|
||||
std::move(op), y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to remove entry " << commit_list_name
|
||||
<< " from the global bucket logging list : "
|
||||
<< COMMIT_LIST_OBJECT_NAME << ". ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(this, 20) << "INFO: removed entry " << commit_list_name
|
||||
<< " from the global bucket logging list : "
|
||||
<< COMMIT_LIST_OBJECT_NAME << dendl;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
// processing of a specific target list
|
||||
void process_commit_list(librados::IoCtx& rados_ioctx,
|
||||
const std::string& list_obj_name,
|
||||
boost::asio::yield_context yield) {
|
||||
auto is_idle = false;
|
||||
auto bucket_deleted = false;
|
||||
auto skip_bucket_deletion = false;
|
||||
const std::string start_marker;
|
||||
|
||||
std::string tenant_name, bucket_name, bucket_id, prefix;
|
||||
int ret = parse_list_name(list_obj_name, tenant_name, bucket_name,
|
||||
bucket_id, prefix);
|
||||
if (ret) {
|
||||
ldpp_dout(this, 1) << "ERROR: commit list name: " << list_obj_name
|
||||
<< " is invalid. Processing will stop." << dendl;
|
||||
return;
|
||||
}
|
||||
ldpp_dout(this, 20) << "INFO: parse commit list name: " << list_obj_name
|
||||
<< ", bucket_name: " << bucket_name
|
||||
<< ", bucket_id: " << bucket_id
|
||||
<< ", tenant_name: " << tenant_name
|
||||
<< ", prefix: " << prefix << dendl;
|
||||
|
||||
while (!m_shutdown) {
|
||||
// if the list was empty the last time, sleep for idle timeout
|
||||
if (is_idle) {
|
||||
async_sleep(yield, m_log_commits_idle_sleep);
|
||||
}
|
||||
|
||||
// Get the entries in the list after locking it
|
||||
is_idle = true;
|
||||
skip_bucket_deletion = false;
|
||||
std::map<std::string, bufferlist> entries;
|
||||
rgw_pool temp_data_pool;
|
||||
auto total_entries = 0U;
|
||||
{
|
||||
librados::ObjectReadOperation op;
|
||||
op.assert_exists();
|
||||
bufferlist obl;
|
||||
int rval;
|
||||
bufferlist pool_obl;
|
||||
int pool_rval;
|
||||
constexpr auto max_chunk = 1024U;
|
||||
std::string start_after;
|
||||
bool more = true;
|
||||
rados::cls::lock::assert_locked(&op, m_lock_name,
|
||||
ClsLockType::EXCLUSIVE,
|
||||
m_lock_cookie,
|
||||
"" /*no tag*/);
|
||||
op.omap_get_vals2(start_after, max_chunk, &entries, &more, &rval);
|
||||
op.getxattr(TEMP_POOL_ATTR.c_str(), &pool_obl, &pool_rval);
|
||||
// check ownership and list entries in one batch
|
||||
auto ret = rgw_rados_operate(this, rados_ioctx, list_obj_name,
|
||||
std::move(op), nullptr, yield);
|
||||
if (ret == -ENOENT) {
|
||||
// list object was deleted
|
||||
ldpp_dout(this, 20) << "INFO: object: " << list_obj_name
|
||||
<< " was removed. Processing will stop" << dendl;
|
||||
return;
|
||||
}
|
||||
if (ret == -EBUSY) {
|
||||
ldpp_dout(this, 10) << "WARNING: commit list : " << list_obj_name
|
||||
<< " ownership moved to another daemon. Processing will stop"
|
||||
<< dendl;
|
||||
return;
|
||||
}
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 10) << "WARNING: failed to get list of entries in "
|
||||
<< "and/or lock object: " << list_obj_name
|
||||
<< ". error: " << ret << " (will retry)" << dendl;
|
||||
continue;
|
||||
}
|
||||
if(pool_rval < 0) {
|
||||
ldpp_dout(this, 10) << "WARNING: failed to get pool information"
|
||||
<< "from commit list: " << list_obj_name << dendl;
|
||||
return;
|
||||
}
|
||||
// Get temp logging object pool information - this is required for
|
||||
// deleting the objects when the log bucket is deleted.
|
||||
try {
|
||||
auto p = pool_obl.cbegin();
|
||||
decode(temp_data_pool, p);
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get pool information from object: "
|
||||
<< list_obj_name << dendl;
|
||||
return;
|
||||
}
|
||||
}
|
||||
total_entries = entries.size();
|
||||
if (total_entries == 0) {
|
||||
// nothing in the list object
|
||||
// Delete the list object if the bucket has been deleted
|
||||
std::unique_ptr<rgw::sal::Bucket> log_bucket;
|
||||
rgw_bucket bucket = rgw_bucket{tenant_name, bucket_name, bucket_id};
|
||||
int ret = m_rados_store->load_bucket(this, bucket, &log_bucket, yield);
|
||||
if (ret == -ENOENT) {
|
||||
ldpp_dout(this, 20) << "ERROR: failed to load bucket : "
|
||||
<< bucket_name << ", id : " << bucket_id
|
||||
<< ", error: " << ret << dendl;
|
||||
bucket_deleted = true;
|
||||
}
|
||||
} else {
|
||||
// log when not idle
|
||||
ldpp_dout(this, 20) << "INFO: found: " << total_entries
|
||||
<< " entries in commit list: " << list_obj_name
|
||||
<< dendl;
|
||||
}
|
||||
auto stop_processing = false;
|
||||
std::set<std::string> remove_entries;
|
||||
tokens_waiter tw(this);
|
||||
for (auto const& [key, val] : entries) {
|
||||
if (stop_processing) {
|
||||
break;
|
||||
}
|
||||
std::string temp_obj_name = val.to_str();
|
||||
|
||||
ldpp_dout(this, 20) << "INFO: processing entry: " << key
|
||||
<< ", value : " << temp_obj_name << dendl;
|
||||
|
||||
tokens_waiter::token token(&tw);
|
||||
boost::asio::spawn(yield, std::allocator_arg, make_stack_allocator(),
|
||||
[this, &is_idle, &list_obj_name, &remove_entries, &stop_processing,
|
||||
&tenant_name, &bucket_name, &bucket_id, &prefix, token = std::move(token),
|
||||
key, temp_obj_name, temp_data_pool, &bucket_deleted , &skip_bucket_deletion]
|
||||
(boost::asio::yield_context yield) {
|
||||
auto result =
|
||||
process_entry(this->get_cct()->_conf, key, temp_obj_name, tenant_name,
|
||||
bucket_name, bucket_id, prefix, temp_data_pool,
|
||||
bucket_deleted, yield);
|
||||
if (result == 0) {
|
||||
ldpp_dout(this, 20) << "INFO: processing of entry: " << key
|
||||
<< " from: " << list_obj_name
|
||||
<< " was successful." << dendl;
|
||||
remove_entries.insert(key);
|
||||
is_idle = false;
|
||||
return;
|
||||
} else {
|
||||
is_idle = true;
|
||||
// Don't delete the bucket if we couldn't process the entry
|
||||
// and the bucket has been deleted
|
||||
skip_bucket_deletion = true;
|
||||
ldpp_dout(this, 20) << "INFO: failed to process entry: " << key
|
||||
<< " from: " << list_obj_name << dendl;
|
||||
}
|
||||
stop_processing = true;
|
||||
}, [] (std::exception_ptr eptr) {
|
||||
if (eptr) std::rethrow_exception(eptr);
|
||||
});
|
||||
}
|
||||
|
||||
if (!entries.empty()) {
|
||||
// wait for all pending work to finish
|
||||
tw.async_wait(yield);
|
||||
}
|
||||
|
||||
if (bucket_deleted && !skip_bucket_deletion) {
|
||||
ret = remove_commit_list(rados_ioctx, list_obj_name, yield);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to remove bucket logging commit list :"
|
||||
<< list_obj_name << ". ret = " << ret << dendl;
|
||||
return;
|
||||
}
|
||||
ldpp_dout(this, 10) << "INFO: bucket :" << bucket_name
|
||||
<< ". was removed. processing will stop" << dendl;
|
||||
return;
|
||||
}
|
||||
// delete all processed entries from list
|
||||
if (!remove_entries.empty()) {
|
||||
// Delete the entries even if the lock no longer belongs
|
||||
// to this instance as we have processed the entries.
|
||||
librados::ObjectWriteOperation op;
|
||||
op.omap_rm_keys(remove_entries);
|
||||
auto ret = rgw_rados_operate(this, rados_ioctx, list_obj_name, std::move(op), yield);
|
||||
if (ret == -ENOENT) {
|
||||
// list was deleted
|
||||
ldpp_dout(this, 10) << "INFO: commit list " << list_obj_name
|
||||
<< ". was removed. processing will stop" << dendl;
|
||||
return;
|
||||
}
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to remove entries from commit list: "
|
||||
<< list_obj_name << ". error: " << ret
|
||||
<< ". This could cause some log records to be lost."
|
||||
<< dendl;
|
||||
return;
|
||||
} else {
|
||||
ldpp_dout(this, 20) << "INFO: removed entries from commit list: "
|
||||
<< list_obj_name << dendl;
|
||||
}
|
||||
}
|
||||
}
|
||||
ldpp_dout(this, 5) << "INFO: BucketLogging manager stopped processing : "
|
||||
<< list_obj_name << dendl;
|
||||
}
|
||||
|
||||
// process all commit log objects
|
||||
// find which of the commit log objects is owned by this daemon and process it
|
||||
void process_global_logging_list(boost::asio::yield_context yield) {
|
||||
auto has_error = false;
|
||||
std::unordered_set<std::string> owned_commit_logs;
|
||||
std::atomic<size_t> processed_list_count = 0;
|
||||
|
||||
std::vector<std::string> commit_list_gc;
|
||||
std::mutex commit_list_gc_lock;
|
||||
auto& rados_ioctx = m_rados_store->getRados()->get_logging_pool_ctx();
|
||||
auto next_check_time = ceph::coarse_real_clock::zero();
|
||||
while (!m_shutdown) {
|
||||
// check if log commit list needs to be refreshed
|
||||
if (ceph::coarse_real_clock::now() > next_check_time) {
|
||||
next_check_time = ceph::coarse_real_clock::now() + m_log_commits_update_period;
|
||||
const auto tp = ceph::coarse_real_time::clock::to_time_t(next_check_time);
|
||||
ldpp_dout(this, 20) << "INFO: processing global logging commit list. next "
|
||||
<< "processing will happen at: " << std::ctime(&tp)
|
||||
<< dendl;
|
||||
} else {
|
||||
// short sleep duration to prevent busy wait when refreshing list
|
||||
// or retrying after error
|
||||
ldpp_dout(this, 20) << "INFO: processing global logging commit list. Sleep now." << dendl;
|
||||
async_sleep(yield, m_log_commits_update_retry);
|
||||
if (!has_error) {
|
||||
// in case of error we will retry
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
std::set<std::string> commit_lists;
|
||||
auto ret = read_global_logging_list(commit_lists, yield);
|
||||
if (ret < 0) {
|
||||
has_error = true;
|
||||
ldpp_dout(this, 1) << "ERROR: failed to read global logging list. Retry. ret:"
|
||||
<< ret << dendl;
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const auto& list_obj_name : commit_lists) {
|
||||
// try to lock the object to check if it is owned by this rgw
|
||||
// or if ownership needs to be taken
|
||||
ldpp_dout(this, 20) << "INFO: processing commit list: " << list_obj_name
|
||||
<< dendl;
|
||||
librados::ObjectWriteOperation op;
|
||||
op.assert_exists();
|
||||
rados::cls::lock::lock(&op, m_lock_name, ClsLockType::EXCLUSIVE,
|
||||
m_lock_cookie, "" /*no tag*/,
|
||||
"" /*no description*/, m_failover_time,
|
||||
LOCK_FLAG_MAY_RENEW);
|
||||
|
||||
ret = rgw_rados_operate(this, rados_ioctx, list_obj_name, std::move(op), yield);
|
||||
if (ret == -EBUSY) {
|
||||
// lock is already taken by another RGW
|
||||
ldpp_dout(this, 20) << "INFO: commit list: " << list_obj_name
|
||||
<< " owned (locked) by another daemon" << dendl;
|
||||
// if the list was owned by this RGW, processing should be stopped, would be deleted from list afterwards
|
||||
continue;
|
||||
}
|
||||
if (ret == -ENOENT) {
|
||||
// commit list is deleted - processing will stop the next time we try to read from it
|
||||
ldpp_dout(this, 20) << "INFO: commit list: " << list_obj_name
|
||||
<< " should not be locked - already deleted" << dendl;
|
||||
continue;
|
||||
}
|
||||
if (ret < 0) {
|
||||
// failed to lock for another reason, continue to process other lists
|
||||
ldpp_dout(this, 10) << "ERROR: failed to lock commit list: "
|
||||
<< list_obj_name << ". error: " << ret << dendl;
|
||||
has_error = true;
|
||||
continue;
|
||||
}
|
||||
// add the commit list to the list of owned commit_logs
|
||||
if (owned_commit_logs.insert(list_obj_name).second) {
|
||||
ldpp_dout(this, 20) << "INFO: " << list_obj_name << " now owned (locked) by this daemon" << dendl;
|
||||
// start processing this list
|
||||
boost::asio::spawn(make_strand(m_io_context), std::allocator_arg, make_stack_allocator(),
|
||||
[this, &commit_list_gc, &commit_list_gc_lock, &rados_ioctx,
|
||||
list_obj_name, &processed_list_count](boost::asio::yield_context yield) {
|
||||
++processed_list_count;
|
||||
process_commit_list(rados_ioctx, list_obj_name, yield);
|
||||
// if list processing ended, it means that the list object was removed or not owned anymore
|
||||
const auto ret = unlock_list_object(rados_ioctx, list_obj_name, yield);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(this, 15) << "WARNING: failed to unlock commit list: "
|
||||
<< list_obj_name << " with error: " << ret
|
||||
<< " (ownership would still move if not renewed)"
|
||||
<< dendl;
|
||||
} else {
|
||||
ldpp_dout(this, 20) << "INFO: commit list: " << list_obj_name
|
||||
<< " not locked (ownership can move)" << dendl;
|
||||
}
|
||||
// mark it for deletion
|
||||
std::lock_guard lock_guard(commit_list_gc_lock);
|
||||
commit_list_gc.push_back(list_obj_name);
|
||||
--processed_list_count;
|
||||
ldpp_dout(this, 20) << "INFO: " << list_obj_name << " marked for removal" << dendl;
|
||||
}, [this, list_obj_name] (std::exception_ptr eptr) {
|
||||
ldpp_dout(this, 10) << "ERROR: " << list_obj_name << " processing failed" << dendl;
|
||||
if (eptr) std::rethrow_exception(eptr);
|
||||
});
|
||||
} else {
|
||||
ldpp_dout(this, 20) << "INFO: " << list_obj_name << " ownership (lock) renewed" << dendl;
|
||||
}
|
||||
}
|
||||
// erase all list objects that were deleted
|
||||
{
|
||||
std::lock_guard lock_guard(commit_list_gc_lock);
|
||||
std::for_each(commit_list_gc.begin(), commit_list_gc.end(), [this, &owned_commit_logs](const std::string& list_obj_name) {
|
||||
owned_commit_logs.erase(list_obj_name);
|
||||
ldpp_dout(this, 20) << "INFO: commit list object: " << list_obj_name << " was removed" << dendl;
|
||||
});
|
||||
commit_list_gc.clear();
|
||||
}
|
||||
}
|
||||
while (processed_list_count > 0) {
|
||||
ldpp_dout(this, 20) << "INFO: BucketLogging manager stopped. "
|
||||
<< processed_list_count
|
||||
<< " commit lists are still being processed" << dendl;
|
||||
async_sleep(yield, m_log_commits_update_retry);
|
||||
}
|
||||
ldpp_dout(this, 5) << "INFO: BucketLogging manager stopped. Done processing all commit lists" << dendl;
|
||||
}
|
||||
|
||||
public:
|
||||
|
||||
~BucketLoggingManager() {
|
||||
}
|
||||
|
||||
void stop() {
|
||||
ldpp_dout(this, 5) << "INFO: BucketLogging manager received stop signal. shutting down..." << dendl;
|
||||
m_shutdown = true;
|
||||
m_work_guard.reset();
|
||||
if (m_worker.joinable()) {
|
||||
// try graceful shutdown first
|
||||
auto future = std::async(std::launch::async, [this]() {m_worker.join();});
|
||||
if (future.wait_for(m_log_commits_update_retry*2) == std::future_status::timeout) {
|
||||
// force stop if graceful shutdown takes too long
|
||||
if (!m_io_context.stopped()) {
|
||||
ldpp_dout(this, 5) << "INFO: force shutdown of BucketLogging Manager" << dendl;
|
||||
m_io_context.stop();
|
||||
}
|
||||
m_worker.join();
|
||||
}
|
||||
}
|
||||
ldpp_dout(this, 5) << "INFO: BucketLogging manager shutdown complete" << dendl;
|
||||
}
|
||||
|
||||
void init() {
|
||||
boost::asio::spawn(make_strand(m_io_context), std::allocator_arg, make_stack_allocator(),
|
||||
[this](boost::asio::yield_context yield) {
|
||||
process_global_logging_list(yield);
|
||||
}, [] (std::exception_ptr eptr) {
|
||||
if (eptr) std::rethrow_exception(eptr);
|
||||
});
|
||||
// start the worker threads to do the actual list processing
|
||||
m_worker = std::thread([this]() {
|
||||
ceph_pthread_setname("bucket-logging-worker");
|
||||
try {
|
||||
ldpp_dout(this, 20) << "INFO: bucket logging worker started" << dendl;
|
||||
m_io_context.run();
|
||||
ldpp_dout(this, 20) << "INFO: bucket logging worker ended" << dendl;
|
||||
} catch (const std::exception& err) {
|
||||
ldpp_dout(this, 10) << "ERROR: bucket logging worker failed with error: "
|
||||
<< err.what() << dendl;
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
ldpp_dout(this, 10) << "INFO: started bucket logging manager" << dendl;
|
||||
}
|
||||
|
||||
BucketLoggingManager(CephContext* _cct, rgw::sal::RadosStore* store, const SiteConfig& site) :
|
||||
m_cct(_cct),
|
||||
m_lock_cookie(gen_rand_alphanumeric(m_cct, COOKIE_LEN)),
|
||||
m_work_guard(boost::asio::make_work_guard(m_io_context)),
|
||||
m_site(site),
|
||||
m_rados_store(store)
|
||||
{
|
||||
ldpp_dout(this, 5) << "INFO: BucketLoggingManager() constructor" << dendl;
|
||||
}
|
||||
};
|
||||
|
||||
std::unique_ptr<BucketLoggingManager> s_manager;
|
||||
|
||||
bool init(const DoutPrefixProvider* dpp, rgw::sal::RadosStore* store,
|
||||
const SiteConfig& site) {
|
||||
if (s_manager) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to init BucketLogging manager: already exists" << dendl;
|
||||
return false;
|
||||
}
|
||||
// TODO: take conf from CephContext
|
||||
ldpp_dout(dpp, 1) << "INFO: initialising BucketLogging manager" << dendl;
|
||||
s_manager = std::make_unique<BucketLoggingManager>(dpp->get_cct(), store, site);
|
||||
s_manager->init();
|
||||
return true;
|
||||
}
|
||||
|
||||
void shutdown() {
|
||||
if (!s_manager) return;
|
||||
s_manager->stop();
|
||||
s_manager.reset();
|
||||
}
|
||||
|
||||
} // namespace rgw::bucket_logging
|
||||
41
src/rgw/driver/rados/rgw_bl_rados.h
Normal file
41
src/rgw/driver/rados/rgw_bl_rados.h
Normal file
@ -0,0 +1,41 @@
|
||||
// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
|
||||
// vim: ts=8 sw=2 smarttab ft=cpp
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
|
||||
// forward declarations
|
||||
namespace rgw { class SiteConfig; }
|
||||
namespace rgw::sal {
|
||||
class RadosStore;
|
||||
}
|
||||
class DoutPrefixProvider;
|
||||
|
||||
namespace rgw::bucketlogging {
|
||||
|
||||
// initialize the bucket logging commit manager
|
||||
bool init(const DoutPrefixProvider* dpp, rgw::sal::RadosStore* store,
|
||||
const rgw::SiteConfig& site);
|
||||
|
||||
// shutdown the bucket logging commit manager
|
||||
void shutdown();
|
||||
|
||||
int add_commit_target_entry(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::RadosStore* store,
|
||||
const rgw::sal::Bucket* log_bucket,
|
||||
const std::string& prefix,
|
||||
const std::string& obj_name,
|
||||
const std::string& tail_obj_name,
|
||||
const rgw_pool& temp_data_pool,
|
||||
optional_yield y);
|
||||
|
||||
int list_pending_commit_objects(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::RadosStore* store,
|
||||
const rgw::sal::Bucket* log_bucket,
|
||||
const std::string& prefix,
|
||||
std::set<std::string>& entries,
|
||||
optional_yield y);
|
||||
|
||||
}
|
||||
|
||||
@ -57,6 +57,7 @@
|
||||
#include "rgw_etag_verifier.h"
|
||||
#include "rgw_worker.h"
|
||||
#include "rgw_notify.h"
|
||||
#include "rgw_bl_rados.h"
|
||||
#include "rgw_http_errors.h"
|
||||
|
||||
#undef fork // fails to compile RGWPeriod::fork() below
|
||||
@ -1144,6 +1145,10 @@ void RGWRados::finalize()
|
||||
v1_topic_migration.stop();
|
||||
}
|
||||
|
||||
if (run_bucket_logging_thread) {
|
||||
rgw::bucketlogging::shutdown();
|
||||
}
|
||||
|
||||
if (use_restore_thread) {
|
||||
restore->stop_processor();
|
||||
}
|
||||
@ -1271,6 +1276,10 @@ int RGWRados::init_complete(const DoutPrefixProvider *dpp, optional_yield y)
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
|
||||
ret = open_logging_pool_ctx(dpp);
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
|
||||
pools_initialized = true;
|
||||
|
||||
if (use_gc) {
|
||||
@ -1423,6 +1432,12 @@ int RGWRados::init_complete(const DoutPrefixProvider *dpp, optional_yield y)
|
||||
|
||||
index_completion_manager = new RGWIndexCompletionManager(this);
|
||||
|
||||
if (run_bucket_logging_thread) {
|
||||
if (!rgw::bucketlogging::init(dpp, this->driver, *svc.site)) {
|
||||
ldpp_dout(dpp, 0) << "ERROR: failed to initialize bucket logging manager" << dendl;
|
||||
}
|
||||
ldpp_dout(dpp, 0) << "INFO: initialized bucket logging manager" << dendl;
|
||||
}
|
||||
if (run_notification_thread) {
|
||||
if (!rgw::notify::init(dpp, driver, *svc.site)) {
|
||||
ldpp_dout(dpp, 0) << "ERROR: failed to initialize notification manager" << dendl;
|
||||
@ -1443,7 +1458,6 @@ int RGWRados::init_complete(const DoutPrefixProvider *dpp, optional_yield y)
|
||||
v1_topic_migration.start(1);
|
||||
}
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
@ -1547,6 +1561,11 @@ int RGWRados::open_notif_pool_ctx(const DoutPrefixProvider *dpp)
|
||||
return rgw_init_ioctx(dpp, get_rados_handle(), svc.zone->get_zone_params().notif_pool, notif_pool_ctx, true, true);
|
||||
}
|
||||
|
||||
int RGWRados::open_logging_pool_ctx(const DoutPrefixProvider *dpp)
|
||||
{
|
||||
return rgw_init_ioctx(dpp, get_rados_handle(), svc.zone->get_zone_params().bucket_logging_pool, logging_pool_ctx, true, true);
|
||||
}
|
||||
|
||||
int RGWRados::open_pool_ctx(const DoutPrefixProvider *dpp, const rgw_pool& pool, librados::IoCtx& io_ctx,
|
||||
bool mostly_omap, bool bulk)
|
||||
{
|
||||
|
||||
@ -364,6 +364,7 @@ class RGWRados
|
||||
int open_objexp_pool_ctx(const DoutPrefixProvider *dpp);
|
||||
int open_reshard_pool_ctx(const DoutPrefixProvider *dpp);
|
||||
int open_notif_pool_ctx(const DoutPrefixProvider *dpp);
|
||||
int open_logging_pool_ctx(const DoutPrefixProvider *dpp);
|
||||
|
||||
int open_pool_ctx(const DoutPrefixProvider *dpp, const rgw_pool& pool, librados::IoCtx& io_ctx,
|
||||
bool mostly_omap, bool bulk);
|
||||
@ -384,6 +385,7 @@ class RGWRados
|
||||
bool run_sync_thread{false};
|
||||
bool run_reshard_thread{false};
|
||||
bool run_notification_thread{false};
|
||||
bool run_bucket_logging_thread{false};
|
||||
|
||||
RGWMetaNotifier* meta_notifier{nullptr};
|
||||
RGWDataNotifier* data_notifier{nullptr};
|
||||
@ -459,6 +461,7 @@ protected:
|
||||
librados::IoCtx objexp_pool_ctx;
|
||||
librados::IoCtx reshard_pool_ctx;
|
||||
librados::IoCtx notif_pool_ctx; // .rgw.notif
|
||||
librados::IoCtx logging_pool_ctx; // .rgw.logging
|
||||
|
||||
bool pools_initialized{false};
|
||||
|
||||
@ -546,6 +549,11 @@ public:
|
||||
return *this;
|
||||
}
|
||||
|
||||
RGWRados& set_run_bucket_logging_thread(bool _run_bucket_logging_thread) {
|
||||
run_bucket_logging_thread = _run_bucket_logging_thread;
|
||||
return *this;
|
||||
}
|
||||
|
||||
librados::IoCtx* get_lc_pool_ctx() {
|
||||
return &lc_pool_ctx;
|
||||
}
|
||||
@ -561,6 +569,10 @@ public:
|
||||
librados::IoCtx& get_notif_pool_ctx() {
|
||||
return notif_pool_ctx;
|
||||
}
|
||||
|
||||
librados::IoCtx& get_logging_pool_ctx() {
|
||||
return logging_pool_ctx;
|
||||
}
|
||||
|
||||
void set_context(CephContext *_cct) {
|
||||
cct = _cct;
|
||||
|
||||
@ -41,6 +41,7 @@
|
||||
#include "rgw_aio_throttle.h"
|
||||
#include "rgw_bucket.h"
|
||||
#include "rgw_bucket_logging.h"
|
||||
#include "rgw_bl_rados.h"
|
||||
#include "rgw_lc.h"
|
||||
#include "rgw_lc_tier.h"
|
||||
#include "rgw_lc_tier.h"
|
||||
@ -1096,6 +1097,7 @@ int RadosBucket::remove_topics(RGWObjVersionTracker* objv_tracker,
|
||||
|
||||
|
||||
#define RGW_ATTR_COMMITTED_LOGGING_OBJ RGW_ATTR_PREFIX "committed-logging-obj"
|
||||
#define RGW_ATTR_LOGGING_EC_POOL RGW_ATTR_PREFIX "logging-ec-pool"
|
||||
|
||||
int get_committed_logging_object(RadosStore* store,
|
||||
const std::string& obj_name_oid,
|
||||
@ -1140,10 +1142,94 @@ int set_committed_logging_object(RadosStore* store,
|
||||
bufferlist bl;
|
||||
bl.append(last_committed);
|
||||
librados::ObjectWriteOperation op;
|
||||
// if object does not exist, we should not create the attribute
|
||||
op.assert_exists();
|
||||
op.setxattr(RGW_ATTR_COMMITTED_LOGGING_OBJ, std::move(bl));
|
||||
return rgw_rados_operate(dpp, io_ctx, obj_name_oid, std::move(op), y);
|
||||
}
|
||||
|
||||
int get_logging_temp_object_pool (const DoutPrefixProvider *dpp,
|
||||
RadosBucket* bucket,
|
||||
RadosStore* rados_store,
|
||||
const std::string& obj_name_oid,
|
||||
rgw_pool& data_pool, bool& is_ec_pool, optional_yield y) {
|
||||
|
||||
int ret;
|
||||
rgw_obj obj {bucket->get_key(), "dummy"};
|
||||
if (!rados_store->getRados()->get_obj_data_pool(bucket->get_placement_rule(), obj, &data_pool)) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << bucket->get_key() <<
|
||||
"' when writing logging object" << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
|
||||
// Find out if the log bucket is on an EC pool
|
||||
// Use an attribute on the logging name object instead of the bucket so
|
||||
// it is not synced in a multi-site setup
|
||||
|
||||
librados::Rados* rados = rados_store->getRados()->get_rados_handle();
|
||||
librados::IoCtx io_ctx;
|
||||
if (ret = rgw_init_ioctx(dpp, rados, data_pool, io_ctx); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: get data pool ioctx for bucket '" << bucket->get_key() <<
|
||||
"' when writing logging object. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
if (!io_ctx.is_valid()) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: invalid data pool ioctx for bucket '" << bucket->get_key() <<
|
||||
"' when writing logging object" << dendl;
|
||||
return ret;
|
||||
}
|
||||
|
||||
is_ec_pool = false;
|
||||
bufferlist bl;
|
||||
librados::ObjectReadOperation op;
|
||||
int rval;
|
||||
op.getxattr(RGW_ATTR_LOGGING_EC_POOL, &bl, &rval);
|
||||
ret = rgw_rados_operate(dpp, io_ctx, obj_name_oid, std::move(op), nullptr, y);
|
||||
if (ret == 0) {
|
||||
try {
|
||||
ceph::decode(is_ec_pool, bl);
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to decode " << RGW_ATTR_LOGGING_EC_POOL
|
||||
<< " attr for bucket '" << bucket->get_key()
|
||||
<< "'. error = " << err.what() << dendl;
|
||||
return -EINVAL;
|
||||
}
|
||||
} else if (ret < 0){
|
||||
// Try to find out if it is an EC pool
|
||||
ret = rados->mon_command(
|
||||
"{\"prefix\": \"osd pool get\", \"pool\": \"" +
|
||||
data_pool.name + "\", \"var\": \"erasure_code_profile\"}",
|
||||
{}, NULL, NULL);
|
||||
if (ret == -EACCES) {
|
||||
is_ec_pool = false;
|
||||
} else if (ret == 0){
|
||||
is_ec_pool = true;
|
||||
} else {
|
||||
ldpp_dout(dpp, 10) << __func__ << " ERROR: failed to find out if pool"
|
||||
<< data_pool.name << " is erasure coded. ret = "
|
||||
<< ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
bufferlist bl;
|
||||
ceph::encode(is_ec_pool, bl);
|
||||
librados::ObjectWriteOperation op;
|
||||
// if object does not exist, we should not create the attribute
|
||||
op.assert_exists();
|
||||
op.setxattr(RGW_ATTR_LOGGING_EC_POOL, std::move(bl));
|
||||
if (const int ret = rgw_rados_operate(dpp, io_ctx, obj_name_oid, std::move(op), y); ret < 0){
|
||||
// Don't return an error as we know if it is an ec pool
|
||||
ldpp_dout(dpp, 10) << "ERROR: failed to set is_ec_pool attr on :" << obj_name_oid
|
||||
<< " ret: " << ret << dendl;
|
||||
}
|
||||
}
|
||||
// If the target log bucket is on an EC pool, the temp
|
||||
// logging object will be created in the default.rgw.log pool
|
||||
if (is_ec_pool) {
|
||||
data_pool = rados_store->svc()->zone->get_zone_params().bucket_logging_pool;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int RadosBucket::get_logging_object_name(std::string& obj_name,
|
||||
const std::string& prefix,
|
||||
optional_yield y,
|
||||
@ -1157,7 +1243,7 @@ int RadosBucket::get_logging_object_name(std::string& obj_name,
|
||||
return -EIO;
|
||||
}
|
||||
bufferlist bl;
|
||||
const int ret = rgw_get_system_obj(store->svc()->sysobj,
|
||||
if (const int ret = rgw_get_system_obj(store->svc()->sysobj,
|
||||
data_pool,
|
||||
obj_name_oid,
|
||||
bl,
|
||||
@ -1166,15 +1252,18 @@ int RadosBucket::get_logging_object_name(std::string& obj_name,
|
||||
y,
|
||||
dpp,
|
||||
nullptr,
|
||||
nullptr);
|
||||
if (ret < 0) {
|
||||
if (ret == -ENOENT) {
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object name '" << obj_name_oid << "' not found. ret = " << ret << dendl;
|
||||
return ret;
|
||||
nullptr); ret < 0) {
|
||||
if (ret != -ENOENT) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get logging object name from '" << obj_name_oid << "'. ret = " << ret << dendl;
|
||||
} else {
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object name does not exist at '" << obj_name_oid << "'" << dendl;
|
||||
}
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get logging object name from '" << obj_name_oid << "'. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
if (bl.length() == 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging object name at '" << obj_name_oid << "' is empty" << dendl;
|
||||
return -ENODATA;
|
||||
}
|
||||
obj_name = bl.to_str();
|
||||
return 0;
|
||||
}
|
||||
@ -1237,14 +1326,14 @@ std::string to_temp_object_name(const rgw::sal::Bucket* bucket, const std::strin
|
||||
obj_name);
|
||||
}
|
||||
|
||||
int RadosBucket::remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) {
|
||||
int RadosBucket::remove_logging_object(const std::string& obj_name, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp) {
|
||||
rgw_pool data_pool;
|
||||
const rgw_obj head_obj{get_key(), obj_name};
|
||||
const auto placement_rule = get_placement_rule();
|
||||
|
||||
if (!store->getRados()->get_obj_data_pool(placement_rule, head_obj, &data_pool)) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_key() <<
|
||||
"' when deleting logging object" << dendl;
|
||||
const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix);
|
||||
bool is_ec_pool;
|
||||
auto ret = get_logging_temp_object_pool (dpp, this, store, obj_name_oid, data_pool, is_ec_pool, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get temp data pool for bucket '" << get_key() <<
|
||||
"' when deleting logging object: " << obj_name << ". ret: " << ret << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
|
||||
@ -1260,7 +1349,7 @@ int RadosBucket::commit_logging_object(const std::string& obj_name,
|
||||
optional_yield y,
|
||||
const DoutPrefixProvider *dpp,
|
||||
const std::string& prefix,
|
||||
std::string* last_committed) {
|
||||
std::string* last_committed, bool async) {
|
||||
rgw_pool data_pool;
|
||||
const rgw_obj head_obj{get_key(), obj_name};
|
||||
const auto placement_rule = get_placement_rule();
|
||||
@ -1270,9 +1359,10 @@ int RadosBucket::commit_logging_object(const std::string& obj_name,
|
||||
"' when committing logging object" << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
int ret;
|
||||
const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix);
|
||||
if (last_committed) {
|
||||
if (const int ret = get_committed_logging_object(store,
|
||||
if (ret = get_committed_logging_object(store,
|
||||
obj_name_oid,
|
||||
data_pool,
|
||||
y,
|
||||
@ -1290,11 +1380,34 @@ int RadosBucket::commit_logging_object(const std::string& obj_name,
|
||||
}
|
||||
|
||||
const auto temp_obj_name = to_temp_object_name(this, obj_name);
|
||||
rgw_pool temp_data_pool;
|
||||
bool is_ec_pool = false;
|
||||
|
||||
ret = get_logging_temp_object_pool (dpp, this, store, obj_name_oid, temp_data_pool, is_ec_pool, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '"
|
||||
<< get_key() << "' when committing logging object: "
|
||||
<< obj_name << ". ret: " << ret << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
if (async) {
|
||||
ret = rgw::bucketlogging::add_commit_target_entry(dpp, store, this, prefix,
|
||||
obj_name, temp_obj_name,
|
||||
temp_data_pool, y);
|
||||
if (ret < 0){
|
||||
return ret;
|
||||
}
|
||||
|
||||
ldpp_dout(dpp, 20) << "INFO: added logging object entry " << obj_name
|
||||
<< " to commit list object." << dendl;
|
||||
return 0;
|
||||
}
|
||||
|
||||
std::map<string, bufferlist> obj_attrs;
|
||||
ceph::real_time mtime;
|
||||
bufferlist bl_data;
|
||||
if (const auto ret = rgw_get_system_obj(store->svc()->sysobj,
|
||||
data_pool,
|
||||
if (ret = rgw_get_system_obj(store->svc()->sysobj,
|
||||
temp_data_pool,
|
||||
temp_obj_name,
|
||||
bl_data,
|
||||
nullptr,
|
||||
@ -1303,13 +1416,43 @@ int RadosBucket::commit_logging_object(const std::string& obj_name,
|
||||
dpp,
|
||||
&obj_attrs,
|
||||
nullptr); ret < 0) {
|
||||
if (ret == -ENOENT) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: temporary logging object '" << temp_obj_name << "' does not exists" << dendl;
|
||||
} else {
|
||||
if (ret != -ENOENT) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to read logging data when committing object '" << temp_obj_name
|
||||
<< ". error: " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
mtime = ceph::real_time::clock::now();
|
||||
ldpp_dout(dpp, 20) << "INFO: temporary logging object '" << temp_obj_name << "' does not exist. committing it empty" << dendl;
|
||||
} else if (is_ec_pool) {
|
||||
if (ret = rgw_put_system_obj(dpp, store->svc()->sysobj,
|
||||
data_pool,
|
||||
temp_obj_name,
|
||||
bl_data,
|
||||
true,
|
||||
nullptr,
|
||||
mtime,
|
||||
y,
|
||||
&obj_attrs); ret < 0){
|
||||
|
||||
if (ret == -EEXIST) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: race detected in committing logging object '" << temp_obj_name << dendl;
|
||||
} else {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to write logging data when committing object '" << temp_obj_name
|
||||
<< ". error: " << ret << dendl;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: wrote logging data when committing object '" << temp_obj_name << dendl;
|
||||
|
||||
if (ret = rgw_delete_system_obj(dpp, store->svc()->sysobj,
|
||||
temp_data_pool,
|
||||
temp_obj_name,
|
||||
nullptr,
|
||||
y); ret < 0 && ret != -ENOENT) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to delete temp logging object when "
|
||||
<< "committing object '" << temp_obj_name
|
||||
<< ". error: " << ret << dendl;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
uint64_t size = bl_data.length();
|
||||
@ -1418,22 +1561,29 @@ void bucket_logging_completion(rados_completion_t completion, void* args) {
|
||||
|
||||
int RadosBucket::write_logging_object(const std::string& obj_name,
|
||||
const std::string& record,
|
||||
const std::string& prefix,
|
||||
optional_yield y,
|
||||
const DoutPrefixProvider *dpp,
|
||||
bool async_completion) {
|
||||
const auto temp_obj_name = to_temp_object_name(this, obj_name);
|
||||
rgw_pool data_pool;
|
||||
rgw_obj obj{get_key(), obj_name};
|
||||
if (!store->getRados()->get_obj_data_pool(get_placement_rule(), obj, &data_pool)) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_key() <<
|
||||
"' when writing logging object" << dendl;
|
||||
|
||||
bool is_ec_pool = false;
|
||||
const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix);
|
||||
int ret = get_logging_temp_object_pool (dpp, this, store, obj_name_oid, data_pool, is_ec_pool, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '"
|
||||
<< get_key() << "' when writing logging object "
|
||||
<< temp_obj_name << ", ret: " << ret << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
|
||||
librados::IoCtx io_ctx;
|
||||
if (const auto ret = rgw_init_ioctx(dpp, store->getRados()->get_rados_handle(), data_pool, io_ctx); ret < 0) {
|
||||
if (const auto ret = rgw_init_ioctx(dpp, store->getRados()->get_rados_handle(), data_pool, io_ctx, true); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get IO context for logging object from data pool:" << data_pool.to_str() << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
|
||||
bufferlist bl;
|
||||
bl.append(record);
|
||||
bl.append("\n");
|
||||
|
||||
@ -803,9 +803,11 @@ class RadosBucket : public StoreBucket {
|
||||
optional_yield y,
|
||||
const DoutPrefixProvider *dpp,
|
||||
RGWObjVersionTracker* objv_tracker) override;
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed) override;
|
||||
int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override;
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override;
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y,
|
||||
const DoutPrefixProvider *dpp, const std::string& prefix,
|
||||
std::string* last_committed, bool async) override;
|
||||
int remove_logging_object(const std::string& obj_name, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp) override;
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override;
|
||||
|
||||
private:
|
||||
int link(const DoutPrefixProvider* dpp, const rgw_owner& new_owner, optional_yield y, bool update_entrypoint = true, RGWObjVersionTracker* objv = nullptr);
|
||||
|
||||
@ -118,6 +118,7 @@ struct RGWZoneParams : RGWSystemMetaObj {
|
||||
rgw_pool account_pool;
|
||||
rgw_pool group_pool;
|
||||
rgw_pool dedup_pool;
|
||||
rgw_pool bucket_logging_pool;
|
||||
|
||||
RGWAccessKey system_key;
|
||||
|
||||
@ -156,7 +157,7 @@ struct RGWZoneParams : RGWSystemMetaObj {
|
||||
const std::string& get_compression_type(const rgw_placement_rule& placement_rule) const;
|
||||
|
||||
void encode(bufferlist& bl) const override {
|
||||
ENCODE_START(17, 1, bl);
|
||||
ENCODE_START(18, 1, bl);
|
||||
encode(domain_root, bl);
|
||||
encode(control_pool, bl);
|
||||
encode(gc_pool, bl);
|
||||
@ -187,11 +188,12 @@ struct RGWZoneParams : RGWSystemMetaObj {
|
||||
encode(group_pool, bl);
|
||||
encode(restore_pool, bl);
|
||||
encode(dedup_pool, bl);
|
||||
encode(bucket_logging_pool, bl);
|
||||
ENCODE_FINISH(bl);
|
||||
}
|
||||
|
||||
void decode(bufferlist::const_iterator& bl) override {
|
||||
DECODE_START(17, bl);
|
||||
DECODE_START(18, bl);
|
||||
decode(domain_root, bl);
|
||||
decode(control_pool, bl);
|
||||
decode(gc_pool, bl);
|
||||
@ -279,6 +281,11 @@ struct RGWZoneParams : RGWSystemMetaObj {
|
||||
} else {
|
||||
dedup_pool = name + ".rgw.dedup";
|
||||
}
|
||||
if (struct_v >= 18) {
|
||||
decode(bucket_logging_pool, bl);
|
||||
} else {
|
||||
bucket_logging_pool = log_pool.name + ":logging";
|
||||
}
|
||||
DECODE_FINISH(bl);
|
||||
}
|
||||
void dump(Formatter *f) const;
|
||||
|
||||
@ -90,6 +90,7 @@ extern "C" {
|
||||
|
||||
#include "driver/rados/rgw_bucket.h"
|
||||
#include "driver/rados/rgw_sal_rados.h"
|
||||
#include "driver/rados/rgw_bl_rados.h"
|
||||
|
||||
#include <iomanip>
|
||||
|
||||
@ -190,6 +191,7 @@ void usage()
|
||||
cout << " bucket radoslist list rados objects backing bucket's objects\n";
|
||||
cout << " bucket logging flush flush pending log records object of source bucket to the log bucket\n";
|
||||
cout << " bucket logging info get info on bucket logging configuration on source bucket or list of sources in log bucket\n";
|
||||
cout << " bucket logging list list the log objects pending commit for the source bucket\n";
|
||||
cout << " bi get retrieve bucket index object entries\n";
|
||||
cout << " bi put store bucket index object entries\n";
|
||||
cout << " bi list list raw bucket index entries\n";
|
||||
@ -725,6 +727,7 @@ enum class OPT {
|
||||
BUCKET_RESYNC_ENCRYPTED_MULTIPART,
|
||||
BUCKET_LOGGING_FLUSH,
|
||||
BUCKET_LOGGING_INFO,
|
||||
BUCKET_LOGGING_LIST,
|
||||
POLICY,
|
||||
LOG_LIST,
|
||||
LOG_SHOW,
|
||||
@ -974,6 +977,7 @@ static SimpleCmd::Commands all_cmds = {
|
||||
{ "bucket resync encrypted multipart", OPT::BUCKET_RESYNC_ENCRYPTED_MULTIPART },
|
||||
{ "bucket logging flush", OPT::BUCKET_LOGGING_FLUSH },
|
||||
{ "bucket logging info", OPT::BUCKET_LOGGING_INFO },
|
||||
{ "bucket logging list", OPT::BUCKET_LOGGING_LIST },
|
||||
{ "policy", OPT::POLICY },
|
||||
{ "log list", OPT::LOG_LIST },
|
||||
{ "log show", OPT::LOG_SHOW },
|
||||
@ -4620,6 +4624,7 @@ int main(int argc, const char **argv)
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false, // No background tasks!
|
||||
null_yield,
|
||||
need_cache && g_conf()->rgw_cache_enabled,
|
||||
@ -7816,25 +7821,26 @@ int main(int argc, const char **argv)
|
||||
return 0;
|
||||
}
|
||||
|
||||
// make sure that the logging source attribute is up-to-date
|
||||
if (ret = rgw::bucketlogging::update_bucket_logging_sources(dpp(), target_bucket, bucket->get_key(), true, null_yield); ret < 0) {
|
||||
cerr << "WARNING: failed to update logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES
|
||||
<< "' in logging target '" << target_bucket->get_key() << "'. error: " << cpp_strerror(ret) << std::endl;
|
||||
}
|
||||
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv_tracker;
|
||||
ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, dpp(), &objv_tracker);
|
||||
if (ret < 0) {
|
||||
cerr << "ERROR: failed to get pending logging object name from target bucket '" << configuration.target_bucket << "'" << std::endl;
|
||||
if (ret < 0 && ret != -ENOENT) {
|
||||
cerr << "ERROR: failed to get pending logging object name from target bucket '" << configuration.target_bucket <<
|
||||
"'. error: " << cpp_strerror(-ret) << std::endl;
|
||||
return -ret;
|
||||
}
|
||||
std::string old_obj;
|
||||
const auto region = driver->get_zone()->get_zonegroup().get_api_name();
|
||||
ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, dpp(), region, bucket, null_yield, true, &objv_tracker, &old_obj);
|
||||
ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, dpp(), region, bucket, null_yield, true, &objv_tracker, false, &old_obj);
|
||||
if (ret < 0) {
|
||||
if (ret == -ENOENT) {
|
||||
cerr << "WARNING: no pending logging object '" << obj_name << "'. nothing to flush";
|
||||
ret = 0;
|
||||
} else {
|
||||
cerr << "ERROR: failed flush pending logging object '" << obj_name << "'";
|
||||
}
|
||||
cerr << " to target bucket '" << configuration.target_bucket << "'. "
|
||||
<< " last committed object is '" << old_obj << "'" << std::endl;
|
||||
cerr << "ERROR: failed to flush pending logging object '" << obj_name << "' to target bucket '" << configuration.target_bucket
|
||||
<< "'. error: " << cpp_strerror(-ret) << std::endl;
|
||||
return -ret;
|
||||
}
|
||||
cout << "flushed pending logging object '" << old_obj
|
||||
@ -7883,6 +7889,55 @@ int main(int argc, const char **argv)
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (opt_cmd == OPT::BUCKET_LOGGING_LIST) {
|
||||
if (bucket_name.empty()) {
|
||||
cerr << "ERROR: bucket not specified" << std::endl;
|
||||
return EINVAL;
|
||||
}
|
||||
if (driver->get_name() != "rados") {
|
||||
cerr << "ERROR: this command is only available with the RADOS driver." << std::endl;
|
||||
return EINVAL;
|
||||
}
|
||||
|
||||
int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
|
||||
if (ret < 0) {
|
||||
return -ret;
|
||||
}
|
||||
|
||||
rgw::bucketlogging::configuration configuration;
|
||||
std::unique_ptr<rgw::sal::Bucket> target_bucket;
|
||||
ret = rgw::bucketlogging::get_target_and_conf_from_source(dpp(),
|
||||
driver, bucket.get(), tenant, configuration, target_bucket, null_yield);
|
||||
if (ret < 0 && ret != -ENODATA) {
|
||||
cerr << "ERROR: failed to get target bucket and logging conf from source bucket '"
|
||||
<< bucket_name << "': " << cpp_strerror(-ret) << std::endl;
|
||||
return -ret;
|
||||
} else if (ret == -ENODATA) {
|
||||
cerr << "ERROR: bucket '" << bucket_name << "' does not have logging enabled" << std::endl;
|
||||
return 0;
|
||||
}
|
||||
std::string target_prefix = configuration.target_prefix;
|
||||
std::set<std::string> entries;
|
||||
|
||||
ret = rgw::bucketlogging::list_pending_commit_objects(dpp(),
|
||||
static_cast<rgw::sal::RadosStore*>(driver), target_bucket.get(),
|
||||
target_prefix, entries, null_yield);
|
||||
|
||||
if (ret < 0) {
|
||||
cerr << "ERROR: failed to get pending log entries for bucket '" << bucket_name
|
||||
<< "': " << cpp_strerror(-ret) << std::endl;
|
||||
return ret;
|
||||
}
|
||||
|
||||
formatter->open_array_section("pending_logs");
|
||||
for (auto &entry: entries) {
|
||||
formatter->dump_string("log", entry);
|
||||
}
|
||||
formatter->close_section(); // objs
|
||||
formatter->flush(cout);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (opt_cmd == OPT::LOG_LIST) {
|
||||
// filter by date?
|
||||
if (date.size() && date.size() != 10) {
|
||||
|
||||
@ -258,7 +258,9 @@ int rgw::AppMain::init_storage()
|
||||
run_quota,
|
||||
run_sync,
|
||||
g_conf().get_val<bool>("rgw_dynamic_resharding"),
|
||||
true, true, null_yield, // run notification thread
|
||||
true, // run notification thread
|
||||
true, // run bucket-logging thread
|
||||
true, null_yield,
|
||||
g_conf()->rgw_cache_enabled);
|
||||
if (!env.driver) {
|
||||
return -EIO;
|
||||
|
||||
@ -288,19 +288,19 @@ int new_logging_object(const configuration& conf,
|
||||
int ret = target_bucket->set_logging_object_name(obj_name, conf.target_prefix, y, dpp, (old_name == std::nullopt), objv_tracker);
|
||||
if (ret == -EEXIST || ret == -ECANCELED) {
|
||||
if (ret = target_bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, nullptr); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of logging bucket '" <<
|
||||
target_bucket_id << "' and prefix '" << conf.target_prefix << "', ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: name already set. got name of logging object '" << obj_name << "' of bucket '" <<
|
||||
ldpp_dout(dpp, 20) << "INFO: name already set. got name of logging object '" << obj_name << "' of logging bucket '" <<
|
||||
target_bucket_id << "' and prefix '" << conf.target_prefix << "'" << dendl;
|
||||
return -ECANCELED;
|
||||
} else if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to write name of logging object '" << obj_name << "' of bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to write name of logging object '" << obj_name << "' of logging bucket '" <<
|
||||
target_bucket_id << "'. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: wrote name of logging object '" << obj_name << "' of bucket '" <<
|
||||
ldpp_dout(dpp, 20) << "INFO: wrote name of logging object '" << obj_name << "' of logging bucket '" <<
|
||||
target_bucket_id << "'" << dendl;
|
||||
return 0;
|
||||
}
|
||||
@ -315,7 +315,7 @@ int commit_logging_object(const configuration& conf,
|
||||
std::string target_tenant_name;
|
||||
int ret = rgw_parse_url_bucket(conf.target_bucket, tenant_name, target_tenant_name, target_bucket_name);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse target bucket '" << conf.target_bucket << "' when commiting logging object, ret = "
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse logging bucket '" << conf.target_bucket << "' when committing logging object, ret = "
|
||||
<< ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
@ -324,26 +324,18 @@ int commit_logging_object(const configuration& conf,
|
||||
ret = driver->load_bucket(dpp, target_bucket_id,
|
||||
&target_bucket, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get target logging bucket '" << target_bucket_id << "' when commiting logging object, ret = "
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get logging bucket '" << target_bucket_id << "' when committing logging object, ret = "
|
||||
<< ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
return commit_logging_object(conf, target_bucket, dpp, y, last_committed);
|
||||
}
|
||||
|
||||
int commit_logging_object(const configuration& conf,
|
||||
const std::unique_ptr<rgw::sal::Bucket>& target_bucket,
|
||||
const DoutPrefixProvider *dpp,
|
||||
optional_yield y,
|
||||
std::string* last_committed) {
|
||||
std::string obj_name;
|
||||
if (const int ret = target_bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, nullptr); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
if (const int ret = target_bucket->commit_logging_object(obj_name, y, dpp, conf.target_prefix, last_committed); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to commit logging object '" << obj_name << "' of bucket '" <<
|
||||
if (const int ret = target_bucket->commit_logging_object(obj_name, y, dpp, conf.target_prefix, last_committed, false); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to commit logging object '" << obj_name << "' of logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
@ -359,34 +351,60 @@ int rollover_logging_object(const configuration& conf,
|
||||
optional_yield y,
|
||||
bool must_commit,
|
||||
RGWObjVersionTracker* objv_tracker,
|
||||
std::string* last_committed) {
|
||||
bool async,
|
||||
std::string* last_committed,
|
||||
std::string* err_message) {
|
||||
std::string target_bucket_name;
|
||||
std::string target_tenant_name;
|
||||
std::ignore = rgw_parse_url_bucket(conf.target_bucket, target_bucket->get_tenant(), target_tenant_name, target_bucket_name);
|
||||
if (target_bucket_name != target_bucket->get_name() || target_tenant_name != target_bucket->get_tenant()) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging bucket name mismatch. conf= '" << conf.target_bucket <<
|
||||
"', bucket= '" << target_bucket->get_key() << "'" << dendl;
|
||||
return -EINVAL;
|
||||
return -EINVAL; // this should never happen
|
||||
}
|
||||
const auto old_obj = obj_name;
|
||||
const int ret = new_logging_object(conf, target_bucket, obj_name, dpp, region, source_bucket, y, old_obj, objv_tracker);
|
||||
if (ret == -ECANCELED) {
|
||||
ldpp_dout(dpp, 20) << "INFO: rollover already performed for object '" << old_obj << "' to logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
return 0;
|
||||
} else if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to rollover object '" << old_obj << "' to logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
|
||||
auto old_obj = obj_name.empty() ? std::nullopt : std::optional<std::string>(obj_name);
|
||||
|
||||
auto handle_error = [&dpp, &old_obj, &target_bucket, err_message](int ret) {
|
||||
if (ret < 0) {
|
||||
if (ret == -ECANCELED) {
|
||||
ldpp_dout(dpp, 20) << "INFO: rollover already performed for logging object '" << old_obj << "' to logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Rollover already performed on logging bucket '{}'", target_bucket->get_name());
|
||||
}
|
||||
} else {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to rollover logging object '" << old_obj << "' to logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Failed to rollover logging object of logging bucket '{}'", target_bucket->get_name());
|
||||
}
|
||||
}
|
||||
}
|
||||
return ret;
|
||||
};
|
||||
|
||||
if (const int ret = handle_error(new_logging_object(conf, target_bucket, obj_name, dpp, region, source_bucket, y, old_obj, objv_tracker)); ret < 0) {
|
||||
return ret;
|
||||
}
|
||||
if (const int ret = target_bucket->commit_logging_object(old_obj, y, dpp, conf.target_prefix, last_committed); ret < 0) {
|
||||
if (!old_obj) {
|
||||
// first time logging old == new
|
||||
old_obj = obj_name;
|
||||
if (const int ret = handle_error(new_logging_object(conf, target_bucket, obj_name, dpp, region, source_bucket, y, old_obj, objv_tracker)); ret < 0) {
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
if (const int ret = target_bucket->commit_logging_object(*old_obj, y, dpp, conf.target_prefix, last_committed, async); ret < 0) {
|
||||
if (must_commit) {
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Failed to commit logging object of logging bucket '{}'", target_bucket->get_name());
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to commit object '" << old_obj << "' to logging bucket '" <<
|
||||
target_bucket->get_key() << "'. ret = " << ret << dendl;
|
||||
// we still want to write the new records to the new object even if commit failed
|
||||
// will try to commit again next time
|
||||
// TODO: should add commit retry mechanism
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@ -449,13 +467,17 @@ int log_record(rgw::sal::Driver* driver,
|
||||
bool log_source_bucket) {
|
||||
if (!s->bucket) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: only bucket operations are logged in bucket logging" << dendl;
|
||||
return -EINVAL;
|
||||
return -EINVAL; // this should never happen
|
||||
}
|
||||
auto set_journal_err = [&conf, s](const std::string& err_message) {
|
||||
if (conf.logging_type == LoggingType::Journal) s->err.message = err_message;
|
||||
};
|
||||
std::string target_bucket_name;
|
||||
std::string target_tenant_name;
|
||||
int ret = rgw_parse_url_bucket(conf.target_bucket, s->bucket_tenant, target_tenant_name, target_bucket_name);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse target logging bucket '" << conf.target_bucket << "', ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse logging bucket name '" << conf.target_bucket << "', ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Faild to parse logging bucket name '{}'", conf.target_bucket));
|
||||
return ret;
|
||||
}
|
||||
const rgw_bucket target_bucket_id(target_tenant_name, target_bucket_name);
|
||||
@ -463,21 +485,29 @@ int log_record(rgw::sal::Driver* driver,
|
||||
ret = driver->load_bucket(dpp, target_bucket_id,
|
||||
&target_bucket, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get target logging bucket '" << target_bucket_id << "'. ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to load logging bucket '" << target_bucket_id << "'. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Faild to load logging bucket '{}'", target_bucket_id.bucket_id));
|
||||
return ret;
|
||||
}
|
||||
|
||||
rgw::ARN target_resource_arn(target_bucket_id, conf.target_prefix);
|
||||
if (ret = verify_target_bucket_policy(dpp, target_bucket.get(), target_resource_arn, s); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to verify target logging bucket policy for bucket '" << target_bucket->get_key() <<
|
||||
"'. ret = " << ret << dendl;
|
||||
return -EACCES;
|
||||
std::string err_message;
|
||||
if (ret = verify_target_bucket_policy(dpp, target_bucket.get(), target_resource_arn, s, &err_message); ret < 0) {
|
||||
set_journal_err(err_message);
|
||||
return ret;
|
||||
}
|
||||
|
||||
if (ret = verify_target_bucket_attributes(dpp, target_bucket.get()); ret < 0) {
|
||||
if (ret = verify_target_bucket_attributes(dpp, target_bucket.get(), &err_message); ret < 0) {
|
||||
set_journal_err(err_message);
|
||||
return ret;
|
||||
}
|
||||
|
||||
// make sure that the logging source attribute is up-to-date
|
||||
if (ret = update_bucket_logging_sources(dpp, target_bucket, s->bucket->get_key(), true, y); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: could not update bucket logging source '" <<
|
||||
s->bucket->get_key() << "' in logging bucket '" << target_bucket_id << "' attribute, during record logging. ret = " << ret << dendl;
|
||||
}
|
||||
|
||||
const auto region = driver->get_zone()->get_zonegroup().get_api_name();
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv_tracker;
|
||||
@ -485,9 +515,10 @@ int log_record(rgw::sal::Driver* driver,
|
||||
if (ret == 0) {
|
||||
const auto time_to_commit = time_from_name(obj_name, dpp) + std::chrono::seconds(conf.obj_roll_time);
|
||||
if (ceph::coarse_real_time::clock::now() > time_to_commit) {
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' exceeded its time, will be committed to bucket '" <<
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' exceeded its time, will be committed to logging bucket '" <<
|
||||
target_bucket_id << "'" << dendl;
|
||||
if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, region, s->bucket, y, false, &objv_tracker, nullptr); ret < 0) {
|
||||
if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, region, s->bucket, y, false, &objv_tracker, true, nullptr, &err_message); ret < 0 && ret != -ECANCELED) {
|
||||
set_journal_err(err_message);
|
||||
return ret;
|
||||
}
|
||||
} else {
|
||||
@ -497,19 +528,21 @@ int log_record(rgw::sal::Driver* driver,
|
||||
// try to create the temporary log object for the first time
|
||||
ret = new_logging_object(conf, target_bucket, obj_name, dpp, region, s->bucket, y, std::nullopt, &objv_tracker);
|
||||
if (ret == 0) {
|
||||
ldpp_dout(dpp, 20) << "INFO: first time logging for bucket '" << target_bucket_id << "' and prefix '" <<
|
||||
ldpp_dout(dpp, 20) << "INFO: first time logging for logging bucket '" << target_bucket_id << "' and prefix '" <<
|
||||
conf.target_prefix << "'" << dendl;
|
||||
} else if (ret == -ECANCELED) {
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' already exists for bucket '" << target_bucket_id << "' and prefix" <<
|
||||
ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' already exists for logging bucket '" << target_bucket_id << "' and prefix" <<
|
||||
conf.target_prefix << "'" << dendl;
|
||||
} else {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to create logging object of bucket '" <<
|
||||
target_bucket_id << "' and prefix '" << conf.target_prefix << "' for the first time. ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to create first time logging object of logging bucket '" <<
|
||||
target_bucket_id << "' and prefix '" << conf.target_prefix << "'. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Faild create first time logging object of logging bucket '{}'", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
} else {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of logging bucket '" <<
|
||||
target_bucket_id << "'. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Faild to get name of logging object of logging bucket '{}'", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
|
||||
@ -544,7 +577,6 @@ int log_record(rgw::sal::Driver* driver,
|
||||
bucket_name = full_bucket_name(s->bucket);
|
||||
}
|
||||
|
||||
|
||||
switch (conf.logging_type) {
|
||||
case LoggingType::Standard:
|
||||
record = fmt::format("{} {} [{:%d/%b/%Y:%H:%M:%S %z}] {} {} {} {} {} \"{} {}{}{} HTTP/1.1\" {} {} {} {} {} {} {} \"{}\" {} {} {} {} {} {} {} {} {}",
|
||||
@ -592,14 +624,15 @@ int log_record(rgw::sal::Driver* driver,
|
||||
case LoggingType::Any:
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to format record when writing to logging object '" <<
|
||||
obj_name << "' due to unsupported logging type" << dendl;
|
||||
return -EINVAL;
|
||||
return -EINVAL; // this should never happen
|
||||
}
|
||||
|
||||
// get quota of the owner of the target bucket
|
||||
RGWQuota user_quota;
|
||||
if (ret = get_owner_quota_info(dpp, y, driver, target_bucket->get_owner(), user_quota); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get quota of owner of target logging bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get quota of owner of logging bucket '" <<
|
||||
target_bucket_id << "' failed. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Faild to get quota of owner of logging bucket '{}'", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
// start with system default quota
|
||||
@ -616,33 +649,39 @@ int log_record(rgw::sal::Driver* driver,
|
||||
}
|
||||
// verify there is enough quota to write the record
|
||||
if (ret = target_bucket->check_quota(dpp, quota, record.length(), y); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: quota check on target logging bucket '" <<
|
||||
ldpp_dout(dpp, 1) << "ERROR: quota check on logging bucket '" <<
|
||||
target_bucket_id << "' failed. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Quota check on logging bucket '{}' failed", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
|
||||
if (ret = target_bucket->write_logging_object(obj_name,
|
||||
record,
|
||||
conf.target_prefix,
|
||||
y,
|
||||
dpp,
|
||||
async_completion); ret < 0 && ret != -EFBIG) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" <<
|
||||
obj_name << "'. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Failed to write record to logging object of logging bucket {}", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
if (ret == -EFBIG) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: logging object '" << obj_name << "' is full, will be committed to bucket '" <<
|
||||
ldpp_dout(dpp, 5) << "WARNING: logging object '" << obj_name << "' is full, will be committed to logging bucket '" <<
|
||||
target_bucket->get_key() << "'" << dendl;
|
||||
if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, region, s->bucket, y, true, &objv_tracker, nullptr); ret < 0 ) {
|
||||
if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, region, s->bucket, y, true, &objv_tracker, true, nullptr, &err_message); ret < 0 && ret != -ECANCELED) {
|
||||
set_journal_err(err_message);
|
||||
return ret;
|
||||
}
|
||||
if (ret = target_bucket->write_logging_object(obj_name,
|
||||
record,
|
||||
conf.target_prefix,
|
||||
y,
|
||||
dpp,
|
||||
async_completion); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" <<
|
||||
obj_name << "'. ret = " << ret << dendl;
|
||||
set_journal_err(fmt::format("Failed to write record to logging object of logging bucket {}", target_bucket->get_name()));
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
@ -720,7 +759,7 @@ int update_bucket_logging_sources(const DoutPrefixProvider* dpp, rgw::sal::Drive
|
||||
std::unique_ptr<rgw::sal::Bucket> target_bucket;
|
||||
const int ret = driver->load_bucket(dpp, target_bucket_id, &target_bucket, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get target logging bucket '" << target_bucket_id <<
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging bucket '" << target_bucket_id <<
|
||||
"' in order to update logging sources. ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
@ -757,8 +796,7 @@ int update_bucket_logging_sources(const DoutPrefixProvider* dpp, std::unique_ptr
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to decode logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES
|
||||
<< "' for bucket '" << bucket->get_key() << "' when updating logging sources. error: " << err.what() << dendl;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: logging source '" << src_bucket_id << "' already " <<
|
||||
(add ? "added to" : "removed from") << " bucket '" << bucket->get_key() << "'" << dendl;
|
||||
// no change
|
||||
return 0;
|
||||
}, y);
|
||||
}
|
||||
@ -772,64 +810,71 @@ int bucket_deletion_cleanup(const DoutPrefixProvider* dpp,
|
||||
// and also delete the object holding the pending object name
|
||||
auto& attrs = bucket->get_attrs();
|
||||
if (const auto iter = attrs.find(RGW_ATTR_BUCKET_LOGGING_SOURCES); iter != attrs.end()) {
|
||||
source_buckets sources;
|
||||
try {
|
||||
source_buckets sources;
|
||||
ceph::decode(sources, iter->second);
|
||||
for (const auto& source : sources) {
|
||||
std::unique_ptr<rgw::sal::Bucket> src_bucket;
|
||||
if (const int ret = driver->load_bucket(dpp, source, &src_bucket, y); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging source bucket '" << source << "' for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
auto& src_attrs = src_bucket->get_attrs();
|
||||
if (const auto iter = src_attrs.find(RGW_ATTR_BUCKET_LOGGING); iter != src_attrs.end()) {
|
||||
configuration conf;
|
||||
try {
|
||||
auto bl_iter = iter->second.cbegin();
|
||||
decode(conf, bl_iter);
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv;
|
||||
if (const int ret = bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging object name for logging bucket '" << bucket->get_key() <<
|
||||
"' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
if (const int ret = bucket->remove_logging_object(obj_name, y, dpp); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to delete pending logging object '" << obj_name << "' for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully deleted pending logging object '" << obj_name << "' from deleted logging bucket '" <<
|
||||
bucket->get_key() << "'" << dendl;
|
||||
if (const int ret = bucket->remove_logging_object_name(conf.target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to delete object holding bucket logging object name for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully deleted object holding bucket logging object name from deleted logging bucket '" <<
|
||||
bucket->get_key() << "'" << dendl;
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING
|
||||
<< "' of bucket '" << src_bucket->get_key() << "' during cleanup. error: " << err.what() << dendl;
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to decode logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES
|
||||
<< "' for logging bucket '" << bucket->get_key() << "'. error: " << err.what() << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
// since we removed the attribute, we should not return error code from now on
|
||||
// any retry, would find no attribute and return success
|
||||
for (const auto& source : sources) {
|
||||
std::unique_ptr<rgw::sal::Bucket> src_bucket;
|
||||
if (const int ret = driver->load_bucket(dpp, source, &src_bucket, y); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging source bucket '" << source << "' for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
auto& src_attrs = src_bucket->get_attrs();
|
||||
if (const auto iter = src_attrs.find(RGW_ATTR_BUCKET_LOGGING); iter != src_attrs.end()) {
|
||||
configuration conf;
|
||||
try {
|
||||
auto bl_iter = iter->second.cbegin();
|
||||
decode(conf, bl_iter);
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING
|
||||
<< "' of bucket '" << src_bucket->get_key() << "' during cleanup. error: " << err.what() << dendl;
|
||||
continue;
|
||||
}
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv;
|
||||
if (const int ret = bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging object name for logging bucket '" << bucket->get_key() <<
|
||||
"' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
if (const int ret = bucket->remove_logging_object_name(conf.target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to delete object holding bucket logging object name for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully deleted object holding bucket logging object name from deleted logging bucket '" <<
|
||||
bucket->get_key() << "'" << dendl;
|
||||
if (const int ret = bucket->remove_logging_object(obj_name, conf.target_prefix, y, dpp); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to delete pending logging object '" << obj_name << "' for logging bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
continue;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully deleted pending logging object '" << obj_name << "' from deleted logging bucket '" <<
|
||||
bucket->get_key() << "'" << dendl;
|
||||
} else {
|
||||
ldpp_dout(dpp, 5) << "WARNING: no logging attribute '" << RGW_ATTR_BUCKET_LOGGING
|
||||
<< "' found in logging source bucket '" << src_bucket->get_key() << "' during cleanup" << dendl;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return source_bucket_cleanup(dpp, driver, bucket, false, y);
|
||||
return source_bucket_cleanup(dpp, driver, bucket, false, y, nullptr);
|
||||
}
|
||||
|
||||
int source_bucket_cleanup(const DoutPrefixProvider* dpp,
|
||||
sal::Driver* driver,
|
||||
sal::Bucket* bucket,
|
||||
bool remove_attr,
|
||||
optional_yield y) {
|
||||
optional_yield y,
|
||||
std::string* last_committed) {
|
||||
std::optional<configuration> conf;
|
||||
if (const int ret = retry_raced_bucket_write(dpp, bucket, [dpp, bucket, &conf, remove_attr, y] {
|
||||
auto& attrs = bucket->get_attrs();
|
||||
@ -842,7 +887,7 @@ int source_bucket_cleanup(const DoutPrefixProvider* dpp,
|
||||
conf = std::move(tmp_conf);
|
||||
} catch (buffer::error& err) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to decode existing logging attribute '" << RGW_ATTR_BUCKET_LOGGING
|
||||
<< "' of bucket '" << bucket->get_key() << "' during cleanup. error: " << err.what() << dendl;
|
||||
<< "' of bucket '" << bucket->get_key() << "' during source cleanup. error: " << err.what() << dendl;
|
||||
return -EIO;
|
||||
}
|
||||
if (remove_attr) {
|
||||
@ -855,41 +900,75 @@ int source_bucket_cleanup(const DoutPrefixProvider* dpp,
|
||||
}, y); ret < 0) {
|
||||
if (remove_attr) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to remove logging attribute '" << RGW_ATTR_BUCKET_LOGGING <<
|
||||
"' from bucket '" << bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
"' from bucket '" << bucket->get_key() << "' during source cleanup. ret = " << ret << dendl;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
if (last_committed) {
|
||||
last_committed->clear();
|
||||
}
|
||||
if (!conf) {
|
||||
// no logging attribute found
|
||||
return 0;
|
||||
}
|
||||
const auto& info = bucket->get_info();
|
||||
if (const int ret = commit_logging_object(*conf, dpp, driver, info.bucket.tenant, y, nullptr); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: could not commit pending logging object of bucket '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
} else {
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully committed pending logging object of bucket '" << bucket->get_key() << "'" << dendl;
|
||||
}
|
||||
// since we removed the attribute, we should not return error code from now on
|
||||
// any retry, would find no attribute and return success
|
||||
rgw_bucket target_bucket_id;
|
||||
const auto& info = bucket->get_info();
|
||||
if (const int ret = get_bucket_id(conf->target_bucket, info.bucket.tenant, target_bucket_id); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to parse target logging bucket '" <<
|
||||
conf->target_bucket << "' during cleanup. ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to parse logging bucket '" <<
|
||||
conf->target_bucket << "' during source cleanup. ret = " << ret << dendl;
|
||||
return 0;
|
||||
}
|
||||
if (const int ret = update_bucket_logging_sources(dpp, driver, target_bucket_id, bucket->get_key(), false, y); ret < 0) {
|
||||
std::unique_ptr<rgw::sal::Bucket> target_bucket;
|
||||
const int ret = driver->load_bucket(dpp, target_bucket_id, &target_bucket, y);
|
||||
if (ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging bucket '" << target_bucket_id <<
|
||||
"' when doing source cleanup. ret = " << ret << dendl;
|
||||
return 0;
|
||||
}
|
||||
if (const int ret = update_bucket_logging_sources(dpp, target_bucket, bucket->get_key(), false, y); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: could not update bucket logging source '" <<
|
||||
bucket->get_key() << "' during cleanup. ret = " << ret << dendl;
|
||||
bucket->get_key() << "' in logging bucket '" << target_bucket_id << "' attribute, during source cleanup. ret = " << ret << dendl;
|
||||
}
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv;
|
||||
if (const int ret = target_bucket->get_logging_object_name(obj_name, conf->target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to get logging object name for logging bucket '" << bucket->get_key() <<
|
||||
"' during source cleanup. ret = " << ret << dendl;
|
||||
// if name cannot be fetched, we should not commit or remove the object holding the name
|
||||
// it is better to leak an object than to cause a possible data loss
|
||||
return 0;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully updated bucket logging source '" <<
|
||||
bucket->get_key() << "' during cleanup"<< dendl;
|
||||
if (const int ret = target_bucket->remove_logging_object_name(conf->target_prefix, y, dpp, &objv); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: failed to delete object holding bucket logging object name for bucket '" <<
|
||||
bucket->get_key() << "' during source cleanup. ret = " << ret << dendl;
|
||||
// this could be because of a race someone else trying to commit the object
|
||||
// (another bucket cleanup with shared prefix, rollover or an explicit commit)
|
||||
// and since committing the object twice will cause data loss, we should not try to commit it here
|
||||
return 0;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully deleted object holding bucket logging object name for bucket '" <<
|
||||
bucket->get_key() << "' during source cleanup" << dendl;
|
||||
// since the object holding the name was deleted, we cannot fetch the last commited name from it
|
||||
if (const int ret = target_bucket->commit_logging_object(obj_name, y, dpp, conf->target_prefix, nullptr, false); ret < 0) {
|
||||
ldpp_dout(dpp, 5) << "WARNING: could not commit pending logging object of bucket '" <<
|
||||
bucket->get_key() << "' during source cleanup. ret = " << ret << dendl;
|
||||
return 0;
|
||||
}
|
||||
ldpp_dout(dpp, 20) << "INFO: successfully committed pending logging object of bucket '" <<
|
||||
bucket->get_key() << "' during source cleanup. ret = " << ret << dendl;
|
||||
if (last_committed) {
|
||||
*last_committed = obj_name;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int verify_target_bucket_policy(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::Bucket* target_bucket,
|
||||
const rgw::ARN& target_resource_arn,
|
||||
req_state* s) {
|
||||
req_state* s,
|
||||
std::string* err_message) {
|
||||
// verify target permissions for bucket logging
|
||||
// this is implementing the policy based permission granting from:
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html#grant-log-delivery-permissions-general
|
||||
@ -898,6 +977,9 @@ int verify_target_bucket_policy(const DoutPrefixProvider* dpp,
|
||||
const auto policy_it = target_attrs.find(RGW_ATTR_IAM_POLICY);
|
||||
if (policy_it == target_attrs.end()) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging bucket '" << target_bucket_id << "' must have bucket policy to allow logging" << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' must have policy to allow logging", target_bucket->get_name());
|
||||
}
|
||||
return -EACCES;
|
||||
}
|
||||
try {
|
||||
@ -914,33 +996,48 @@ int verify_target_bucket_policy(const DoutPrefixProvider* dpp,
|
||||
"' must have a bucket policy that allows logging service principal to put objects in the following resource ARN: '" <<
|
||||
target_resource_arn.to_string() << "' from source bucket ARN: '" << source_bucket_arn <<
|
||||
"' and source account: '" << source_account << "'" << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' policy does not allow logging", target_bucket->get_name());
|
||||
}
|
||||
return -EACCES;
|
||||
}
|
||||
} catch (const rgw::IAM::PolicyParseException& err) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse logging bucket '" << target_bucket_id <<
|
||||
"' policy. error: " << err.what() << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' has invalid policy", target_bucket->get_name());
|
||||
}
|
||||
return -EACCES;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int verify_target_bucket_attributes(const DoutPrefixProvider* dpp, rgw::sal::Bucket* target_bucket) {
|
||||
int verify_target_bucket_attributes(const DoutPrefixProvider* dpp, rgw::sal::Bucket* target_bucket, std::string* err_message) {
|
||||
const auto& target_info = target_bucket->get_info();
|
||||
if (target_info.requester_pays) {
|
||||
// target bucket must not have requester pays set on it
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging target bucket '" << target_bucket->get_key() << "', is configured with requester pays" << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging bucket '" << target_bucket->get_key() << "', is configured with requester pays" << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' is configured with requester pays", target_bucket->get_name());
|
||||
}
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
const auto& target_attrs = target_bucket->get_attrs();
|
||||
if (target_attrs.find(RGW_ATTR_BUCKET_LOGGING) != target_attrs.end()) {
|
||||
// target bucket must not have logging set on it
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging target bucket '" << target_bucket->get_key() << "', is configured with bucket logging" << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging bucket '" << target_bucket->get_key() << "', is configured with bucket logging" << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' is configured with bucket logging", target_bucket->get_name());
|
||||
}
|
||||
return -EINVAL;
|
||||
}
|
||||
if (target_attrs.find(RGW_ATTR_BUCKET_ENCRYPTION_POLICY) != target_attrs.end()) {
|
||||
// verify target bucket does not have encryption
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging target bucket '" << target_bucket->get_key() << "', is configured with encryption" << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: logging bucket '" << target_bucket->get_key() << "', is configured with encryption" << dendl;
|
||||
if (err_message) {
|
||||
*err_message = fmt::format("Logging bucket '{}' is configured with encryption", target_bucket->get_name());
|
||||
}
|
||||
return -EINVAL;
|
||||
}
|
||||
return 0;
|
||||
@ -973,13 +1070,13 @@ int get_target_and_conf_from_source(
|
||||
|
||||
rgw_bucket target_bucket_id;
|
||||
if (const int ret = get_bucket_id(configuration.target_bucket, tenant, target_bucket_id); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse target bucket '" << configuration.target_bucket << "', ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to parse logging bucket '" << configuration.target_bucket << "', ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
|
||||
if (const int ret = driver->load_bucket(dpp, target_bucket_id,
|
||||
&target_bucket, y); ret < 0) {
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get target bucket '" << target_bucket_id << "', ret = " << ret << dendl;
|
||||
ldpp_dout(dpp, 1) << "ERROR: failed to get logging bucket '" << target_bucket_id << "', ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
return 0;
|
||||
|
||||
@ -173,6 +173,7 @@ int log_record(rgw::sal::Driver* driver,
|
||||
// and create a new pending log object
|
||||
// if "must_commit" is "false" the function will return success even if the pending log object was not committed
|
||||
// if "last_committed" is not null, it will be set to the name of the last committed object
|
||||
// optional error message is returned by reference
|
||||
int rollover_logging_object(const configuration& conf,
|
||||
const std::unique_ptr<rgw::sal::Bucket>& bucket,
|
||||
std::string& obj_name,
|
||||
@ -182,28 +183,9 @@ int rollover_logging_object(const configuration& conf,
|
||||
optional_yield y,
|
||||
bool must_commit,
|
||||
RGWObjVersionTracker* objv_tracker,
|
||||
std::string* last_committed);
|
||||
|
||||
// commit the pending log object to the log bucket
|
||||
// use this for cleanup, when new pending object is not needed
|
||||
// and target bucket is known
|
||||
// if "last_committed" is not null, it will be set to the name of the last committed object
|
||||
int commit_logging_object(const configuration& conf,
|
||||
const std::unique_ptr<rgw::sal::Bucket>& target_bucket,
|
||||
const DoutPrefixProvider *dpp,
|
||||
optional_yield y,
|
||||
std::string* last_committed);
|
||||
|
||||
// commit the pending log object to the log bucket
|
||||
// use this for cleanup, when new pending object is not needed
|
||||
// and target bucket shoud be loaded based on the configuration
|
||||
// if "last_committed" is not null, it will be set to the name of the last committed object
|
||||
int commit_logging_object(const configuration& conf,
|
||||
const DoutPrefixProvider *dpp,
|
||||
rgw::sal::Driver* driver,
|
||||
const std::string& tenant_name,
|
||||
optional_yield y,
|
||||
std::string* last_committed);
|
||||
bool async,
|
||||
std::string* last_committed,
|
||||
std::string* err_message = nullptr);
|
||||
|
||||
// return the oid of the object holding the name of the temporary logging object
|
||||
// bucket - log bucket
|
||||
@ -251,24 +233,31 @@ int bucket_deletion_cleanup(const DoutPrefixProvider* dpp,
|
||||
// in addition:
|
||||
// any pending log objects should be comitted to the log bucket
|
||||
// and the log bucket should be updated to remove the bucket as a source
|
||||
// if "last_committed" is not null, it will be set to the name of the last committed object
|
||||
int source_bucket_cleanup(const DoutPrefixProvider* dpp,
|
||||
sal::Driver* driver,
|
||||
sal::Bucket* bucket,
|
||||
bool remove_attr,
|
||||
optional_yield y);
|
||||
optional_yield y,
|
||||
std::string* last_committed);
|
||||
|
||||
// verify that the target bucket has the correct policy to allow the source bucket to log to it
|
||||
// note that this function adds entries to the request state environment
|
||||
// optional error message is returned by reference
|
||||
int verify_target_bucket_policy(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::Bucket* target_bucket,
|
||||
const rgw::ARN& target_resource_arn,
|
||||
req_state* s);
|
||||
req_state* s,
|
||||
std::string* err_message = nullptr);
|
||||
|
||||
// verify that target bucket does not have:
|
||||
// - bucket logging
|
||||
// - requester pays
|
||||
// - encryption
|
||||
int verify_target_bucket_attributes(const DoutPrefixProvider* dpp, rgw::sal::Bucket* target_bucket);
|
||||
// optional error message is returned by reference
|
||||
int verify_target_bucket_attributes(const DoutPrefixProvider* dpp,
|
||||
rgw::sal::Bucket* target_bucket,
|
||||
std::string* err_message = nullptr);
|
||||
|
||||
// given a source bucket this function is parsing the configuration object
|
||||
// extracting the target bucket and load it
|
||||
|
||||
@ -1255,7 +1255,7 @@ struct req_info {
|
||||
meta_map_t crypt_attribute_map;
|
||||
|
||||
std::string host;
|
||||
const char *method;
|
||||
const char *method = nullptr;
|
||||
std::string script_uri;
|
||||
std::string request_uri;
|
||||
std::string request_uri_aws4;
|
||||
|
||||
@ -105,7 +105,7 @@ int main(const int argc, const char **argv)
|
||||
exit(1);
|
||||
}
|
||||
|
||||
driver = DriverManager::get_storage(&dp, g_ceph_context, cfg, context_pool, site, false, false, false, false, false, false, false, true, null_yield);
|
||||
driver = DriverManager::get_storage(&dp, g_ceph_context, cfg, context_pool, site, false, false, false, false, false, false, false, false, true, null_yield);
|
||||
if (!driver) {
|
||||
std::cerr << "couldn't init storage provider" << std::endl;
|
||||
return EIO;
|
||||
|
||||
@ -309,9 +309,15 @@ public:
|
||||
}
|
||||
virtual const char* name() const = 0;
|
||||
virtual RGWOpType get_type() { return RGW_OP_UNKNOWN; }
|
||||
virtual std::string canonical_name() const { return fmt::format("REST.{}.{}", s->info.method, name()); }
|
||||
virtual std::string canonical_name() const {
|
||||
return fmt::format("REST.{}.{}",
|
||||
s->info.method != nullptr ? s->info.method : "UNKNOWN",
|
||||
name());
|
||||
}
|
||||
// by default we log all bucket operations
|
||||
virtual bool always_do_bucket_logging() const { return s->bucket != nullptr; }
|
||||
virtual bool always_do_bucket_logging() const {
|
||||
return s->bucket != nullptr;
|
||||
}
|
||||
|
||||
virtual uint32_t op_mask() { return 0; }
|
||||
|
||||
@ -482,6 +488,12 @@ public:
|
||||
virtual int send_response_data(bufferlist& bl, off_t ofs, off_t len) = 0;
|
||||
|
||||
const char* name() const override { return "get_obj"; }
|
||||
std::string canonical_name() const override {
|
||||
if (get_torrent) {
|
||||
return fmt::format("REST.{}.TORRENT", s->info.method);
|
||||
}
|
||||
return fmt::format("REST.{}.OBJECT", s->info.method);
|
||||
}
|
||||
RGWOpType get_type() override { return RGW_OP_GET_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
virtual bool need_object_expiration() { return false; }
|
||||
@ -523,6 +535,7 @@ class RGWGetObjTags : public RGWOp {
|
||||
|
||||
virtual void send_response_data(bufferlist& bl) = 0;
|
||||
const char* name() const override { return "get_obj_tags"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_OBJ_TAGGING; }
|
||||
|
||||
@ -538,6 +551,7 @@ class RGWPutObjTags : public RGWOp {
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
const char* name() const override { return "put_obj_tags"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_OBJ_TAGGING; }
|
||||
|
||||
@ -550,6 +564,7 @@ class RGWDeleteObjTags: public RGWOp {
|
||||
void execute(optional_yield y) override;
|
||||
|
||||
const char* name() const override { return "delete_obj_tags"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_OBJ_TAGGING;}
|
||||
};
|
||||
@ -565,6 +580,7 @@ public:
|
||||
|
||||
virtual void send_response_data(bufferlist& bl) = 0;
|
||||
const char* name() const override { return "get_bucket_tags"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.TAGGING", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_TAGGING; }
|
||||
};
|
||||
@ -580,6 +596,7 @@ public:
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(const DoutPrefixProvider *dpp, optional_yield y) = 0;
|
||||
const char* name() const override { return "put_bucket_tags"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.TAGGING", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_TAGGING; }
|
||||
};
|
||||
@ -605,6 +622,7 @@ public:
|
||||
|
||||
virtual void send_response_data() = 0;
|
||||
const char* name() const override { return "get_bucket_replication"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.REPLICATION", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_REPLICATION; }
|
||||
};
|
||||
@ -620,6 +638,7 @@ public:
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
const char* name() const override { return "put_bucket_replication"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.REPLICATION", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_REPLICATION; }
|
||||
};
|
||||
@ -633,6 +652,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
|
||||
const char* name() const override { return "delete_bucket_replication"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.REPLICATION", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_BUCKET_REPLICATION;}
|
||||
};
|
||||
@ -708,6 +728,7 @@ public:
|
||||
void send_response() override = 0;
|
||||
|
||||
const char* name() const override { return "bulk_delete"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BULK_DELETE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_BULK_DELETE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
dmc::client_id dmclock_client() override { return dmc::client_id::data; }
|
||||
@ -779,6 +800,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
|
||||
const char* name() const override { return "bulk_upload"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BULK_UPLOAD", s->info.method); }
|
||||
|
||||
RGWOpType get_type() override {
|
||||
return RGW_OP_BULK_UPLOAD;
|
||||
@ -901,6 +923,7 @@ public:
|
||||
virtual bool supports_account_metadata() { return false; }
|
||||
|
||||
const char* name() const override { return "list_buckets"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_LIST_BUCKETS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
}; // class RGWListBuckets
|
||||
@ -985,6 +1008,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "list_bucket"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_LIST_BUCKET; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
virtual bool need_container_stats() { return false; }
|
||||
@ -999,6 +1023,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_bucket_location"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LOCATION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOCATION; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1017,6 +1042,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_bucket_versioning"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.VERSIONING", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_VERSIONING; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1045,6 +1071,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "set_bucket_versioning"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.VERSIONING", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_SET_BUCKET_VERSIONING; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1059,6 +1086,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_bucket_website"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.WEBSITE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_WEBSITE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1078,6 +1106,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "set_bucket_website"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.WEBSITE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_SET_BUCKET_WEBSITE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1092,6 +1121,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "delete_bucket_website"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.WEBSITE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_SET_BUCKET_WEBSITE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1141,6 +1171,7 @@ class RGWCreateBucket : public RGWOp {
|
||||
virtual int get_params(optional_yield y) { return 0; }
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "create_bucket"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_CREATE_BUCKET; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1158,6 +1189,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "delete_bucket"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_BUCKET; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
};
|
||||
@ -1327,6 +1359,10 @@ public:
|
||||
virtual int get_data(bufferlist& bl) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "put_obj"; }
|
||||
std::string canonical_name() const override {
|
||||
const bool multipart = !multipart_upload_id.empty();
|
||||
return fmt::format("REST.{}.{}", s->info.method, multipart ? "PART" : "OBJECT");
|
||||
}
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
dmc::client_id dmclock_client() override { return dmc::client_id::data; }
|
||||
@ -1379,6 +1415,7 @@ public:
|
||||
virtual int get_data(ceph::bufferlist& bl, bool& again) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "post_obj"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_POST_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
dmc::client_id dmclock_client() override { return dmc::client_id::data; }
|
||||
@ -1485,6 +1522,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "restore_obj"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.RESTORE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_RESTORE_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1522,6 +1560,7 @@ public:
|
||||
virtual int get_params(optional_yield y) { return 0; }
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "delete_obj"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
virtual bool need_object_expiration() { return false; }
|
||||
@ -1619,6 +1658,7 @@ public:
|
||||
virtual void send_partial_response(off_t ofs) {}
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "copy_obj"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_COPY_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
dmc::client_id dmclock_client() override { return dmc::client_id::data; }
|
||||
@ -1637,6 +1677,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_acls"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_ACLS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1658,6 +1699,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "put_acls"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_ACLS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1702,6 +1744,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_obj_attrs"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_ATTRIBUTES", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_OBJ_ATTRS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
}; /* RGWGetObjAttrs */
|
||||
@ -1719,6 +1762,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_lifecycle"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_LC; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1751,6 +1795,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "put_lifecycle"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_LC; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1764,6 +1809,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "delete_lifecycle"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_LC; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1779,6 +1825,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_cors"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_CORS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1798,6 +1845,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "put_cors"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_CORS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1813,6 +1861,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "delete_cors"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_CORS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1849,6 +1898,7 @@ public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
void execute(optional_yield y) override;
|
||||
const char* name() const override { return "put_bucket_encryption"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.ENCRYPTION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_ENCRYPTION; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1863,6 +1913,7 @@ public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
void execute(optional_yield y) override;
|
||||
const char* name() const override { return "get_bucket_encryption"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.ENCRYPTION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_ENCRYPTION; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1877,6 +1928,7 @@ public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
void execute(optional_yield y) override;
|
||||
const char* name() const override { return "delete_bucket_encryption"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.ENCRYPTION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_BUCKET_ENCRYPTION; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1894,6 +1946,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "get_request_payment"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.REQUEST_PAYMENT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_REQUEST_PAYMENT; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -1913,6 +1966,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "set_request_payment"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.REQUEST_PAYMENT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_SET_REQUEST_PAYMENT; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -1941,6 +1995,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "init_multipart"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_INIT_MULTIPART; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
virtual int prepare_encryption(std::map<std::string, bufferlist>& attrs) { return 0; }
|
||||
@ -1976,6 +2031,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "complete_multipart"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_COMPLETE_MULTIPART; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
bool always_do_bucket_logging() const override { return false; }
|
||||
@ -1993,6 +2049,7 @@ public:
|
||||
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "abort_multipart"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_ABORT_MULTIPART; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
};
|
||||
@ -2022,6 +2079,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "list_multipart"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_LIST_MULTIPART; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -2061,6 +2119,7 @@ public:
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
void send_response() override = 0;
|
||||
const char* name() const override { return "list_bucket_multiparts"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_LIST_BUCKET_MULTIPARTS; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -2161,6 +2220,7 @@ public:
|
||||
int ret) = 0;
|
||||
virtual void end_response() = 0;
|
||||
const char* name() const override { return "multi_object_delete"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.DELETE_MULTI_OBJECT", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_DELETE_MULTI_OBJ; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
|
||||
@ -2471,6 +2531,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
int get_params(optional_yield y);
|
||||
const char* name() const override { return "put_bucket_policy"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETPOLICY", s->info.method); }
|
||||
RGWOpType get_type() override {
|
||||
return RGW_OP_PUT_BUCKET_POLICY;
|
||||
}
|
||||
@ -2487,6 +2548,7 @@ public:
|
||||
}
|
||||
void execute(optional_yield y) override;
|
||||
const char* name() const override { return "get_bucket_policy"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETPOLICY", s->info.method); }
|
||||
RGWOpType get_type() override {
|
||||
return RGW_OP_GET_BUCKET_POLICY;
|
||||
}
|
||||
@ -2503,6 +2565,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
int get_params(optional_yield y);
|
||||
const char* name() const override { return "delete_bucket_policy"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETPOLICY", s->info.method); }
|
||||
RGWOpType get_type() override {
|
||||
return RGW_OP_DELETE_BUCKET_POLICY;
|
||||
}
|
||||
@ -2522,6 +2585,7 @@ public:
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
const char* name() const override { return "put_bucket_object_lock"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_LOCK", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_OBJ_LOCK; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
};
|
||||
@ -2533,6 +2597,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
virtual void send_response() override = 0;
|
||||
const char* name() const override {return "get_bucket_object_lock"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_LOCK", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_OBJ_LOCK; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -2551,6 +2616,7 @@ public:
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
const char* name() const override { return "put_obj_retention"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_OBJ_RETENTION; }
|
||||
};
|
||||
@ -2564,6 +2630,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
virtual void send_response() override = 0;
|
||||
const char* name() const override {return "get_obj_retention"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_OBJ_RETENTION; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -2579,6 +2646,7 @@ public:
|
||||
virtual void send_response() override = 0;
|
||||
virtual int get_params(optional_yield y) = 0;
|
||||
const char* name() const override { return "put_obj_legal_hold"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_OBJ_LEGAL_HOLD; }
|
||||
};
|
||||
@ -2592,6 +2660,7 @@ public:
|
||||
void execute(optional_yield y) override;
|
||||
virtual void send_response() override = 0;
|
||||
const char* name() const override {return "get_obj_legal_hold"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_OBJ_LEGAL_HOLD; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -2676,6 +2745,7 @@ public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
const char* name() const override { return "put_bucket_public_access_block";}
|
||||
virtual RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_PUBLIC_ACCESS_BLOCK; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.PUBLIC_ACCESS_BLOCK", s->info.method); }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
int get_params(optional_yield y);
|
||||
void execute(optional_yield y) override;
|
||||
@ -2688,6 +2758,7 @@ protected:
|
||||
public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
const char* name() const override { return "get_bucket_public_access_block";}
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.PUBLIC_ACCESS_BLOCK", s->info.method); }
|
||||
virtual RGWOpType get_type() override { return RGW_OP_GET_BUCKET_PUBLIC_ACCESS_BLOCK; }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
int get_params(optional_yield y);
|
||||
@ -2701,6 +2772,7 @@ protected:
|
||||
public:
|
||||
int verify_permission(optional_yield y) override;
|
||||
const char* name() const override { return "delete_bucket_public_access_block";}
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.PUBLIC_ACCESS_BLOCK", s->info.method); }
|
||||
virtual RGWOpType get_type() override { return RGW_OP_DELETE_BUCKET_PUBLIC_ACCESS_BLOCK; }
|
||||
virtual uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
int get_params(optional_yield y);
|
||||
|
||||
@ -129,7 +129,10 @@ void RGWRealmReloader::reload()
|
||||
cct->_conf->rgw_enable_quota_threads,
|
||||
cct->_conf->rgw_run_sync_thread,
|
||||
cct->_conf.get_val<bool>("rgw_dynamic_resharding"),
|
||||
true, true, null_yield, // run notification thread
|
||||
true,
|
||||
true, // run notification thread
|
||||
true, // run bucket logging thread
|
||||
null_yield,
|
||||
cct->_conf->rgw_cache_enabled);
|
||||
}
|
||||
|
||||
|
||||
@ -129,39 +129,30 @@ public:
|
||||
}
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetObjTags_ObjStore : public RGWGetObjTags {
|
||||
public:
|
||||
RGWGetObjTags_ObjStore() {};
|
||||
~RGWGetObjTags_ObjStore() {};
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutObjTags_ObjStore: public RGWPutObjTags {
|
||||
public:
|
||||
RGWPutObjTags_ObjStore() {};
|
||||
~RGWPutObjTags_ObjStore() {};
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetBucketTags_ObjStore : public RGWGetBucketTags {
|
||||
public:
|
||||
RGWGetBucketTags_ObjStore() = default;
|
||||
virtual ~RGWGetBucketTags_ObjStore() = default;
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutBucketTags_ObjStore: public RGWPutBucketTags {
|
||||
public:
|
||||
RGWPutBucketTags_ObjStore() = default;
|
||||
virtual ~RGWPutBucketTags_ObjStore() = default;
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetBucketReplication_ObjStore : public RGWGetBucketReplication {
|
||||
@ -186,56 +177,42 @@ class RGWListBuckets_ObjStore : public RGWListBuckets {
|
||||
public:
|
||||
RGWListBuckets_ObjStore() {}
|
||||
~RGWListBuckets_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetUsage_ObjStore : public RGWGetUsage {
|
||||
public:
|
||||
RGWGetUsage_ObjStore() {}
|
||||
~RGWGetUsage_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.USER_USAGE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWListBucket_ObjStore : public RGWListBucket {
|
||||
public:
|
||||
RGWListBucket_ObjStore() {}
|
||||
~RGWListBucket_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWStatAccount_ObjStore : public RGWStatAccount {
|
||||
public:
|
||||
RGWStatAccount_ObjStore() {}
|
||||
~RGWStatAccount_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_STATUS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWStatBucket_ObjStore : public RGWStatBucket {
|
||||
public:
|
||||
RGWStatBucket_ObjStore() {}
|
||||
~RGWStatBucket_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_STATUS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWCreateBucket_ObjStore : public RGWCreateBucket {
|
||||
public:
|
||||
RGWCreateBucket_ObjStore() {}
|
||||
~RGWCreateBucket_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteBucket_ObjStore : public RGWDeleteBucket {
|
||||
public:
|
||||
RGWDeleteBucket_ObjStore() {}
|
||||
~RGWDeleteBucket_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutObj_ObjStore : public RGWPutObj
|
||||
@ -247,11 +224,6 @@ public:
|
||||
int verify_params() override;
|
||||
int get_params(optional_yield y) override;
|
||||
int get_data(bufferlist& bl) override;
|
||||
|
||||
virtual std::string canonical_name() const override {
|
||||
const bool multipart = !multipart_upload_id.empty();
|
||||
return fmt::format("REST.{}.{}", s->info.method, multipart ? "PART" : "OBJECT");
|
||||
}
|
||||
};
|
||||
|
||||
class RGWPostObj_ObjStore : public RGWPostObj
|
||||
@ -322,7 +294,6 @@ public:
|
||||
~RGWPostObj_ObjStore() override {}
|
||||
|
||||
int verify_params() override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
|
||||
@ -331,8 +302,6 @@ class RGWPutMetadataAccount_ObjStore : public RGWPutMetadataAccount
|
||||
public:
|
||||
RGWPutMetadataAccount_ObjStore() {}
|
||||
~RGWPutMetadataAccount_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_METADATA", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutMetadataBucket_ObjStore : public RGWPutMetadataBucket
|
||||
@ -340,8 +309,6 @@ class RGWPutMetadataBucket_ObjStore : public RGWPutMetadataBucket
|
||||
public:
|
||||
RGWPutMetadataBucket_ObjStore() {}
|
||||
~RGWPutMetadataBucket_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_METADATA", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutMetadataObject_ObjStore : public RGWPutMetadataObject
|
||||
@ -349,24 +316,18 @@ class RGWPutMetadataObject_ObjStore : public RGWPutMetadataObject
|
||||
public:
|
||||
RGWPutMetadataObject_ObjStore() {}
|
||||
~RGWPutMetadataObject_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_METADATA", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWRestoreObj_ObjStore : public RGWRestoreObj {
|
||||
public:
|
||||
RGWRestoreObj_ObjStore() {}
|
||||
~RGWRestoreObj_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteObj_ObjStore : public RGWDeleteObj {
|
||||
public:
|
||||
RGWDeleteObj_ObjStore() {}
|
||||
~RGWDeleteObj_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetCrossDomainPolicy_ObjStore : public RGWGetCrossDomainPolicy {
|
||||
@ -385,25 +346,19 @@ class RGWCopyObj_ObjStore : public RGWCopyObj {
|
||||
public:
|
||||
RGWCopyObj_ObjStore() {}
|
||||
~RGWCopyObj_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetACLs_ObjStore : public RGWGetACLs {
|
||||
public:
|
||||
RGWGetACLs_ObjStore() {}
|
||||
~RGWGetACLs_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutACLs_ObjStore : public RGWPutACLs {
|
||||
public:
|
||||
RGWPutACLs_ObjStore() {}
|
||||
~RGWPutACLs_ObjStore() override {}
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetObjAttrs_ObjStore : public RGWGetObjAttrs {
|
||||
@ -421,8 +376,6 @@ class RGWGetLC_ObjStore : public RGWGetLC {
|
||||
public:
|
||||
RGWGetLC_ObjStore() {}
|
||||
~RGWGetLC_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutLC_ObjStore : public RGWPutLC {
|
||||
@ -431,47 +384,36 @@ public:
|
||||
~RGWPutLC_ObjStore() override {}
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteLC_ObjStore : public RGWDeleteLC {
|
||||
public:
|
||||
RGWDeleteLC_ObjStore() {}
|
||||
~RGWDeleteLC_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetCORS_ObjStore : public RGWGetCORS {
|
||||
public:
|
||||
RGWGetCORS_ObjStore() {}
|
||||
~RGWGetCORS_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutCORS_ObjStore : public RGWPutCORS {
|
||||
public:
|
||||
RGWPutCORS_ObjStore() {}
|
||||
~RGWPutCORS_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteCORS_ObjStore : public RGWDeleteCORS {
|
||||
public:
|
||||
RGWDeleteCORS_ObjStore() {}
|
||||
~RGWDeleteCORS_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWOptionsCORS_ObjStore : public RGWOptionsCORS {
|
||||
public:
|
||||
RGWOptionsCORS_ObjStore() {}
|
||||
~RGWOptionsCORS_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetBucketEncryption_ObjStore : public RGWGetBucketEncryption {
|
||||
@ -496,16 +438,12 @@ class RGWInitMultipart_ObjStore : public RGWInitMultipart {
|
||||
public:
|
||||
RGWInitMultipart_ObjStore() {}
|
||||
~RGWInitMultipart_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWCompleteMultipart_ObjStore : public RGWCompleteMultipart {
|
||||
public:
|
||||
RGWCompleteMultipart_ObjStore() {}
|
||||
~RGWCompleteMultipart_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
int get_params(optional_yield y) override;
|
||||
};
|
||||
|
||||
@ -513,16 +451,12 @@ class RGWAbortMultipart_ObjStore : public RGWAbortMultipart {
|
||||
public:
|
||||
RGWAbortMultipart_ObjStore() {}
|
||||
~RGWAbortMultipart_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWListMultipart_ObjStore : public RGWListMultipart {
|
||||
public:
|
||||
RGWListMultipart_ObjStore() {}
|
||||
~RGWListMultipart_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
|
||||
int get_params(optional_yield y) override;
|
||||
};
|
||||
|
||||
@ -530,8 +464,6 @@ class RGWListBucketMultiparts_ObjStore : public RGWListBucketMultiparts {
|
||||
public:
|
||||
RGWListBucketMultiparts_ObjStore() {}
|
||||
~RGWListBucketMultiparts_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
|
||||
int get_params(optional_yield y) override;
|
||||
};
|
||||
|
||||
@ -539,16 +471,12 @@ class RGWBulkDelete_ObjStore : public RGWBulkDelete {
|
||||
public:
|
||||
RGWBulkDelete_ObjStore() {}
|
||||
~RGWBulkDelete_ObjStore() override {}
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_DELETE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWBulkUploadOp_ObjStore : public RGWBulkUploadOp {
|
||||
public:
|
||||
RGWBulkUploadOp_ObjStore() = default;
|
||||
~RGWBulkUploadOp_ObjStore() = default;
|
||||
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_UPLOAD", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteMultiObj_ObjStore : public RGWDeleteMultiObj {
|
||||
@ -557,7 +485,6 @@ public:
|
||||
~RGWDeleteMultiObj_ObjStore() override {}
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.DELETE_MULTI_OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWInfo_ObjStore : public RGWInfo {
|
||||
@ -583,14 +510,12 @@ class RGWPutObjRetention_ObjStore : public RGWPutObjRetention {
|
||||
public:
|
||||
RGWPutObjRetention_ObjStore() = default;
|
||||
~RGWPutObjRetention_ObjStore() override = default;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetObjRetention_ObjStore : public RGWGetObjRetention {
|
||||
public:
|
||||
RGWGetObjRetention_ObjStore() = default;
|
||||
~RGWGetObjRetention_ObjStore() = default;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.RETENTION", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWPutObjLegalHold_ObjStore : public RGWPutObjLegalHold {
|
||||
@ -598,13 +523,11 @@ public:
|
||||
RGWPutObjLegalHold_ObjStore() = default;
|
||||
~RGWPutObjLegalHold_ObjStore() override = default;
|
||||
int get_params(optional_yield y) override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetObjLegalHold_ObjStore : public RGWGetObjLegalHold {
|
||||
public:
|
||||
RGWGetObjLegalHold_ObjStore() = default;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.LEGAL_HOLD", s->info.method); }
|
||||
~RGWGetObjLegalHold_ObjStore() = default;
|
||||
};
|
||||
|
||||
|
||||
@ -111,6 +111,7 @@ public:
|
||||
}
|
||||
|
||||
void send_response() override {
|
||||
set_req_state_err(s, op_ret);
|
||||
dump_errno(s);
|
||||
if (mtime) {
|
||||
dump_last_modified(s, *mtime);
|
||||
@ -123,7 +124,9 @@ public:
|
||||
s->formatter->close_section();
|
||||
rgw_flush_formatter_and_reset(s, s->formatter);
|
||||
}
|
||||
|
||||
const char* name() const override { return "get_bucket_logging"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LOGGING", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOGGING; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
};
|
||||
@ -135,6 +138,7 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
// and usd in execute()
|
||||
rgw::bucketlogging::configuration configuration;
|
||||
std::unique_ptr<rgw::sal::Bucket> target_bucket;
|
||||
std::string old_obj; // used when conf change triggers a rollover
|
||||
|
||||
int init_processing(optional_yield y) override {
|
||||
if (const auto ret = verify_bucket_logging_params(this, s); ret < 0) {
|
||||
@ -177,13 +181,13 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
|
||||
rgw_bucket target_bucket_id;
|
||||
if (const auto ret = rgw::bucketlogging::get_bucket_id(configuration.target_bucket, s->bucket_tenant, target_bucket_id); ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to parse target bucket '" << configuration.target_bucket << "', ret = " << ret << dendl;
|
||||
ldpp_dout(this, 1) << "ERROR: failed to parse logging bucket '" << configuration.target_bucket << "', ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
|
||||
if (const auto ret = driver->load_bucket(this, target_bucket_id,
|
||||
&target_bucket, y); ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get target bucket '" << target_bucket_id << "', ret = " << ret << dendl;
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get logging bucket '" << target_bucket_id << "', ret = " << ret << dendl;
|
||||
return ret;
|
||||
}
|
||||
|
||||
@ -215,6 +219,7 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
}
|
||||
|
||||
const char* name() const override { return "put_bucket_logging"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LOGGING", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_LOGGING; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
|
||||
@ -232,7 +237,7 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
}
|
||||
|
||||
if (!configuration.enabled) {
|
||||
op_ret = rgw::bucketlogging::source_bucket_cleanup(this, driver, src_bucket.get(), true, y);
|
||||
op_ret = rgw::bucketlogging::source_bucket_cleanup(this, driver, src_bucket.get(), true, y, &old_obj);
|
||||
return;
|
||||
}
|
||||
|
||||
@ -240,14 +245,14 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
const auto& target_bucket_id = target_bucket->get_key();
|
||||
if (target_bucket_id == src_bucket_id) {
|
||||
// target bucket must be different from source bucket (cannot change later on)
|
||||
ldpp_dout(this, 1) << "ERROR: target bucket '" << target_bucket_id << "' must be different from source bucket" << dendl;
|
||||
ldpp_dout(this, 1) << "ERROR: logging bucket '" << target_bucket_id << "' must be different from source bucket" << dendl;
|
||||
op_ret = -EINVAL;
|
||||
return;
|
||||
}
|
||||
const auto& target_info = target_bucket->get_info();
|
||||
if (target_info.zonegroup != src_bucket->get_info().zonegroup) {
|
||||
// target bucket must be in the same zonegroup as source bucket (cannot change later on)
|
||||
ldpp_dout(this, 1) << "ERROR: target bucket '" << target_bucket_id << "' zonegroup '" <<
|
||||
ldpp_dout(this, 1) << "ERROR: logging bucket '" << target_bucket_id << "' zonegroup '" <<
|
||||
target_info.zonegroup << "' is different from the source bucket '" << src_bucket_id <<
|
||||
"' zonegroup '" << src_bucket->get_info().zonegroup << "'" << dendl;
|
||||
op_ret = -EINVAL;
|
||||
@ -299,30 +304,51 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
ldpp_dout(this, 20) << "INFO: new logging configuration added to bucket '" << src_bucket_id << "'. configuration: " <<
|
||||
configuration.to_json_str() << dendl;
|
||||
if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, target_bucket, src_bucket_id, true, y); ret < 0) {
|
||||
ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of target bucket '" <<
|
||||
ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of logging bucket '" <<
|
||||
target_bucket_id << "', ret = " << ret << dendl;
|
||||
}
|
||||
} else if (*old_conf != configuration) {
|
||||
// conf changed - do cleanup
|
||||
if (const auto ret = commit_logging_object(*old_conf, target_bucket, this, y, nullptr); ret < 0) {
|
||||
ldpp_dout(this, 1) << "WARNING: could not commit pending logging object when updating logging configuration of bucket '" <<
|
||||
src_bucket->get_key() << "', ret = " << ret << dendl;
|
||||
RGWObjVersionTracker objv_tracker;
|
||||
std::string obj_name;
|
||||
if (int ret = target_bucket->get_logging_object_name(obj_name, old_conf->target_prefix, y, this, nullptr); ret < 0 && ret != -ENOENT) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get name of logging object of logging bucket '" <<
|
||||
target_bucket_id << "' and prefix '" << configuration.target_prefix << "', ret = " << ret << dendl;
|
||||
op_ret = ret;
|
||||
return;
|
||||
}
|
||||
const auto region = driver->get_zone()->get_zonegroup().get_api_name();
|
||||
if (const auto ret = rollover_logging_object(*old_conf,
|
||||
target_bucket,
|
||||
obj_name,
|
||||
this,
|
||||
region,
|
||||
src_bucket,
|
||||
y,
|
||||
false, // rollover should happen even if commit failed
|
||||
&objv_tracker,
|
||||
false,
|
||||
&old_obj); ret < 0) {
|
||||
ldpp_dout(this, 1) << "WARNING: failed to flush pending logging object '" << obj_name << "'"
|
||||
<< " to target bucket '" << target_bucket_id << "'. "
|
||||
<< "' when updating logging configuration of bucket '" << src_bucket->get_key() << ". error: " << ret << dendl;
|
||||
} else {
|
||||
ldpp_dout(this, 20) << "INFO: committed pending logging object when updating logging configuration of bucket '" <<
|
||||
src_bucket->get_key() << "'" << dendl;
|
||||
ldpp_dout(this, 20) << "INFO: flushed pending logging object '" << old_obj
|
||||
<< "' to target bucket '" << target_bucket_id << "' when updating logging configuration of bucket '"
|
||||
<< src_bucket->get_key() << "'" << dendl;
|
||||
}
|
||||
if (old_conf->target_bucket != configuration.target_bucket) {
|
||||
rgw_bucket old_target_bucket_id;
|
||||
if (const auto ret = rgw::bucketlogging::get_bucket_id(old_conf->target_bucket, s->bucket_tenant, old_target_bucket_id); ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to parse target bucket '" << old_conf->target_bucket << "', ret = " << ret << dendl;
|
||||
ldpp_dout(this, 1) << "ERROR: failed to parse logging bucket '" << old_conf->target_bucket << "', ret = " << ret << dendl;
|
||||
return;
|
||||
}
|
||||
if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, driver, old_target_bucket_id, src_bucket_id, false, y); ret < 0) {
|
||||
ldpp_dout(this, 1) << "WARNING: failed to remove source bucket '" << src_bucket_id << "' from logging sources of original target bucket '" <<
|
||||
ldpp_dout(this, 1) << "WARNING: failed to remove source bucket '" << src_bucket_id << "' from logging sources of original logging bucket '" <<
|
||||
old_target_bucket_id << "', ret = " << ret << dendl;
|
||||
}
|
||||
if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, target_bucket, src_bucket_id, true, y); ret < 0) {
|
||||
ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of target bucket '" <<
|
||||
ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of logging bucket '" <<
|
||||
target_bucket_id << "', ret = " << ret << dendl;
|
||||
}
|
||||
}
|
||||
@ -332,6 +358,19 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
ldpp_dout(this, 20) << "INFO: logging configuration of bucket '" << src_bucket_id << "' did not change" << dendl;
|
||||
}
|
||||
}
|
||||
|
||||
void send_response() override {
|
||||
set_req_state_err(s, op_ret);
|
||||
dump_errno(s);
|
||||
end_header(s, this, to_mime_type(s->format));
|
||||
if (!old_obj.empty()) {
|
||||
dump_start(s);
|
||||
s->formatter->open_object_section_in_ns("PutBucketLoggingOutput", XMLNS_AWS_S3);
|
||||
s->formatter->dump_string("FlushedLoggingObject", old_obj);
|
||||
s->formatter->close_section();
|
||||
rgw_flush_formatter_and_reset(s, s->formatter);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Post /<bucket name>/?logging
|
||||
@ -373,38 +412,44 @@ class RGWPostBucketLoggingOp : public RGWDefaultResponseOp {
|
||||
}
|
||||
|
||||
const char* name() const override { return "post_bucket_logging"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.LOGGING", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_POST_BUCKET_LOGGING; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
|
||||
void execute(optional_yield y) override {
|
||||
const auto& target_bucket_id = target_bucket->get_key();
|
||||
// make sure that the logging source attribute is up-to-date
|
||||
if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, target_bucket, source_bucket->get_key(), true, y); ret < 0) {
|
||||
ldpp_dout(this, 5) << "WARNING: failed to update logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES
|
||||
<< "' in logging target '" << target_bucket_id << "'. error: " << ret << dendl;
|
||||
}
|
||||
std::string obj_name;
|
||||
RGWObjVersionTracker objv_tracker;
|
||||
op_ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, this, &objv_tracker);
|
||||
if (op_ret < 0) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get pending logging object name from target bucket '" << target_bucket_id << "'" << dendl;
|
||||
op_ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, y, this, &objv_tracker);
|
||||
if (op_ret < 0 && op_ret != -ENOENT) {
|
||||
ldpp_dout(this, 1) << "ERROR: failed to get pending logging object name from logging bucket '" << target_bucket_id <<
|
||||
"'. error: " << op_ret << dendl;
|
||||
return;
|
||||
}
|
||||
if (op_ret == -ENOENT) {
|
||||
// no pending object - nothing to flush
|
||||
ldpp_dout(this, 5) << "INFO: no pending logging object in logging bucket '" << target_bucket_id << "'. new object should be created" << dendl;
|
||||
}
|
||||
const auto region = driver->get_zone()->get_zonegroup().get_api_name();
|
||||
op_ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, this, region, source_bucket, null_yield, true, &objv_tracker, &old_obj);
|
||||
op_ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, this, region, source_bucket, y, true, &objv_tracker, false, &old_obj);
|
||||
if (op_ret < 0) {
|
||||
if (op_ret == -ENOENT) {
|
||||
ldpp_dout(this, 5) << "WARNING: no pending logging object '" << obj_name << "'. nothing to flush"
|
||||
<< " to target bucket '" << target_bucket_id << "'. "
|
||||
<< " last committed object is '" << old_obj << "'" << dendl;
|
||||
op_ret = 0;
|
||||
} else {
|
||||
ldpp_dout(this, 1) << "ERROR: failed flush pending logging object '" << obj_name << "'"
|
||||
<< " to target bucket '" << target_bucket_id << "'. "
|
||||
<< " last committed object is '" << old_obj << "'" << dendl;
|
||||
}
|
||||
ldpp_dout(this, 1) << "ERROR: failed to flush pending logging object '" << obj_name << "'"
|
||||
<< " to logging bucket '" << target_bucket_id << "'. "
|
||||
<< " last committed object is '" << old_obj <<
|
||||
"'. error: " << op_ret << dendl;
|
||||
return;
|
||||
}
|
||||
ldpp_dout(this, 20) << "INFO: flushed pending logging object '" << old_obj
|
||||
<< "' to target bucket '" << target_bucket_id << "'" << dendl;
|
||||
<< "' to logging bucket '" << target_bucket_id << "'" << dendl;
|
||||
}
|
||||
|
||||
void send_response() override {
|
||||
set_req_state_err(s, op_ret);
|
||||
dump_errno(s);
|
||||
end_header(s, this, to_mime_type(s->format));
|
||||
dump_start(s);
|
||||
|
||||
@ -1189,6 +1189,7 @@ public:
|
||||
}
|
||||
|
||||
const char* name() const override { return "pubsub_notification_create_s3"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.NOTIFICATION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUBSUB_NOTIF_CREATE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
|
||||
|
||||
@ -1444,6 +1445,7 @@ class RGWPSDeleteNotifOp : public RGWDefaultResponseOp {
|
||||
}
|
||||
|
||||
const char* name() const override { return "pubsub_notification_delete_s3"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.NOTIFICATION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUBSUB_NOTIF_DELETE; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_DELETE; }
|
||||
|
||||
@ -1548,6 +1550,7 @@ public:
|
||||
}
|
||||
|
||||
const char* name() const override { return "pubsub_notifications_get_s3"; }
|
||||
std::string canonical_name() const override { return fmt::format("REST.{}.NOTIFICATION", s->info.method); }
|
||||
RGWOpType get_type() override { return RGW_OP_PUBSUB_NOTIF_LIST; }
|
||||
uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
|
||||
|
||||
|
||||
@ -83,7 +83,6 @@ class RGWDeleteObjTags_ObjStore_S3 : public RGWDeleteObjTags
|
||||
public:
|
||||
~RGWDeleteObjTags_ObjStore_S3() override {}
|
||||
void send_response() override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWGetBucketTags_ObjStore_S3 : public RGWGetBucketTags_ObjStore
|
||||
@ -215,7 +214,6 @@ public:
|
||||
~RGWGetBucketWebsite_ObjStore_S3() override {}
|
||||
|
||||
void send_response() override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWSetBucketWebsite_ObjStore_S3 : public RGWSetBucketWebsite {
|
||||
@ -225,7 +223,6 @@ public:
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
void send_response() override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWDeleteBucketWebsite_ObjStore_S3 : public RGWDeleteBucketWebsite {
|
||||
@ -234,7 +231,6 @@ public:
|
||||
~RGWDeleteBucketWebsite_ObjStore_S3() override {}
|
||||
|
||||
void send_response() override;
|
||||
virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
|
||||
};
|
||||
|
||||
class RGWStatBucket_ObjStore_S3 : public RGWStatBucket_ObjStore {
|
||||
@ -604,7 +600,6 @@ class RGWConfigBucketMetaSearch_ObjStore_S3 : public RGWConfigBucketMetaSearch {
|
||||
public:
|
||||
RGWConfigBucketMetaSearch_ObjStore_S3() {}
|
||||
~RGWConfigBucketMetaSearch_ObjStore_S3() {}
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); }
|
||||
|
||||
int get_params(optional_yield y) override;
|
||||
void send_response() override;
|
||||
@ -622,7 +617,6 @@ class RGWDelBucketMetaSearch_ObjStore_S3 : public RGWDelBucketMetaSearch {
|
||||
public:
|
||||
RGWDelBucketMetaSearch_ObjStore_S3() {}
|
||||
~RGWDelBucketMetaSearch_ObjStore_S3() {}
|
||||
virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); }
|
||||
|
||||
void send_response() override;
|
||||
};
|
||||
|
||||
@ -97,4 +97,5 @@ public:
|
||||
return RGWGetObj_ObjStore_S3::get_params(y);
|
||||
}
|
||||
}
|
||||
std::string canonical_name() const override { return fmt::format("WEBSITE.{}.OBJECT", s->info.method); }
|
||||
};
|
||||
|
||||
@ -95,7 +95,8 @@ rgw::sal::Driver* DriverManager::init_storage_provider(const DoutPrefixProvider*
|
||||
bool quota_threads,
|
||||
bool run_sync_thread,
|
||||
bool run_reshard_thread,
|
||||
bool run_notification_thread,
|
||||
bool run_notification_thread,
|
||||
bool run_bucket_logging_thread,
|
||||
bool use_cache,
|
||||
bool use_gc,
|
||||
bool background_tasks,
|
||||
@ -117,6 +118,7 @@ rgw::sal::Driver* DriverManager::init_storage_provider(const DoutPrefixProvider*
|
||||
.set_run_sync_thread(run_sync_thread)
|
||||
.set_run_reshard_thread(run_reshard_thread)
|
||||
.set_run_notification_thread(run_notification_thread)
|
||||
.set_run_bucket_logging_thread(run_bucket_logging_thread)
|
||||
.init_begin(cct, dpp, background_tasks, site_config) < 0) {
|
||||
delete driver;
|
||||
return nullptr;
|
||||
@ -149,6 +151,7 @@ rgw::sal::Driver* DriverManager::init_storage_provider(const DoutPrefixProvider*
|
||||
.set_run_sync_thread(run_sync_thread)
|
||||
.set_run_reshard_thread(run_reshard_thread)
|
||||
.set_run_notification_thread(run_notification_thread)
|
||||
.set_run_bucket_logging_thread(run_bucket_logging_thread)
|
||||
.init_begin(cct, dpp, background_tasks, site_config) < 0) {
|
||||
delete driver;
|
||||
return nullptr;
|
||||
|
||||
@ -1053,12 +1053,15 @@ class Bucket {
|
||||
RGWObjVersionTracker* objv_tracker) = 0;
|
||||
/** Move the pending bucket logging object into the bucket
|
||||
if "last_committed" is not null, it will be set to the name of the last committed object
|
||||
if async is true, write the entry to the commit lists to be processed by the BucketLoggingManager
|
||||
* */
|
||||
virtual int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed) = 0;
|
||||
virtual int commit_logging_object(const std::string& obj_name, optional_yield y,
|
||||
const DoutPrefixProvider *dpp, const std::string& prefix,
|
||||
std::string* last_committed, bool async) = 0;
|
||||
//** Remove the pending bucket logging object */
|
||||
virtual int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) = 0;
|
||||
virtual int remove_logging_object(const std::string& obj_name, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp) = 0;
|
||||
/** Write a record to the pending bucket logging object */
|
||||
virtual int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) = 0;
|
||||
virtual int write_logging_object(const std::string& obj_name, const std::string& record, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) = 0;
|
||||
|
||||
/* dang - This is temporary, until the API is completed */
|
||||
virtual rgw_bucket& get_key() = 0;
|
||||
@ -1948,6 +1951,7 @@ public:
|
||||
bool run_sync_thread,
|
||||
bool run_reshard_thread,
|
||||
bool run_notification_thread,
|
||||
bool run_bucket_logging_thread,
|
||||
bool background_tasks,
|
||||
optional_yield y,
|
||||
bool use_cache = true,
|
||||
@ -1961,6 +1965,7 @@ public:
|
||||
run_sync_thread,
|
||||
run_reshard_thread,
|
||||
run_notification_thread,
|
||||
run_bucket_logging_thread,
|
||||
use_cache, use_gc,
|
||||
background_tasks, y);
|
||||
return driver;
|
||||
@ -1988,6 +1993,7 @@ public:
|
||||
bool run_sync_thread,
|
||||
bool run_reshard_thread,
|
||||
bool run_notification_thread,
|
||||
bool run_bucket_logging_thread,
|
||||
bool use_metadata_cache,
|
||||
bool use_gc, bool background_tasks,
|
||||
optional_yield y);
|
||||
|
||||
@ -697,14 +697,14 @@ public:
|
||||
RGWObjVersionTracker* objv_tracker) override {
|
||||
return next->remove_logging_object_name(prefix, y, dpp, objv_tracker);
|
||||
}
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed) override {
|
||||
return next->commit_logging_object(obj_name, y, dpp, prefix, last_committed);
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed, bool async) override {
|
||||
return next->commit_logging_object(obj_name, y, dpp, prefix, last_committed, async);
|
||||
}
|
||||
int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override {
|
||||
return next->remove_logging_object(obj_name, y, dpp);
|
||||
int remove_logging_object(const std::string& obj_name, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp) override {
|
||||
return next->remove_logging_object(obj_name, prefix, y, dpp);
|
||||
}
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
|
||||
return next->write_logging_object(obj_name, record, y, dpp, async_completion);
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
|
||||
return next->write_logging_object(obj_name, record, prefix, y, dpp, async_completion);
|
||||
}
|
||||
|
||||
virtual rgw_bucket& get_key() override { return next->get_key(); }
|
||||
|
||||
@ -266,9 +266,9 @@ class StoreBucket : public Bucket {
|
||||
optional_yield y,
|
||||
const DoutPrefixProvider *dpp,
|
||||
RGWObjVersionTracker* objv_tracker) override { return 0; }
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed) override { return 0; }
|
||||
int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override { return 0; }
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
|
||||
int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp, const std::string& prefix, std::string* last_committed, bool async) override { return 0; }
|
||||
int remove_logging_object(const std::string& obj_name, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp) override { return 0; }
|
||||
int write_logging_object(const std::string& obj_name, const std::string& record, const std::string& prefix, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
@ -301,6 +301,7 @@ void RGWZoneParams::decode_json(JSONObj *obj)
|
||||
JSONDecoder::decode_json("topics_pool", topics_pool, obj);
|
||||
JSONDecoder::decode_json("account_pool", account_pool, obj);
|
||||
JSONDecoder::decode_json("group_pool", group_pool, obj);
|
||||
JSONDecoder::decode_json("bucket_logging_pool", bucket_logging_pool, obj);
|
||||
JSONDecoder::decode_json("system_key", system_key, obj);
|
||||
JSONDecoder::decode_json("placement_pools", placement_pools, obj);
|
||||
JSONDecoder::decode_json("tier_config", tier_config, obj);
|
||||
@ -330,6 +331,7 @@ void RGWZoneParams::dump(Formatter *f) const
|
||||
encode_json("topics_pool", topics_pool, f);
|
||||
encode_json("account_pool", account_pool, f);
|
||||
encode_json("group_pool", group_pool, f);
|
||||
encode_json("bucket_logging_pool", bucket_logging_pool, f);
|
||||
encode_json_plain("system_key", system_key, f);
|
||||
encode_json("placement_pools", placement_pools, f);
|
||||
encode_json("tier_config", tier_config, f);
|
||||
@ -479,6 +481,7 @@ void add_zone_pools(const RGWZoneParams& info,
|
||||
pools.insert(info.dedup_pool);
|
||||
pools.insert(info.gc_pool);
|
||||
pools.insert(info.log_pool);
|
||||
pools.insert(info.bucket_logging_pool);
|
||||
pools.insert(info.intent_log_pool);
|
||||
pools.insert(info.usage_log_pool);
|
||||
pools.insert(info.user_keys_pool);
|
||||
@ -1295,6 +1298,7 @@ int init_zone_pool_names(const DoutPrefixProvider *dpp, optional_yield y,
|
||||
info.otp_pool = fix_zone_pool_dup(pools, info.name, ".rgw.otp", info.otp_pool);
|
||||
info.oidc_pool = fix_zone_pool_dup(pools, info.name, ".rgw.meta:oidc", info.oidc_pool);
|
||||
info.notif_pool = fix_zone_pool_dup(pools, info.name, ".rgw.log:notif", info.notif_pool);
|
||||
info.bucket_logging_pool = fix_zone_pool_dup(pools, info.name, ".rgw.log:logging", info.bucket_logging_pool);
|
||||
info.topics_pool =
|
||||
fix_zone_pool_dup(pools, info.name, ".rgw.meta:topics", info.topics_pool);
|
||||
info.account_pool = fix_zone_pool_dup(pools, info.name, ".rgw.meta:accounts", info.account_pool);
|
||||
|
||||
@ -52,6 +52,7 @@
|
||||
bucket radoslist list rados objects backing bucket's objects
|
||||
bucket logging flush flush pending log records object of source bucket to the log bucket
|
||||
bucket logging info get info on bucket logging configuration on source bucket or list of sources in log bucket
|
||||
bucket logging list list the log objects pending commit for the source bucket
|
||||
bi get retrieve bucket index object entries
|
||||
bi put store bucket index object entries
|
||||
bi list list raw bucket index entries
|
||||
|
||||
@ -351,7 +351,7 @@ int main(int argc, const char **argv)
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true, true, null_yield,
|
||||
true, true, true, null_yield,
|
||||
false));
|
||||
if (!store) {
|
||||
std::cerr << "couldn't init storage provider" << std::endl;
|
||||
|
||||
Loading…
Reference in New Issue
Block a user
