mirror of
https://github.com/cubefs/cubefs.git
synced 2026-08-02 02:00:56 +00:00
docs(config): update docs for supporting consumer group
Signed-off-by: pengtianyue <pengtianyue@oppo.com>
This commit is contained in:
pengtianyue
pengtianyue
parent
4c01579daf
commit
a90af25ce7
@ -71,42 +71,29 @@ Scheduler的配置是基于[公有配置](./base.md)，以下配置说明主要
|
||||
|
||||
### kafka示例
|
||||
|
||||
::: tip 提示
|
||||
v3.3.0版本开始支持消费组，之前版本请参考对应版本配置文件
|
||||
:::
|
||||
|
||||
* broker_list，kafka节点列表
|
||||
* fail_msg_sender_timeout_ms，消息消费失败后重新投递至失败主题的超时时间，默认为1000ms
|
||||
* shard_repair，修补消息主题及消费分区指定（分区未指定默认消费所有分区），其中包含三类主题：
|
||||
* normal，普通主题，默认为shard_repair
|
||||
* failed，失败主题（正常消息消费失败后会将消息重新投递至该主题），默认为shard_repair_failed
|
||||
* priority，优先消费主题，主要存放一些需要高优先级消费的修补消息，默认为shard_repair_prior
|
||||
* blob_delete，删除消息主题及消费分区指定，其中包含两类主题
|
||||
* normal，普通主题，默认为blob_delete
|
||||
* failed，失败主题（正常消息消费失败后会将消息重新投递至该主题），默认为blob_delete_failed
|
||||
* topics，主题信息
|
||||
* shard_repair，修补主题，包含两个：普通修补主题与高优修补主题，默认为`shard_repair`与`shard_repair_prior`
|
||||
* shard_repair_failed，修补失败主题，默认为`shard_repair_failed`
|
||||
* blob_delete，删除主题，默认`blob_delete`
|
||||
* blob_delete_failed，删除失败主题，默认`blob_delete_failed`
|
||||
```json
|
||||
{
|
||||
"broker_list": ["127.0.0.1:9095","127.0.0.1:9095","127.0.0.1:9095"],
|
||||
"fail_msg_sender_timeout_ms": 1000,
|
||||
"shard_repair": {
|
||||
"normal": {
|
||||
"topic": "shard_repair",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"failed": {
|
||||
"topic": "shard_repair_failed",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"priority": {
|
||||
"topic": "shard_repair_prior",
|
||||
"partitions": [0,1]
|
||||
}
|
||||
},
|
||||
"blob_delete": {
|
||||
"normal": {
|
||||
"topic": "blob_delete",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"failed": {
|
||||
"topic": "blob_delete_failed",
|
||||
"partitions": [0,1]
|
||||
}
|
||||
"topics": {
|
||||
"shard_repair": [
|
||||
"shard_repair",
|
||||
"shard_repair_prior"
|
||||
],
|
||||
"shard_repair_failed": "shard_repair_failed",
|
||||
"blob_delete": "blob_delete",
|
||||
"blob_delete_failed": "blob_delete_failed"
|
||||
}
|
||||
}
|
||||
```
|
||||
@ -202,10 +189,8 @@ v3.3.0版本开始支持并发修复磁盘。
|
||||
}
|
||||
```
|
||||
### shard_repair示例
|
||||
|
||||
* task_pool_size，修补任务的并发度，默认10
|
||||
* normal_handle_batch_cnt，批量消费普通消息大小，默认100
|
||||
* fail_handle_batch_cnt，批量消费失败消息大小，默认100
|
||||
* fail_msg_consume_interval_ms，失败消息消费时间间隔，默认10000ms
|
||||
* orphan_shard_log，记录修补失败的孤本信息，dir需要配置，chunkbits为日志文件轮转大小，默认29（2^29字节）
|
||||
```json
|
||||
{
|
||||
@ -227,9 +212,6 @@ v3.3.0版本开始支持配置数据删除时间段。
|
||||
:::
|
||||
|
||||
* task_pool_size，修补任务的并发度，默认10
|
||||
* normal_handle_batch_cnt，批量消费普通消息大小，默认100
|
||||
* fail_handle_batch_cnt，批量消费失败消息大小，默认100
|
||||
* fail_msg_consume_interval_ms，失败消息消费时间间隔，默认10000ms
|
||||
* safe_delay_time_h，删除保护期，默认72h，如果配置负数则表示直接删除
|
||||
* delete_log，删除日志保留目录，需要配置，chunkbits默认为29
|
||||
* delete_hour_range，支持配置删除时间段，24小时制，比如以下配置表示凌晨1点到3点中间时间段才会发起删除请求，如果不配置默认全天删除
|
||||
|
||||
@ -72,42 +72,30 @@ The configuration of the Scheduler is based on the [public configuration](./base
|
||||
|
||||
### kafka
|
||||
|
||||
::: tip Note
|
||||
Starting from v3.3.0, consumer groups are supported. For previous versions, please refer to the corresponding configuration file.
|
||||
:::
|
||||
|
||||
* broker_list, Kafka node list
|
||||
* fail_msg_sender_timeout_ms, timeout for resending messages to the failed topic after message consumption fails, default is 1000ms
|
||||
* shard_repair, repair message topic and consumption partition specification (if the partition is not specified, all partitions will be consumed), which contains three types of topics:
|
||||
* normal, normal topic, default is shard_repair
|
||||
* failed, failed topic (after normal message consumption fails, the message will be resent to this topic), default is shard_repair_failed
|
||||
* priority, priority consumption topic, mainly used to store repair messages that need to be consumed with high priority, default is shard_repair_prior
|
||||
* blob_delete, deletion message topic and consumption partition specification, which contains two types of topics
|
||||
* normal, normal topic, default is blob_delete
|
||||
* failed, failed topic (after normal message consumption fails, the message will be resent to this topic), default is blob_delete_failed
|
||||
* topics，consume topics
|
||||
* shard_repair, normal topic, default are `shard_repair` and `shard_repair_prior`
|
||||
* shard_repair_failed, failed topic, default is `shard_repair_failed`
|
||||
* blob_delete, normal topic, default is `blob_delete`
|
||||
* blob_delete_failed, failed topic, default is `blob_delete_failed`
|
||||
|
||||
```json
|
||||
{
|
||||
"broker_list": ["127.0.0.1:9095","127.0.0.1:9095","127.0.0.1:9095"],
|
||||
"fail_msg_sender_timeout_ms": 1000,
|
||||
"shard_repair": {
|
||||
"normal": {
|
||||
"topic": "shard_repair",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"failed": {
|
||||
"topic": "shard_repair_failed",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"priority": {
|
||||
"topic": "shard_repair_prior",
|
||||
"partitions": [0,1]
|
||||
}
|
||||
},
|
||||
"blob_delete": {
|
||||
"normal": {
|
||||
"topic": "blob_delete",
|
||||
"partitions": [0,1]
|
||||
},
|
||||
"failed": {
|
||||
"topic": "blob_delete_failed",
|
||||
"partitions": [0,1]
|
||||
}
|
||||
"topics": {
|
||||
"shard_repair": [
|
||||
"shard_repair",
|
||||
"shard_repair_prior"
|
||||
],
|
||||
"shard_repair_failed": "shard_repair_failed",
|
||||
"blob_delete": "blob_delete",
|
||||
"blob_delete_failed": "blob_delete_failed"
|
||||
}
|
||||
}
|
||||
```
|
||||
@ -205,9 +193,6 @@ Starting from version v3.3.0, concurrent disk repair is supported.
|
||||
### shard_repair
|
||||
|
||||
* task_pool_size, concurrency of repair tasks, default is 10
|
||||
* normal_handle_batch_cnt, batch consumption size of normal messages, default is 100
|
||||
* fail_handle_batch_cnt, batch consumption size of failed messages, default is 100
|
||||
* fail_msg_consume_interval_ms, time interval for consuming failed messages, default is 10000ms
|
||||
* orphan_shard_log, record information of orphan data repair failures, directory needs to be configured, chunkbits is the log file rotation size, default is 29 (2^29 bytes)
|
||||
```json
|
||||
{
|
||||
@ -229,9 +214,6 @@ Starting from version v3.3.0, it is supported to configure the data deletion tim
|
||||
:::
|
||||
|
||||
* task_pool_size, concurrency of deletion tasks, default is 10
|
||||
* normal_handle_batch_cnt, batch consumption size of normal messages, default is 100
|
||||
* fail_handle_batch_cnt, batch consumption size of failed messages, default is 100
|
||||
* fail_msg_consume_interval_ms, time interval for consuming failed messages, default is 10000ms
|
||||
* safe_delay_time_h, deletion protection period, default is 72h. If a negative value is configured, the data will be deleted directly.
|
||||
* delete_log, directory for storing deletion logs, needs to be configured, chunkbits default is 29
|
||||
* delete_hour_range, supports configuring the deletion time period in 24-hour format. For example, the following configuration indicates that deletion requests will only be initiated during the time period between 1:00 a.m. and 3:00 a.m. If not configured, deletion will be performed all day.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user
