From a90af25ce787f3e242541399ef244d024a1dcc71 Mon Sep 17 00:00:00 2001 From: pengtianyue Date: Fri, 2 Jun 2023 17:43:45 +0800 Subject: [PATCH] docs(config): update docs for supporting consumer group Signed-off-by: pengtianyue --- .../configs/blobstore/scheduler.md | 54 +++++++------------ .../configs/blobstore/scheduler.md | 54 +++++++------------ 2 files changed, 36 insertions(+), 72 deletions(-) diff --git a/docs-zh/source/maintenance/configs/blobstore/scheduler.md b/docs-zh/source/maintenance/configs/blobstore/scheduler.md index 5fef02b38..3f0b9fb17 100644 --- a/docs-zh/source/maintenance/configs/blobstore/scheduler.md +++ b/docs-zh/source/maintenance/configs/blobstore/scheduler.md @@ -71,42 +71,29 @@ Scheduler的配置是基于[公有配置](./base.md)，以下配置说明主要 ### kafka示例 +::: tip 提示 +v3.3.0版本开始支持消费组，之前版本请参考对应版本配置文件 +::: + * broker_list，kafka节点列表 * fail_msg_sender_timeout_ms，消息消费失败后重新投递至失败主题的超时时间，默认为1000ms -* shard_repair，修补消息主题及消费分区指定（分区未指定默认消费所有分区），其中包含三类主题： - * normal，普通主题，默认为shard_repair - * failed，失败主题（正常消息消费失败后会将消息重新投递至该主题），默认为shard_repair_failed - * priority，优先消费主题，主要存放一些需要高优先级消费的修补消息，默认为shard_repair_prior -* blob_delete，删除消息主题及消费分区指定，其中包含两类主题 - * normal，普通主题，默认为blob_delete - * failed，失败主题（正常消息消费失败后会将消息重新投递至该主题），默认为blob_delete_failed +* topics，主题信息 + * shard_repair，修补主题，包含两个：普通修补主题与高优修补主题，默认为`shard_repair`与`shard_repair_prior` + * shard_repair_failed，修补失败主题，默认为`shard_repair_failed` + * blob_delete，删除主题，默认`blob_delete` + * blob_delete_failed，删除失败主题，默认`blob_delete_failed` ```json { "broker_list": ["127.0.0.1:9095","127.0.0.1:9095","127.0.0.1:9095"], "fail_msg_sender_timeout_ms": 1000, - "shard_repair": { - "normal": { - "topic": "shard_repair", - "partitions": [0,1] - }, - "failed": { - "topic": "shard_repair_failed", - "partitions": [0,1] - }, - "priority": { - "topic": "shard_repair_prior", - "partitions": [0,1] - } - }, - "blob_delete": { - "normal": { - "topic": "blob_delete", - "partitions": [0,1] - }, - "failed": { - "topic": "blob_delete_failed", - "partitions": [0,1] - } + "topics": { + "shard_repair": [ + "shard_repair", + "shard_repair_prior" + ], + "shard_repair_failed": "shard_repair_failed", + "blob_delete": "blob_delete", + "blob_delete_failed": "blob_delete_failed" } } ``` @@ -202,10 +189,8 @@ v3.3.0版本开始支持并发修复磁盘。 } ``` ### shard_repair示例 + * task_pool_size，修补任务的并发度，默认10 -* normal_handle_batch_cnt，批量消费普通消息大小，默认100 -* fail_handle_batch_cnt，批量消费失败消息大小，默认100 -* fail_msg_consume_interval_ms，失败消息消费时间间隔，默认10000ms * orphan_shard_log，记录修补失败的孤本信息，dir需要配置，chunkbits为日志文件轮转大小，默认29（2^29字节） ```json { @@ -227,9 +212,6 @@ v3.3.0版本开始支持配置数据删除时间段。 ::: * task_pool_size，修补任务的并发度，默认10 -* normal_handle_batch_cnt，批量消费普通消息大小，默认100 -* fail_handle_batch_cnt，批量消费失败消息大小，默认100 -* fail_msg_consume_interval_ms，失败消息消费时间间隔，默认10000ms * safe_delay_time_h，删除保护期，默认72h，如果配置负数则表示直接删除 * delete_log，删除日志保留目录，需要配置，chunkbits默认为29 * delete_hour_range，支持配置删除时间段，24小时制，比如以下配置表示凌晨1点到3点中间时间段才会发起删除请求，如果不配置默认全天删除 diff --git a/docs/source/maintenance/configs/blobstore/scheduler.md b/docs/source/maintenance/configs/blobstore/scheduler.md index 699f4b924..835d51d33 100644 --- a/docs/source/maintenance/configs/blobstore/scheduler.md +++ b/docs/source/maintenance/configs/blobstore/scheduler.md @@ -72,42 +72,30 @@ The configuration of the Scheduler is based on the [public configuration](./base ### kafka +::: tip Note +Starting from v3.3.0, consumer groups are supported. For previous versions, please refer to the corresponding configuration file. +::: + * broker_list, Kafka node list * fail_msg_sender_timeout_ms, timeout for resending messages to the failed topic after message consumption fails, default is 1000ms -* shard_repair, repair message topic and consumption partition specification (if the partition is not specified, all partitions will be consumed), which contains three types of topics: - * normal, normal topic, default is shard_repair - * failed, failed topic (after normal message consumption fails, the message will be resent to this topic), default is shard_repair_failed - * priority, priority consumption topic, mainly used to store repair messages that need to be consumed with high priority, default is shard_repair_prior -* blob_delete, deletion message topic and consumption partition specification, which contains two types of topics - * normal, normal topic, default is blob_delete - * failed, failed topic (after normal message consumption fails, the message will be resent to this topic), default is blob_delete_failed +* topics，consume topics + * shard_repair, normal topic, default are `shard_repair` and `shard_repair_prior` + * shard_repair_failed, failed topic, default is `shard_repair_failed` + * blob_delete, normal topic, default is `blob_delete` + * blob_delete_failed, failed topic, default is `blob_delete_failed` + ```json { "broker_list": ["127.0.0.1:9095","127.0.0.1:9095","127.0.0.1:9095"], "fail_msg_sender_timeout_ms": 1000, - "shard_repair": { - "normal": { - "topic": "shard_repair", - "partitions": [0,1] - }, - "failed": { - "topic": "shard_repair_failed", - "partitions": [0,1] - }, - "priority": { - "topic": "shard_repair_prior", - "partitions": [0,1] - } - }, - "blob_delete": { - "normal": { - "topic": "blob_delete", - "partitions": [0,1] - }, - "failed": { - "topic": "blob_delete_failed", - "partitions": [0,1] - } + "topics": { + "shard_repair": [ + "shard_repair", + "shard_repair_prior" + ], + "shard_repair_failed": "shard_repair_failed", + "blob_delete": "blob_delete", + "blob_delete_failed": "blob_delete_failed" } } ``` @@ -205,9 +193,6 @@ Starting from version v3.3.0, concurrent disk repair is supported. ### shard_repair * task_pool_size, concurrency of repair tasks, default is 10 -* normal_handle_batch_cnt, batch consumption size of normal messages, default is 100 -* fail_handle_batch_cnt, batch consumption size of failed messages, default is 100 -* fail_msg_consume_interval_ms, time interval for consuming failed messages, default is 10000ms * orphan_shard_log, record information of orphan data repair failures, directory needs to be configured, chunkbits is the log file rotation size, default is 29 (2^29 bytes) ```json { @@ -229,9 +214,6 @@ Starting from version v3.3.0, it is supported to configure the data deletion tim ::: * task_pool_size, concurrency of deletion tasks, default is 10 -* normal_handle_batch_cnt, batch consumption size of normal messages, default is 100 -* fail_handle_batch_cnt, batch consumption size of failed messages, default is 100 -* fail_msg_consume_interval_ms, time interval for consuming failed messages, default is 10000ms * safe_delay_time_h, deletion protection period, default is 72h. If a negative value is configured, the data will be deleted directly. * delete_log, directory for storing deletion logs, needs to be configured, chunkbits default is 29 * delete_hour_range, supports configuring the deletion time period in 24-hour format. For example, the following configuration indicates that deletion requests will only be initiated during the time period between 1:00 a.m. and 3:00 a.m. If not configured, deletion will be performed all day.