gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 02:00:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,190 Commits 48 Branches 52 Tags 378 MiB
master
Commit Graph

598 Commits

Author SHA1 Message Date
Wu Huocheng
22ca01df2f perf(datanode): reduce the extent info struct size.#1000005968 
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
 2025-08-08 14:59:26 +08:00
Wu Huocheng
30b29217a1 feat(datanode): evict the extent cache by ttl.#1000005968 
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
 2025-08-08 14:59:26 +08:00
Wu Huocheng
167e4c488a perf(datanode): replace make with bytespool.#1000005968 
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
 2025-08-08 14:59:26 +08:00
zhumingze
d7182d70bd feat(datameta): Record the read-only reason for dp&mp replicas. #1000060034 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:26 +08:00
Victor1319
6e05be7156 refactor(data): refactor the usage of the buffer pool during the prefetch process. #1000048465 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2025-08-08 14:59:26 +08:00
zhumingze
24cb92350d feat(data): Mark the disks that encounter disk errors during startup as bad disks. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:26 +08:00
zhumingze
bfe93e050c feat(datamaster): Supplement test cases related to lost and bad disks. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
zhumingze
1964c60240 feat(datamaster): Optimize the check disk lost logic. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
zhumingze
f1ad286a21 feat(data): Adjust some parameters and initialize. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
zhumingze
bf6d4f062f feat(masterdatacli): Add alerts for the completion of disk repair and improve cli display. 
. #1000004886

Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
zhumingze
e0a1ff8c0f feat(data): Regularly check dp availability. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
zhumingze
1c794069a1 feat(masterdatacli): lost disks detection and reporting. #1000004886 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-08 14:59:25 +08:00
clinx
6af47faed0 fix(datanode): change warn message log level #1000091570 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-08 14:59:24 +08:00
leonrayang
1ab568cfc1 fix(datanode): Reducing the sync process to enhance the speed of creating extent #23118077 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-08 14:59:24 +08:00
leonrayang
efeeea3dcd fix(datanode): Fix the enhance the persistence process of extentId #23118077 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-08 14:59:24 +08:00
clinx
a1fe50c7be fix(datanode): change write with return try other addr message to warn #1000082414 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-08 14:52:12 +08:00
clinx
818e4834b0 fix(datanode): opstreamfollowerread change timeout message to warn #1000082031 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-08 14:52:12 +08:00
leonrayang
5bd3da5846 feat(datanode): Enhance the persistence process of extentId #23118077 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-08 14:52:12 +08:00
clinx
3641507fbb fix(datanode): change the log level from error to warn for doStreamFixTinyDeleteRecord #1000068627 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-08 14:52:03 +08:00
chihe
78c727cda3 fix(datanode): adjust log level for no leader 
close:#1000001079

Signed-off-by: chihe <chihe@oppo.com>
 2025-08-08 14:52:02 +08:00
chihe
99a71d0cd7 fix(datanode): do not print log if diskBaseFileID equals to 0 
close:#1000012686

Signed-off-by: chihe <chihe@oppo.com>
 2025-08-08 14:51:45 +08:00
shuqiang-zheng
57fefaa661 feat(meta): perform security checks when configuring gogc, set upper and lower limits. 
close:#1000011090

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2025-08-08 14:42:25 +08:00
shuqiang-zheng
501dc61922 feat(meta): Support for dynamic adjustment of gogc. 
close:#1000011090

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2025-08-08 14:42:25 +08:00
chihe
ca90d3bac8 feat(master): add setRead/WriteLimits for flashnode 
close:#1000004942

Signed-off-by: chihe <chihe@oppo.com>
 2025-08-08 14:39:41 +08:00
chihe
380b8dde52 fix(flashnode): reply to client as soon as possible 
close:#23109728

Signed-off-by: chihe <chihe@oppo.com>
 2025-08-08 14:39:41 +08:00
Victor1319
4ed781a41a fix(data): fix gc clear data failed bug. #22957485 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2025-08-07 16:00:30 +08:00
chihe
be79605ea0 fix(flashnode): read data source one by one 
close:#23095482

Signed-off-by: chihe <chihe@oppo.com>
 2025-08-07 16:00:30 +08:00
leonrayang
8d897d1289 feat(datanode): Enable crc check for BaseExtentID information #23118077 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:29 +08:00
leonrayang
549128cf1a fix(datanode): Stop the tiny extent markdelete process while store stopped #23115007 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:29 +08:00
leonrayang
8d3fe7499c fix(datanode): Temporarily disable the crc read due to the wrong extentid in meta EXTENT_META 
close: #23118077

Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:29 +08:00
zhumingze
175e6f4ee7 fix(datamaster): Adjust some error logs. #23101965 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-07 16:00:29 +08:00
zhumingze
3c91dcbf84 fix(data): Add dp information to cache in advance to avoid start raft error. #23113653 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-07 16:00:28 +08:00
leonrayang
72bf18cdff fix(data): Temporily modifying the test case has a high ratio of failure out #23059813 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:26 +08:00
leonrayang
2e8bf80543 fix(testcase): comment the test case has high ratio of failure out #23059813 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:26 +08:00
leonrayang
88ca620bf8 fix(datanode): Optimize io limiter exit process. #23059813 
Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:26 +08:00
Wu Huocheng
45afc5aee0 fix(raft): use raftfsm.replicas instead of get peers.@formatter:off 
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
 2025-08-07 16:00:26 +08:00
Wu Huocheng
ed61daa568 feat(raft): display raft peers.@formatter:off 
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
 2025-08-07 16:00:26 +08:00
leonrayang
30b221b2af feat(datanode): Return error if request is hung by io limiter and exceeds maximum wait time 
. #23057829

Signed-off-by: leonrayang <chl696@sina.com>
 2025-08-07 16:00:26 +08:00
shuqiang-zheng
75d6aca05e fix(datanode): fixes for 1.read/write requests still being processed 
2.log files not being found after service stopped.

close: #23013195

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2025-08-07 16:00:25 +08:00
clinx
56fc4fa9b1 fix(data): outofslice error #22883846 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-07 15:57:57 +08:00
clinx
2b0f857d01 feat(data): compatible json serialization for old datanode. #22883846 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-07 15:57:56 +08:00
clinx
0eb05bd9f7 feat(data): compatible json serialization for old datanode. #22883846 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-07 15:57:56 +08:00
clinx
e3ae2ea3f1 feat(data): change extentinfo serialize for getting remote extend info#22883846 
Signed-off-by: clinx <chenlin1@oppo.com>
 2025-08-07 15:57:56 +08:00
zhumingze
1a20920be2 feat(datameta): Modify gcRecyclePercent to be configurable. #22995150 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2025-08-07 15:57:56 +08:00
slasher
131b332136 chore(ci): split run test for cubefs and blobstore 
@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
 2025-07-09 10:05:10 +08:00
yanbin027
f25c5b9cc7 feat(client): "client support aheadread" 
Signed-off-by: yanbin027 <yanbin027@ke.com>
 2025-03-20 14:18:21 +08:00
morphes1995
ab398d669d fix(master): RaftPartitionCanUsingDifferentPort make effective after and all dn/mn upgraded 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-12-31 17:36:32 +08:00
morphes1995
9a334cd7e8 feat(master): add param to control whether raft partition can use different port 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-12-31 17:36:32 +08:00
morphes1995
00a3ee193f fix(master): fix port inconsistent when create dp/mp during upgrading process 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-12-31 17:36:32 +08:00
morphes1995
f4c7755316 feat(metanode): support multi metanode process using different port on single node 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-12-31 17:36:32 +08:00
morphes1995
b23df49eca feat(datanode): support multi datanode process using different port on single node 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-12-31 17:36:32 +08:00
Victor1319
8bb1131500 refactor(all): refactor go code with golint. #22906448 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
0c406f55d9 refactor(master): resolve conflicts when picking to master branch. #22906448 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
dac1a12d0a refactor(master): resolve conflicts when picking to master branch. #22906448 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
leonrayang
3396accd80 feat(projectstructure): reorganize the module directory in the root to a suitable subdirectory 
close:#22458839

Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
chihe
4c62f60ef1 fix(datanode): trigger disk error when read data from tiny extent 
close:#22771554

Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
zhumingze
911db95789 fix(datacmd): modify the alarm level of logLeftSpaceLimitRatio parameter parsing errors #22904699 
Signed-off-by: zhumingze <zhumingze@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
f9c2f49204 fix(meta): don't block extent delete req when no success exts. #22852635 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
ced8ca0978 feat(masterdata): add config to control whether vol read direct disk. #22818122 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
ab8217379e refactor(datanode): refactor data read and write performance. #22504692 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
fba5a5d6f3 refactor(master): check vol forbidden write type when add vol storage class. #22814514 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
Reliey
96f4b0101b feat(datanode): add endpoint for setting enablePid in exporter and handle response building #22257509 
Signed-off-by: Reliey <zhumingze@oppo.com>
 2024-12-26 09:24:22 +08:00
Reliey
5f3c6e2d4a feat(cli): support reporting and viewing of cluster and vol dimension disk data #22257509 
Signed-off-by: Reliey <zhumingze@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
abc2ab5912 refactor(master): support double check master media cfg type. #22727537 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
2bced8d337 refactor(datameta): hybrid cloud optimize some logs related to upgrade compatibility 
close:#22716686
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
168fa0418c feat(mastermeta): add LegacyDataMediaType in master cluster values, and metaNode fetch it from master. 
close:#22716686
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
716aa7970a feat(masterdata): datanode quit starting if master returns mediaType not match when register 
close:#22727852
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
53d939fad6 fix(datameta): use default value if master not support the API AdminGetVolListForbidWriteOpOfProtoVer0. 
close:#22644059
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
63d0b94f57 feat(master): add an API to get volumes those set forbidden write op codes of protocaol version-0 
close#22531837
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
9f7ffeb194 feat(masterdatameta): support volume to forbidden write operate codes of lower packet protocol version 
close#22531837

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
a43f96d403 feat(dataclientmaster): support configuring datanode and metanode to forbidden 
write operate codes of lower packet protocol version.
       	 close#22531837
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
037df5f5f9 fix(sdkclient): client compatible with older version server modules 
#22400600
	@formatter:off
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
NaturalSelect
3b9ef65dbe chore(all): format code 
@formatter:off

Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
14d7e0e19c feat(masterclientcli): hybrid cloud add property mediaType to datapartition 
(1)master: when creating vol, master creates dps of each mediaType according to vol.allowedStorageClass
	(2)client: when select dp to do append write, choose dp with spcific mediaType according to inode's storageClass

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
3081a30168 feat(master): hybrid cloud add mediaType property to zone 
(1) zone mediaType is set as the first add-in datanode's mediaType
	(2) once zone mediaType is set, all datanodes in the zone must be the same mediaType.
	(3) the cmd 'cfs-cli zone info' shows zone mediaType.

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
f70b3cfe38 feat(masterdatacli): hybrid cloud add datanode's mediaType 
for module master and datanode; the cmd 'cfs-cli datanode info' can show datanode's mediaType

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
d3912d2042 fix(datanode): check initPartitionSize when calculating progress 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
852dafafd2 fix(datanode): if dpBackupTimeout is not set, then do not delete back up directories. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
leonrayang
111f9e9903 feat(dataemeta): cluster switch on snapshot take effect on module handle protocal process 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
chihe
cd73e762e4 fix(datanode): iterating through the cache of dps and disks when stopping space manager 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
cd5376cc20 fix(datanode): launch sechedule for raft log before staring raft when loading dp 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
1442c33a29 fix(master): if the replica being decommissioned is the one intended for decommissioning, no error is reported. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
1888c9b8a8 fix(raft): if the previous member change did not return, reject the submission of the next member change 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
448ab8e54f test(data): add testcase for datanode op log when disk is full. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
2ea78a48f1 fix(datanode): if reading applyid fails when loading dp, then delete from disk cache 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
ab6d34d481 fix(datanode): use lock for getPartitionsAPI 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
c52f8b35eb fix(master): release token before resetting decommission dst 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
ac54435341 feat(datanode): increasing retry interval when DataNode connects to Master during startup. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
2645319183 fix(data): donot persist applyId by raftForce 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
96dd26f01a fix(master): the dp's zone selection logic supports decommission operations. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
b8742d469a fix(master): use lock to set dp markDecommission 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
leonrayang
78bbb209f2 feat(datanode): enhance the preformace of the stat for disk opertion 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
W9068822
6e51516a8c feat(stat): datanode disk and dp operation log 
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
eb5787a7b3 fix(master): traverse all nodesets to check if dp is decommissioning 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
64afe661cb fix(master): do not check size of datanode when retrying to decommission special replica 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
ef8183b648 fix(cli): remove unnecessary commands for cli 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
c3131b01f9 fix(datanode): use dp leader real size as to be repaired size. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
323133a9f9 fix(datanode): return error if receive OpDeleteDataPartition when disks are not loaded 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
53107ee952 fix(datanode): modifying the regular expression for backup directory: 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
61e0ab3aa6 fix(master): fix deadlock for getDataPartitionsView 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00