gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 02:00:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,258 Commits 48 Branches 52 Tags 378 MiB
blobstore-develop
Commit Graph

429 Commits

Author SHA1 Message Date
true1064
af298477f3 fix(master): check if volume's zoneName list has the resource when add allowed storageClass 
#22395995
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
e2e9943f47 feat(cli): while creating volume, auto set crossZone as true if assigned more than one zone. 
#22376966

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
be1e6ba732 feat(master): hybrid cloud support assigning mediaType when manually create datapartiton. 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
NaturalSelect
7a63703de6 feat(master): hybrid cloud support query cluster storage class info 
close: #22310366
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-12-26 09:24:22 +08:00
NaturalSelect
f029d45f6a feat(cli): hybrid cloud support query vol hybrid storage info 
close: #22304812 #22304800
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
0ec2d1c2b6 fix(cli): hybrid cloud show if the node is set as read-only when get node information 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
0173aa7e48 fix(master): hybrid cloud fix test cases of creating datapartitions 
#22196488
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
3561f231ba fix(cli): fix the display of the 'metapartition check' command of the cfs-cli tool 
clsoe:#22152841

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
309d89f1a3 fix(meta): hybrid cloud check err when invoking (*Inode).UnmarshalInodeValue 
close:#22112817
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
698311248a fix(master): hybrid cloud set ebsblockSize when add blobStore to volume's allowedStorageClass 
Signed-off-by: true1064 <true1063@163.com>
 2024-12-26 09:24:22 +08:00
true1064
4173cc9f4d fix(master): hybrid cloud When updating a volume, ebsBlockSize can be modified if allowedStorageClass contains blobstore. 
Signed-off-by: true1064 <true1063@163.com>
 2024-12-26 09:24:22 +08:00
true1064
3de7e847a9 feat(master): handle compatibility of master when older version upgrade to hybrid cloud: 
(1) auto set mediaType of datanode, zone, datapartiton by config legacy "legacyDataMediaType" and persist;
	(2) auto set storageClass of volume by config legacy "legacyDataMediaType" and persist;
	(3) if need to rollback master to older version, need to do nothing, just replace to old master and restart. because hybrid cloud master only add new fields to meta by not changed old fields.

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
35ed550795 feat(master): hybrid cloud forbidden AdminCreateVersion API because hybrid cloud not support snapshot version yet 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
9d2bf3b483 feat(mastercli): add API to add storageClass to volume's allowedStorageClass list 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
5936cdfe8d fix(master): amend some logs 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
443199e0b0 fix(client): hybrid cloud only clear dp when volume storage class is blobstore 
Signed-off-by: chihe <chi.he@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
f04993070b fix(master): some fix about datapartition creation 
(1) while creating dp in vol creation, check dp count by mediaType to judge if success.
	(2) when background periodical check if need to create dp, skip volumes whose createTime is too recently, to avoid background create dp while volume is creating.
	(3) display mediaType of dp in cfs-cli cmd "volume info -d"

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
51554785e3 fix(master): hybrid cloud not set zoneName as default value if not assigned in the create vol request. 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
7a7b663b6c feat(mastercli): hybrid cloud vol.volStorageClass can be changed by "cfs-cli volume update" 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
1f9dab0eda feat(masterclientpreload): add property CacheDpStorageClass to volume. 
CacheDpStorageClass is used to inform SDK which storageClass to use when access cache dp.

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
14d7e0e19c feat(masterclientcli): hybrid cloud add property mediaType to datapartition 
(1)master: when creating vol, master creates dps of each mediaType according to vol.allowedStorageClass
	(2)client: when select dp to do append write, choose dp with spcific mediaType according to inode's storageClass

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
6cafecdb7d feat(masterclientcli): add property volStorageClass and allowedStorageClass to volume 
now volType is not uesed while creating volume, but is maintained for compability

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
3081a30168 feat(master): hybrid cloud add mediaType property to zone 
(1) zone mediaType is set as the first add-in datanode's mediaType
	(2) once zone mediaType is set, all datanodes in the zone must be the same mediaType.
	(3) the cmd 'cfs-cli zone info' shows zone mediaType.

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
true1064
f70b3cfe38 feat(masterdatacli): hybrid cloud add datanode's mediaType 
for module master and datanode; the cmd 'cfs-cli datanode info' can show datanode's mediaType

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2024-12-26 09:24:22 +08:00
leonrayang
0647717f79 fix(client): enable Client with version 3.4.0 compatible with 3.3.2 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
chihe
852dafafd2 fix(datanode): if dpBackupTimeout is not set, then do not delete back up directories. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
a3fb3d301b fix(master): if the decommission fails and requires retry, set the status to 'mark' only at the end of the decommission function 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
64afe661cb fix(master): do not check size of datanode when retrying to decommission special replica 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
ef8183b648 fix(cli): remove unnecessary commands for cli 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
323133a9f9 fix(datanode): return error if receive OpDeleteDataPartition when disks are not loaded 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
da33bc2068 feat(cli): support raftForce for datanode decommission by cfs-cli 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
46157d2c06 refactor(all): resolve conflict when pick commit from 3.3.2 to 3.4.0. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
69ca9cd0f4 fix(metanode): only sync at among replicas when inode is not nil 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
3a4a54f3bc fix(cli): use macro value instead of hard code 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
Victor1319
b5252646d0 refactor(cli): refactor cli hint message when update transaction info. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
a305b32b9c feat(master): cli support setting for persistting accessTime 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
6970b4570d feat(master): support set trashInterval with cli 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
a487a828b4 feat(master): api for enable/disbale persist for accesstime 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
0f64f5010e feat(master): api for setting accesstime vaild interval by vol 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
shuqiang-zheng
aefb1b2bd9 refactor(cli): update instructions for freezing clusters 
@formatter:off
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
f70f1ee08c fix(master): fix dead lock by getting copy of dp map 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
2fb56e7282 feat(master): auto delete backup directories 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
leonrayang
d6e066f899 fix(util): improve the code style according to warnings from code check 2024-12-26 09:24:22 +08:00
NaturalSelect
5365949f7c feat(master): support config decommission settings 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-12-26 09:24:22 +08:00
chihe
04ac20e2cb fix(master): calculating the decommission progress of the disk includes ignoring dps. 
Signed-off-by: chihe <chihe@oppo.com>
 2024-12-26 09:24:22 +08:00
shuqiang-zheng
f8c677a041 fix(security): slice memory allocation with excessive size value. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-08-09 19:04:31 +08:00
chihe
47a7a6dfa4 feat(master):recover replica deleted by raftForce 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-22 17:10:59 +08:00
chihe
e9804d0043 feature(datanode): do not remove root dir for dp deleted by raftForce with auto decommmission mode 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
bc5dd77fbd feat(master): support set dp heartbeat timeout 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
6aca325797 feat(master): feat support vol level dp meta repair 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
414229e0b5 fix(master): update dp cache after create vol 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
chihe
e287f3a9ce fix(master): redundant replica not participate when decommissioning mutiple replicas of dp 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
48506bbd18 fix(data): avaliable space calculate consider decommissioned disks 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
7a1c60eeea feat(master): support enable or disable auto dp meta repair 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
d68b68a2a2 fix(master): support config dp repiar timeout 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
cf772279a7 feat(master): nodeset info return can alloc metanode/datanode cnt 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
W9068822
24618be53c fix(cli): fix DataNodeMigrate, @formatter:off 
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
 2024-07-18 16:57:38 +08:00
W9068822
bf336dc34d feat(master): add getAllDataNodes and getAllMetaNodes apis 
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
c19b690d1d feat(master): support reset dp restore status 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
4acdeca1fd feat(master): return can alloc partition when query node info 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
4df91ebc6e refactor(all): reslove merge conflicts between 3.3.x and 3.4.0. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
leonrayang
a9b3d54028 feat(master): Volume update enable crossZone 
Signed-off-by: leonrayang <changliang@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
ec188331f0 feat(cli): support set trash interval 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
53b269d7e5 fix(master): update decommission limit when load nodeset 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
d95f005df9 fix(master): Avoid concurrent access exceptions when responding to HTTP requests with a map. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
c7aba207cc refactor(cli): Optimize the print output results of fileInCore. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
c976358bbe refactor(cli): remove pasue flag in transaction flag hint. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
Victor1319
ee9aba1f3a fix(master&client&cli): Add a new variable EnableTransactionV1 to avoid compatibility issues 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
7c08fedcd8 feat(master): add forbid vol feature 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
625a7a3d62 feat(master): enable audit log by default 
NOTE: Please be careful when cherry-picking this commit.

Signed-off-by: NaturalSelect <2145973003@qq.com>
 2024-07-18 16:57:38 +08:00
shuqiang-zheng
68cf97ab68 fix(master):fix the problem of TestCreateColdVol 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
3b2a91bdf6 fix(cli): fix ui error occured by cherry-pick 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
d201f99c04 feat(master): support set decommission disk limit 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
shuqiang-zheng
eed62c913f enhance(log): Modify the threshold triggering log to logLeftSpaceLimitRatio of the total disk space 
@formatter:off

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-07-18 16:57:38 +08:00
chihe
7a6febec4b feat(cli): add decommissioned disks info for datanode node info display 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-05 17:05:43 +08:00
chihe
65ee50a40a fix(cli): cli can display dps encountering IO errors during loading 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
fdec047ee4 feat(master): support abort disk decommission 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
0a158c3e96 feat(master): support query decommission failed disks 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
3b03341425 feat(master): support max mp cnt limit 
@formatter:off

Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
14ce5dc2cd test(master): fix unit tests 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-05 17:05:43 +08:00
W9068822
470967aae4 feat(ci): fix format failed. 
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
 2024-07-05 17:05:43 +08:00
chihe
f704f3fdac feat(cli): add report for disk error dp replica for cli 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
96ed7fd818 fix(master): persist decommission error message 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
NaturalSelect
7329d1f1e8 feat(master): check datapartition status when set dp discard 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
NaturalSelect
9ca77ba7aa feat(master): if dp is discard, decommission will success directly 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
NaturalSelect
18e83cd45d fix(cli): foramt decommission dp info 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
NaturalSelect
610e6bd457 feat(master): support auto decommission disk and config 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
NaturalSelect
38bd1924e0 fix(cli): datapartition query-progress 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
S9054862
8ad316ecfc feat(data): repair block size support master config 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-01 15:09:55 +08:00
leonrayang
4ee2208263 feat(master): Optimize logic of uid calculate for better performance 
1.use channel to instead of lock and isolate with main heartbeat routine
2.calculate in an aysnchoronus way periodically instead of real time
3.move heartbeat outside the goroutine, the startup of goroutine may delay the response time

Signed-off-by: leonrayang <changliang@oppo.com>
 2024-07-01 15:09:55 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools 
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
 2024-05-14 09:54:13 +08:00
morphes1995
b8c26ba20f feat(cli): cfs-cli add functionality to show disk info 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-04-19 14:06:35 +08:00
slasher
bdffd66650 feat(cli): auto format struct and table 
with #21965811

Signed-off-by: slasher <shenjie1@oppo.com>
 2024-03-29 14:22:29 +08:00
shuqiang-zheng
e59f07a084 feat(reconstruct): gofmt by gofumpt. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
af739c6a3f fix(master): Fix the problem of updating a cold volume when cachecap and replicaNum is all 0. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
42fa6628f4 enhance(cli): Modifying Volume Deletion Related Error Messages. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
6d0de0df25 fix(master): Fixed the problem that delayed deleted volumes do not support migration. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
452d2a3239 enhance(cli): Dynamically configure volDeletionDelayTime through the interface. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
325694a4f7 enhance(cli): Add query interface to show the current delayed deletion time of the volume. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
shuqiang-zheng
9c5185e9be feat(master): Freeze the volume when performing volume deletion. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
NaturalSelect
df58ccb9b4 feat(master): show error message when decommission dp fails 
close: #21952214

Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-03-27 14:24:50 +08:00
morphes1995
09ebe39b80 feat(metanode): Support create multiple meta partitions at one time 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-02-29 16:43:50 +08:00
NaturalSelect
24cbd9c1ab fix(meta): GetAllExtsOfflineInode failed to delete extent keys 
close: #3140

Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-02-29 14:33:40 +08:00
NaturalSelect
8bd2aac465 feat(cli): support discard dp 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-02-28 16:31:57 +08:00
lily-lee
84dea94603 feat(master): compress client/partitions response data 
Signed-off-by: lily-lee <lilylee88756@gmail.com>
 2024-02-27 10:30:21 +08:00
slasher
5d5e45cb0a chore(util): store current time to atomic value 
BenchmarkGetCurrentTime-4    100000000    11.60 ns/op            0 B/op          0 allocs/op
-->
BenchmarkGetCurrentTime-4    895091701     1.31 ns/op            0 B/op          0 allocs/op

Signed-off-by: slasher <mcq.sejust@gmail.com>
 2024-01-31 19:13:25 +08:00
morphes1995
38fa309da4 [fix]: use original NodeStatInfo field type to adapt old services 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-01-26 18:50:15 +08:00
leonrayang
8ee2c10102 Revert "[Enhancement]: Support reporting the total size of writable space in the cluster." 
This reverts commit 37bdadfc77.

Signed-off-by: leonrayang <chl696@sina.com>
 2024-01-24 16:07:56 +08:00
morphes1995
37bdadfc77 [Enhancement]: Support reporting the total size of writable space in the cluster. 
Signed-off-by: morphes1995 <morphes1995@gmail.com>
 2024-01-23 11:22:45 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt 
Signed-off-by: slasher <shenjie1@oppo.com>
 2024-01-12 14:21:16 +08:00
lily-lee
f45b4cd606 feat(dp): set dpCount when create volume 
Signed-off-by: lily-lee <lilylee88756@gmail.com>
 2024-01-08 16:51:38 +08:00
NaturalSelect
25af0b0935 feat(cli): support query decommission disk progress in cli 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2024-01-05 14:00:29 +08:00
slasher
9b1f6270ae chore(cli): catch errors on root of cmd 
Signed-off-by: slasher <shenjie1@oppo.com>
 2024-01-04 17:37:31 +08:00
slasher
98a9f5b07f chore(proto): parse http response in proto.HTTPReplyRaw 
Signed-off-by: slasher <shenjie1@oppo.com>
 2024-01-04 17:37:31 +08:00
NaturalSelect
f6e774b7cc fix(cli): fix typo 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-12-28 18:34:45 +08:00
NaturalSelect
5aeb164b58 feat(cli): cli support to recommission disk 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-12-15 15:04:55 +08:00
NaturalSelect
0291ff91fc feat(cli): cli support to decommission disk 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-12-15 15:04:55 +08:00
NaturalSelect
1003212650 feat(metanode): audit log support master control 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-11-09 15:10:23 +08:00
true1064
5af7084e0e refactor(data): optimize the determination of datanode disk's unavailable status: 
(1) add 2 conf item: diskUnavailableErrorCount and diskUnavailablePartitionErrorCount.
(2) when encounter disk error, set disk as unavailable if disk error count reaches diskUnavailableErrorCount or partition count with disk error reaches diskUnavailablePartitionErrorCount.
(3) if a partition encounter 1 disk error, set the partition's status as unavailable.

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-11-01 11:47:58 +08:00
chihe
ff2d5f6bd5 bugfix(datanode,cli):1.remove logic of fetching replica from master after adding raft member 
2.cli show error msg when decommission failed 3.keep disk original decommission status when load
from rocksdb

Signed-off-by: chihe <chi.he@oppo.com>
(cherry picked from commit 846ac29e57e838cb728c1cd9495d4ecf0bd4c839)
 2023-11-01 11:33:44 +08:00
chihe
b5de4b97b2 fix(master): panic when concurrent read/write decommissiondatapartition list 
Signed-off-by: chihe <chi.he@oppo.com>
(cherry picked from commit 3f20228d8edd8be9809d69e3a0cae7a75bea8703)
 2023-10-31 14:22:38 +08:00
slasher
74be4e4970 chore(cli): cli output with format 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-10-23 15:31:07 +08:00
slasher
7f406e5d0e chore(cli): golangci lint apply into cli 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-10-23 15:31:07 +08:00
slasher
ff476f4d7d chore(cli): gofumpt apply into cli 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-10-23 15:31:07 +08:00
shuqiang-zheng
06d542cc98 fix(cli): fix the bug of empty unavailable replica may appear when using cli command with metapartition check. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2023-10-23 14:27:38 +08:00
NaturalSelect
fdfa176a97 fix(cli): fix fmt bug 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-10-16 15:57:54 +08:00
true1064
a5dd54d60f feat(master/datanode/cli): 
(1)datanode bad disk report to master adds: TotalPartititionCnt and DiskErrPartitionList
(2)cfs-cli display bad disk adds TotalPartititionCnt and DiskErrPartitionList accordingly
(3)datanode's data partition add disk err count record and log print

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-10-12 14:52:52 +08:00
xuruida
7a6279d3b8 enhance(cli/master): rename volume create option "size" to "dp-size" (cli) / "dpSize" (api) 
Signed-off-by: xuruida <xuruida@oppo.com>
 2023-10-07 15:56:35 +08:00
baijiaruo
3a42e65af4 feat(sdk): Solve warnings that do not comply with go language rules 
Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-09-06 10:09:43 +08:00
NaturalSelect
c3ed9bafb5 feat(master): add forbid vol feature 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-09-01 16:23:21 +08:00
true1064
08c4672e95 enhance(master/cli): Reduce too much dp info printing about replica file count and size differ: 
(1)master: not record replica file count and size differ if dp is in decommission
(2)cli: only print number of dp with replica file count and size differ in default; and add an argument to control whether print such dp info

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-08-30 20:05:32 +08:00
leonrayang
86d0e5b8a9 enhance(master): snapshot.update ver from seconde to microSecond for future develop multi snapshot create one second 
Closes: #2499

Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
NaturalSelect
a93dce93e4 feat(master,cli): support to query and update node selector of nodeset 
close: #1921

Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-08-30 20:05:32 +08:00
NaturalSelect
4913de7e2e feat(master,cli): support cluster level node selector query and update 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-08-30 20:05:32 +08:00
NaturalSelect
b398946a82 feat(master,cli): support zone level node selector query and update 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-08-30 20:05:32 +08:00
NaturalSelect
4f058bd163 feat(master,cli): support query and update nodeset selector of zone 
Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-08-30 20:05:32 +08:00
setcy
c43dcb4962 feat(cli): support listing nodeSets and displaying space capacity with cli 
Signed-off-by: setcy <2670061551@qq.com>
 2023-08-30 20:05:32 +08:00
chihe
3638432b81 [bugfix]1.rollback will release token 2.new replica created by decommission status is recovering 
3.fix pause then resume progress error

Signed-off-by: chihe <chi.he@oppo.com>
 2023-08-30 20:05:32 +08:00
Patrick Wu
7e72ede897 feature: 1. support setting dp decommission control info. 2. add dp decommission progress for dp check cmd 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-08-30 20:05:32 +08:00
chihe
952142617e [bugfix]1.fix some bugs about decommission enhance 
2.cli add recovery status
    3.add decommission disk type for decommission disk api
    4.load decommission disk to add to nodeset decommission disk list

Signed-off-by: chihe <chi.he@oppo.com>
 2023-08-30 20:05:32 +08:00
leonrayang
88c2c5d7ff update: snapshot.Reduce extent count triggerd by ver inconsistent 
optimize code on function naming and abstract public function

Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
leonrayang
f7e324a98f update. enable cli support version strategy 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
leonrayang
5ae500768b feature: support multi version snapshot 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
NaturalSelect
e1a69d5314 feat(cli): support query cpu util of datanode and metanode and disk util of datanode 
close: #1922

Signed-off-by: NaturalSelect <2145973003@qq.com>
 2023-08-30 20:05:32 +08:00
M1eyu2018
65ec21d1a6 [Enhancement] add authentication for master admin api 
Signed-off-by: M1eyu2018 <857037797@qq.com>
 2023-08-24 16:05:12 +08:00
shuqiang-zheng
a0651b75d9 feat(Inode Lock Time): Changed the default value of optDeleteLockTime when updating volume information, and solved the problem of deletion lock failure after the client's system time has been changed. 
close: #2341

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2023-08-10 20:50:06 +08:00
shuqiang-zheng
1349490959 feat(Inode Lock Time): Added support for delete-lock-time to Inode in transactional situations, and provided a zero setting to disable file locking. 
close: #2341

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2023-08-09 14:01:52 +08:00
shuqiang-zheng
346c604692 fix(log): fix bug that logs were not cleaned in time. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2023-08-09 14:01:52 +08:00
shuqiang-zheng
680dc0158c feat(Inode Lock Time): Added support for delete-lock-time to Inode. 
close: #2341

Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2023-08-09 14:01:52 +08:00
baijiaruo
e1748ee989 fix(cli): update volume operation cause quota switch to be off 
close: #2310
Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-08-09 14:01:52 +08:00
Victor1319
b087888438 refactor(master & client): no tx operation should also check txConflict error. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2023-08-09 14:01:52 +08:00
leonrayang
7c23a8fbc1 fix(cli):Fix code scanning alert - Incorrect conversion between integer types 
close: #2432

Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-08 09:43:59 +08:00
Victor1319
2199fd061c fix(master): fix the bug found in security code-scanning. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2023-08-07 14:56:20 +08:00
setcy
d423d2d640 feat(master): partitions support displaying in which nodeSet 
Signed-off-by: setcy <2670061551@qq.com>
 2023-08-07 11:03:59 +08:00
baijiaruo
10dfcfd791 fix(metanode): fix quota capacity statistics are inaccurate 
close: #2271

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
80a6e4c4d0 fix(metanode): fix metanode takes up too much memory 
close: #2249

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
db013e2a90 feat(master/metanode/cli): Add vol quota switch 
close: #2244

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
7c274013c8 fix(cli,master): 1.Directory nesting issues with different quotas 
2.root directory not checked out for nesting

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
a8b61ad172 fix(master/metanode/cli): Fix the following bugs 
1.Subdirectories under the quota directory cannot be moved
2.After creating the quota, delete the quota root directory and create it again, but the apply still succeeds
3.Multi-level directory setting quota report does not has only one /
4.2000w directory file after the quota is created apply error

close: #2189

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
leonrayang
959a00b1b6 feat(transaction): support transaction limitition in metaPartition dimension and realize in a simplify way temporily 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
78e997b4c0 fix(cli): 1.No error reported when creating quota using multiple directories with the same name 
2.Deleting quota without revoking it first will cause directories to be undeletable
close: #2171

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-07-20 10:58:50 +08:00
true1064
f323548268 feat(master): add cluster frobid meta partition decommission switch 
close: #2139
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-07-20 10:58:50 +08:00
baijiaruo
53b12d9b4c feat(master/metanode/cli): Enhance cfs-cli apply and revoke operations can swim to the root directory 
close: #2144

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-06-16 17:42:02 +08:00
Patrick Wu
e622d34f35 fix(cli,sdk): 1.fix tx display bug in cli. 2.fix tx request retry bug 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-06-16 17:42:02 +08:00
Patrick Wu
68df3ea00d fix(cli,master): set default transaction value for old volume from previous version when loading from rocksdb 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-06-16 17:42:02 +08:00
baijiaruo
d97d066f5c fix(master,cli): 1.create quota dir without / without error. 
2.Create quota with the same directory error message optimization
    3.A single quota increases the directory limit to 5
    4.Creating a quota, the scene of multiple directories shows that the number of used files is incorrect
    5.list quota Sort by quotaId from small to large
close: #2107

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-06-16 17:42:02 +08:00
Patrick Wu
d88585dac8 fix(cli,client,master,sdk): 1.add ability to force reset transaction mask. 2. sdk will retry transaction operation when conflict occurs, retry times and interval are configurable 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-06-16 17:42:02 +08:00
chihe
82b9b1fd80 enhance(cli,client):1. add check inode max mp check 2.client return err 
if user is not correct immediately

Signed-off-by: chihe <chihe@oppo.com>
 2023-06-16 17:42:02 +08:00
Patrick Wu
d309886dc3 fix(metanode, client, cli): 1. set SyncSendTaskDeadlineTime to 30 seconds, 2. fix tx timeout unit display in cli 3.tidy up, remove todos and comments 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-06-16 17:42:02 +08:00
baijiaruo
a2f7ac1f53 enhance(master/cli): Added multi-directory federation quota 
close: #2087

Signed-off-by: baijiaruo <baijiaruo@126.com>
 2023-06-16 17:42:02 +08:00
baijiaruo
b87a3a41e7 feat(master/metanode/cli): The directory transfer of quotaId has been changed from metanode to cli operation and rename is only allowed within a quota or outside the quota 
Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-06-16 17:42:02 +08:00
true1064
d7e6dd2f2e feat(master,cli): enhance discard data partition report: 
1. add cmd "cfs-cli datapartition get-discard" to display all discard data partitions;
2. missing dp metrics not display discard data partitions;

Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-06-16 17:42:02 +08:00
true1064
014629c40f feat(cli):add an argument "--ignoreDiscard" to cmd "cfs-cli datapartition check", the datapartitions which is marked discard will not be displayed if this arg is set 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-06-16 17:42:02 +08:00
setcy
d991494ab8 feat(cli): give suggestions when root command not found 
Signed-off-by: setcy <2670061551@qq.com>
 2023-06-09 09:08:32 +08:00
setcy
b75a644040 feat(cli): add newline in errouts 
Signed-off-by: setcy <2670061551@qq.com>
 2023-06-09 09:08:32 +08:00
setcy
8e60a92829 docs(cil): improve documentation of cli tool 
Signed-off-by: setcy <2670061551@qq.com>
 2023-06-05 17:08:02 +08:00
xiaodi22013
8dab8348b0 feat(cfs-cli): cfs-cli cluster info supports displaying all master information 
Signed-off-by: xiaodi22013 <xiaodi22013@gmail.com>
 2023-05-31 11:32:41 +08:00
baijiaruo
e0375721cb enhance: 1.the number of quotas is configurable 
2.delete the quota using the path
         3.added listAll interface to query which volumes are configured with quotas

Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-26 11:17:10 +08:00
chihe
dffdf2f94c enchance(cli): add bad disk check 
Signed-off-by: chihe <chi.he@oppo.com>
 2023-05-26 11:17:10 +08:00
chihe
f108ee55ea enchance(cli): add mp inode and dentry count check 
Signed-off-by: chihe <chi.he@oppo.com>
 2023-05-26 11:17:10 +08:00
leonrayang
d5bb45ea49 docs. update to cubefs 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-05-26 11:17:10 +08:00
yuxiaobo
ac75137295 build: the build script of blobstore is merged into cubefs 
1. rocksdb upgrade to 6.3.6
2. optimize the build scripts
3. fix build errors
4. add env variable for blobstore unit test
5. update git ignore file to ignore build and test result
6. modify open files limit for unit test

Signed-off-by: yuxiaobo <yuxiaobo@oppo.com>
Signed-off-by: slasher <shenjie1@oppo.com>
Signed-off-by: Victor1319 <834863182@qq.com>
 2023-05-23 15:24:53 +08:00
Patrick Wu
b0eacf6c10 fix: parent inode rollback item is not added when dealing with dentry create and deletion 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-05-18 10:48:57 +08:00
true1064
448a180ed7 feat(master,cli): cfs-cli metapartition check add: (1)display unavailable mp, (2)display excess replica num mp, (3) mp no leader consider heartbeat report time 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-05-18 10:48:57 +08:00
true1064
82a10e751c enhance: 'cfs-cli datapartition check' adds display of datapartitions with excessive replica num 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-05-18 10:48:57 +08:00
true1064
ec2aa61ab9 enhance: add isDiscard flag to datapartion; master provides setup isDiscard flag API accordingly; when client try to read an discard dp it will fast fail and avoid retry 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-05-18 10:48:57 +08:00
baijiaruo
7e34c2b4ec fix: update quota default maxBytes update is incorrect 
Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-18 10:48:57 +08:00
baijiaruo
3e14be1f6e enhance:1.set and update without inodeId 
2.When renaming the upper-level or higher-level directory of the quota directory, the full path information of the quota is also modified accordingly.
        3.quota update operation, if only one parameter value is changed, other parameter values remain unchanged
        4.Change the display time to no longer a timestamp
Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-18 10:48:57 +08:00
true1064
bf75c8e6d8 fix: cli newVolUpdateCmd() not set DpReadOnlyWhenVolFull flag by default 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-05-18 10:48:57 +08:00
leonrayang
5403f28479 update: enable cli tool reduce volume replicaNum count and set followerRead automiclly 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-05-18 10:48:57 +08:00
true1064
b8f5e362d2 enhance: 'cfs-cli datapartition check' added to show datapartitions whose replica's used size and file count differ significantly 
Signed-off-by: true1064 <tangjingyu@oppo.com>
 2023-05-18 10:48:57 +08:00
Patrick Wu
7632e3f050 fix: tx timeout may fail when time are not synchronized among metanodes 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-05-18 10:48:57 +08:00
leonrayang
67a1785a46 enhance(master): When the number of replicas is 1 or 2, followerRead is set to true by default in vol create 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-05-18 10:48:57 +08:00
Patrick Wu
79f7cb2ca0 enhance : 1. change transaction info container from map to btree, which can take snapshot and dump to file. 2. set transaction timeout upper limit to 60, and add transaction info to http response 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-05-18 10:48:57 +08:00
baijiaruo
6d434a9108 Fix: 1.delete file failed due to wrong rootinode 
2.maxfiles reaches limited, you can still touch files

Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-18 10:48:57 +08:00
baijiaruo
7bccbc4ea2 Fix: 1.quota list optimization 
2.The path of quota update is consistent, but the inode is inconsistent, so it needs to fail
     3.To delete the rootquota directory, you need to delete the quota first
     4.Solve the problem that the quota has been deleted, but after restarting the master, the quota reappears
Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-18 10:48:57 +08:00
Patrick Wu
342233ac00 enhance: tx mask info optimization 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-05-18 10:48:57 +08:00
baijiaruo
035cb168ad enhance: add directory quota feature 
Signed-off-by: baijiaruo <huyao2@oppo.com>
 2023-05-18 10:48:57 +08:00
Patrick Wu
dc7e5b9885 fix&enhance(meta):transaction bugfix and enhance 
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
 2023-05-18 10:48:57 +08:00
chihe
13069ccf67 [enhance]1.cli support raftForce 2.metanode add log to check leader change 
Signed-off-by: chihe <chi.he@oppo.com>
 2023-05-18 10:48:57 +08:00