gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 02:00:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,153 Commits 48 Branches 52 Tags 378 MiB
c492f6c445
Commit Graph

140 Commits

Author SHA1 Message Date
leonrayang
5c82a084f1 fix(datanode): The contents on which the CRC check depends are inconsistent with each other 
Update the crc check output from error to warn, the inconsistent may be false report and require
more output and make a jugement.

The applyId and crc cann't be atomic under the architecture and are not necessary for performance.

Signed-off-by: leonrayang <chl696@sina.com>
 2024-07-22 17:10:59 +08:00
leonrayang
c9611557a2 fix(datanode):Refactor the paramter of storage write at datanode 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
85445548b2 fix(data): avoid flush extent when it is not dirty 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
f88206c3a2 fix(data): adjust extent flush period to 5min 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
Victor1319
0527b01ce9 refactor(data): remove autoComputeCrc func from snapshot process. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
f665fbbfab fix(client): avoid remove dp concurrent 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-22 17:10:59 +08:00
NaturalSelect
f70de7e82f fix(data): tiny extent snapshot off fix to 128M 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
402da3957d chore(data): change get store used size log level 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
025d2ff89c fix(data): load extents consider snapshot 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
4b91bec9aa fix(data): remove rlock to avoid deadlock. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
4df91ebc6e refactor(all): reslove merge conflicts between 3.3.x and 3.4.0. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
14f86322d0 feat(data): rewrite space manager stop function 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
bc87fcbeb1 fix(data): avoid panic during apply rand write after store close 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
b7eeda10b4 refactor(data): revert data node dp delay load 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
978254d5a5 fix(data): avoid write extent after store closed 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
91c65e17ab feat(data): speed data node restart 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
a0fcae90e6 feat(data): flush extent cache in timer 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
S9054862
297fd4627c feat(data): load extent info on demand 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
0d16adfbc7 feat(fsck): support gc extent mark and delete at the same time. 
1. support mark gc extents and delete gc extents.
2. after delete gc extents success, rename extents to xx.succ.

Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
S9054862
e812d1c37a fix(data): speed up initBaseFileID 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
S9054862
00ecb62b94 feat(data): limit io current on a disk when load/stop dp 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
45b6896c46 feat(data): when extent is mark delete, write op not failed. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
Victor1319
941b71a2cf ๐Ÿž fix(data): when read locked extent, only print error log not return error. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
shuqiang-zheng
68cf97ab68 fix(master):fix the problem of TestCreateColdVol 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-07-18 16:57:38 +08:00
Victor1319
0c62eecc3a ๐Ÿž fix(data): not check before time when lock extent to avoid fail. 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
Victor1319
9ede5a8cfe ๐Ÿž fix(fsck&data): support repeat delete extent 
Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
Victor1319
2d79270c41 โœจ feat(fsck & data): refacotr gc logic 
1. "getAllExtent?id=xx&beforeTime=xx" supports get extent GC flags.
2. When deleting extent, also check for GC flags. If there are no GC flags, return an error.
3. Optimize related log printing by outputting relationship information and execution time, reduce unnecessary logs, and improve performance.
4. Optimize the implementation of the "getExtents" list interface and the performance of batch locking interfaces. Read all the latest information of extents at once to avoid accessing the disk for each extent, reducing timeout calls.
5. Support outputting profiles for easy performance analysis.
6. "cleanBadExtents" and "rollbackBadExtents" support a "clean" parameter to control whether to perform data cleanup and overwrite.
7. Use a task pool to support multi-threaded concurrent tasks.
8. Limit the concurrent task number to 3 for each node.
9. Back up the execution result of the previous command each time it is executed for easy tracing.
10. Remove "from-dp" from "getMpExtents" and "getDpExtents" and directly support concurrent retrieval of full volume information.
11. When obtaining MP information, take the maximum value among the three nodes as the reference.
12. When persisting MP information, use a buffer to optimize performance and avoid reading and writing to disk each time.
13. Analyze performance bottlenecks and optimize the process of obtaining MP extents.
14. If there is an exception during the MP retrieval process, exit directly.

Signed-off-by: Victor1319 <834863182@qq.com>
 2024-07-18 16:57:38 +08:00
huyao2
60070d5696 feat(fsck): Optimize fsck gc features, including: 
1. The before of getDpextent must be at least 3 hours smaller than the current time.
2. The use of locks in the lock extent is changed from mutual exclusion locks to read-write locks.
3. Add a time judgment when locking extent. If the time is greater than before time, it means it has been changed and it will fail.
4. Support concurrency when cleaning bad extent
5. Add verification of master and volume names

Signed-off-by: huyao2 <huyao2@oppo.com>
 2024-07-18 16:57:38 +08:00
baijiaruo
58fec1f635 fix(datanode): Fix the problem of extent lock error when the extent is copied in the scene 
Signed-off-by: baijiaruo <baijiaruo@126.com>
 2024-07-18 16:57:38 +08:00
baijiaruo
5b6723ab0f enhance(cli): add fsck gc tool 
Signed-off-by: baijiaruo <baijiaruo@126.com>
 2024-07-18 16:57:38 +08:00
chihe
db4ce8dcb0 fix(datanode): add debug log for reading extent repair packet 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-18 16:57:38 +08:00
W9068822
470967aae4 feat(ci): fix format failed. 
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
 2024-07-05 17:05:43 +08:00
NaturalSelect
495b8e4d77 fix(data): batch delete normal extent not punch del, if disable snap 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-05 17:05:43 +08:00
chihe
755a827cdb fix(datanode): when datanode remove raft memeber, use local nodeID 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-02 16:35:56 +08:00
S9054862
d6532c1794 fix(client): treat try again error as a special error 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-02 16:35:56 +08:00
S9054862
8ad316ecfc feat(data): repair block size support master config 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-01 15:09:55 +08:00
chihe
1330695c0f fix(datnode): get copy of dp replica when excuting updateMaxMinAppliedID 
Signed-off-by: chihe <chihe@oppo.com>
 2024-07-01 15:09:55 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools 
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
 2024-05-14 09:54:13 +08:00
shuqiang-zheng
ae3b170fe6 feat(master): Make a volume unreadable when it freezes.. 
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
 2024-03-28 11:43:55 +08:00
leonrayang
7c1382d762 feat(reconstruct): reconstruct code 
Signed-off-by: leonrayang <changliang@oppo.com>
 2024-03-27 14:24:50 +08:00
leonrayang
7b93604b34 fix(datanode): dp may on decommision while doing auto compute crc, so change it to warn 
close:#21979158

Signed-off-by: leonrayang <changliang@oppo.com>
 2024-03-27 14:24:50 +08:00
leonrayang
c867f09bd7 feat(datanode): add testcase for repair routine 
close:#21938895

Signed-off-by: leonrayang <changliang@oppo.com>
 2024-03-27 14:24:50 +08:00
leonrayang
b5895c07b1 fix(datanode): normal extent repair need punch hole in senario of snapshot 
close:#21938895

Signed-off-by: leonrayang <changliang@oppo.com>
 2024-03-27 14:24:50 +08:00
chihe
f66c732104 feat(datanode): Support the functionality of allowing only one extent data read at a time per disk 
Signed-off-by: chihe <chihe@oppo.com>
 2024-03-27 14:24:50 +08:00
chihe
1042fa9857 feat(datanode): add log for extent repair speed 
Signed-off-by: chihe <chi.he@oppo.com>
 2024-03-27 14:24:50 +08:00
leonrayang
91a3448515 feat(datanode): add testCase for snapshot crc and reconstruct some related code 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-02-29 15:19:42 +08:00
NaturalSelect
b247cad482 enhance(data): return error when failed to mark delete extent 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-02-28 16:31:57 +08:00
slasher
5d5e45cb0a chore(util): store current time to atomic value 
BenchmarkGetCurrentTime-4    100000000    11.60 ns/op            0 B/op          0 allocs/op
-->
BenchmarkGetCurrentTime-4    895091701     1.31 ns/op            0 B/op          0 allocs/op

Signed-off-by: slasher <mcq.sejust@gmail.com>
 2024-01-31 19:13:25 +08:00
slasher
fc431f6360 chore(ci): remove ltp in docker if no ltptest 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-12-28 17:13:05 +08:00
leonrayang
f6691a7b74 enhance(datanode):Empty extents are also repaired, and the data partition check considers the count of empty extents as a factor 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-12-20 15:48:38 +08:00