gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 02:00:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,677 Commits 48 Branches 52 Tags 378 MiB
3396accd80
Commit Graph

48 Commits

Author SHA1 Message Date
leonrayang
3396accd80 feat(projectstructure): reorganize the module directory in the root to a suitable subdirectory 
close:#22458839

Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
leonrayang
111f9e9903 feat(dataemeta): cluster switch on snapshot take effect on module handle protocal process 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-12-26 09:24:22 +08:00
leonrayang
c9611557a2 fix(datanode):Refactor the paramter of storage write at datanode 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-07-22 17:10:59 +08:00
Victor1319
4df91ebc6e refactor(all): reslove merge conflicts between 3.3.x and 3.4.0. 
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
 2024-07-18 16:57:38 +08:00
NaturalSelect
bc87fcbeb1 fix(data): avoid panic during apply rand write after store close 
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
 2024-07-18 16:57:38 +08:00
chihe
84223360b4 fix(client): trash support back-end audit 2024-07-18 16:57:38 +08:00
S9054862
8ad316ecfc feat(data): repair block size support master config 
Signed-off-by: S9054862 <huangzhibin1@oppo.com>
 2024-07-01 15:09:55 +08:00
leonrayang
b5895c07b1 fix(datanode): normal extent repair need punch hole in senario of snapshot 
close:#21938895

Signed-off-by: leonrayang <changliang@oppo.com>
 2024-03-27 14:24:50 +08:00
leonrayang
91a3448515 feat(datanode): add testCase for snapshot crc and reconstruct some related code 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-02-29 15:19:42 +08:00
leonrayang
cdc7ecc01f fix(datanode): store verlist at metadata of data partition for replay randomWrite 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-02-29 15:19:42 +08:00
leonrayang
d58736436e fix(update|client): snapshot. Refactor some codes to solve the synchronization update problem of client streamer's seq and datanode 
Signed-off-by: leonrayang <chl696@sina.com>
 2024-01-22 15:22:12 +08:00
leonrayang
d275ce4423 fix(datanode):snapshot. Verlist update requires copy replacement assignment 
bug:dps of volume be assign with the same verlist

Signed-off-by: leonrayang <chl696@sina.com>
 2024-01-22 15:22:12 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt 
Signed-off-by: slasher <shenjie1@oppo.com>
 2024-01-12 14:21:16 +08:00
leonrayang
77f2746efa fix(metanode):inode lock scope in inner tempory snapshot deletion should not include process of fsm operation and trigger deadlock 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-12-07 10:44:08 +08:00
slasher
a07b5cae3a feat(datanode): limit disk flow and io concurrency 
yun #20494841

Signed-off-by: slasher <shenjie1@oppo.com>
 2023-11-29 18:40:43 +08:00
lily-lee
a8620dc1e9 fix(code scan): fix code-scanning problems. 
Signed-off-by: lily-lee <lilylee88756@gmail.com>
 2023-11-28 12:32:15 +08:00
chihe
4db47095d5 bugfix(datanode):fix datanode raft snapshot exceed the limit 
close: #2738

Signed-off-by: chihe <chi.he@oppo.com>
 2023-11-22 16:59:29 +08:00
leonrayang
67e82b7491 fix(datanode/client):snapshot.reuse extent may get wrong extent and offset exceed normal scope 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-10-17 14:54:57 +08:00
leonrayang
3c28424667 fix(datanode):snapshot.rebase code wrong,datanode apply random write run twice and wrong" -s 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-09-27 14:34:28 +08:00
leonrayang
34819db546 update: datanode repair need take consider partition of snapshot append write after on the 128M 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-09-19 14:05:49 +08:00
leonrayang
3dfbf48255 enhance(metanode|client): Optimizing the Logic for Synchronizing Verseq between Nodes 
The synchronization process now includes synchronizing the "verlist". If there is a difference in version numbers,
the client and metanode will mutually push the latest "verlist" to each other.
Client writes will have increased retries to carry the "verlist".
The purpose of all this is to ensure that the metanode obtains the latest and most comprehensive data,
reducing the possibility of missing data during individual Verseq pushes.

To make it simple, packet of randomWriteVer from client always contains verlist
VerList in dataNode not persist

This update also optimized the message synchronization logic in the process, adopting an asynchronous approach using channels to handle it.

Signed-off-by: leonrayang <chl696@sina.com>
 2023-09-19 14:05:49 +08:00
slasher
cf415f1acf style(datanode): format code of datanode 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-09-11 15:38:54 +08:00
slasher
5e93d2cea8 style(datanode): gofmt for datanode 
Signed-off-by: slasher <shenjie1@oppo.com>
 2023-09-11 15:38:54 +08:00
leonrayang
ca241eb1b7 fix(datanode): The result of RandomWriteSubmit should be checked for the retcode interface type. Failure to do so may trigger exceptions or errors in the code execution. 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
leonrayang
88c2c5d7ff update: snapshot.Reduce extent count triggerd by ver inconsistent 
optimize code on function naming and abstract public function

Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
leonrayang
5ae500768b feature: support multi version snapshot 
Signed-off-by: leonrayang <chl696@sina.com>
 2023-08-30 20:05:32 +08:00
Tao Li
bf8aa1640f [Enhancement] Optimize the qos for random write 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-23 09:58:26 +08:00
leonrayang
9a30b9f884 update: update to CubeFS 
Signed-off-by: leonrayang <chl696@sina.com>
 2022-08-22 18:16:09 +08:00
leonrayang
3fdd01cae9 feature: add qos flow limit control process for 2nd cache layer 
1) update master inteface of datanode enable qos
2) master clean client info if long time no report
3) enable qos online shutdown and startup
4) master set client alloc preriod and count of girds hit trigger condition
5) datanode qos limit value set by master and add disable conf and http interface
6) client qos init buffer calculate according on real buffer storage
7) create limiter for both hot and cold volume

Signed-off-by: leonrayang <chl696@sina.com>
 2022-08-10 22:05:45 +08:00
leonrayang
2a67ed4104 feature: enable go mod and update vendor 
1. add chubaofs-blobstore client
 2. move origin modified vendor code to depends
    1) raft
    2) spf13
    3) fuse
    3) jacobsa
    4) rocksdb

Signed-off-by: leonrayang <chl696@sina.com>
 2022-04-13 15:57:55 +08:00
leonrayang
2970e2c3d1 update. go import path change from chubaofs to cubefs 
Signed-off-by: leonrayang <chl696@sina.com>
 2022-01-25 18:17:02 +08:00
Victor1319
b923b56d8d fix overwrite panic exception when available space is less than reserved disk space 
Signed-off-by: Victor1319 <834863182@qq.com>
 2021-04-21 14:17:00 +08:00
wanghongyan
7a056735ec fix: handle dataPartition raft apply error 
Signed-off-by: wanghongyan <wanghongyan@oppo.com>
 2021-01-27 11:29:27 +08:00
xrefft
e65631d904 fix: fix raft error handling logic 
fix raft error handling logic, non-disk errors will stop the current raft instance and will not
panic the entire process

Signed-off-by: xrefft <cserxiao@qq.com>
 2021-01-27 11:29:21 +08:00
awzhgw
d22d8dd2ae Enhancement: datanode accelerate recovery rate 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2020-09-07 19:13:51 +08:00
awzhgw
0c41b2a78c Fix: datanode start failed 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2020-09-03 16:18:59 +08:00
awzhgw
ad6423fb16 Fix: datanode cannot start,if randomwrite request apply error,then igone it 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2020-09-03 15:47:58 +08:00
awzhgw
73355d2f1f Fix: Storage :when write tinyExtent,if offset!=e.datasize,return error 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-08-28 13:56:06 +08:00
awzhgw
b370e8d220 Fix: Datanode: when tinyExtentRepair auto repair,it has been 
repairTimeout

Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-08-28 13:56:06 +08:00
awzhgw
4dd740c435 Fix Datanode retain RaftLog: 
1.Datanode add log on recovery
2.Datanode Change raft log retain log to 10
3.delete unused file datanode/partition_metrics.go

Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-08-22 17:51:36 +08:00
awzhgw
c366fcde54 Refactoring:change partitionId varliable to partitionID 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-08-16 14:42:09 +08:00
bjguoweilong
02e1989065 Enhancement: when datanode or metanode start,must compare partition with master and gofmt project 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-07-16 22:06:24 +08:00
awzhgw
6144a94c28 change random write raft serialize not use json 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-07-10 15:59:18 +08:00
awzhgw
76ac26b91e add when datanode disk error ,then ump warn,and change disk error to -1 
Signed-off-by: awzhgw <guowl18702995996@gmail.com>
 2019-06-11 12:53:17 +08:00
awzhgw
36d97f1034 master: 
1.add ump warn packet
  2.partition has recover,but the status don't change
  3.remove redundancy calling loadMetaData method
  4.reload meta data when change leader

metanode:
   1. add totalMem in configFile
   2.change not raft leader to tryOtherAddr error

 datanode:
   1.when datapartition not exsit,tell client with proto.OptryAgain resultcode
   2.change disk retrain default space to 20gb
   3.heartbeat report add volname

 raft:
   1.use raft.ErrNotLeader replace ErrNotLeader
   2.raft became leader must apply from oldapplyid to commitid

 client:
   1.  create a dummy node instance if inode does not exist
   2. add ump alarm for read/write/fsync errors

log:
   1.   auto create subdir to logdir

 error:
   1.delete juju error packet
 2019-04-30 09:31:58 +08:00
awzhgw
97f2a18f5c rename repo from cfs to chubaofs 2019-04-28 15:12:17 +08:00
awzhgw
493af438b0 client: 
1.try other hosts for network errors
  2.use single instance for File and Dir object
metanode:
  1.use tcp packet notify all raft followers to free inodes
datanode:
  1.remove read check appliedID
  2.change extentcache fp to 100
  3.fix always repair has delete extents
repl:
  1.modify funcName from reciveFromFollower to checkLocalResultAndReceiveFromFollower
 2019-04-02 10:24:10 +08:00
Skypigltp
3eae179b1b init datapartition raft 2019-03-15 12:32:43 +08:00