Commit Graph

6089 Commits

Author SHA1 Message Date
Wu Huocheng
5e0a67da5b chore(metanode): update the codes of apply snapshot log.#1000578895
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 17:36:39 +08:00
Wu Huocheng
664f2a9586 fix(master): fix the bug that add learner plan's overload is null.#1000578895
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 15:44:05 +08:00
Wu Huocheng
7ea467d7cb fix(metanode): batch commit in ApplySnapshot and trace op time.#1000579599
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 15:34:44 +08:00
Wu Huocheng
b9bd14a91b feat(metanode): add warn metric for rocksdb disk usage.#1000578895
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 14:50:09 +08:00
Wu Huocheng
85cdb71774 feat(master): add default volume store mode into configuration.#1000578895
fix(master): limit the max learner number to 5.#1000587619

Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 14:48:09 +08:00
Wu Huocheng
93c4a75db3 chore(master): optimization of the master codes.#1000511705
fix(master): selete one address tobe delete.#1000577106

fix(master): count the required mp replica. #1000577106

Restore the count define back to original.

fix(master): check commit in the raft status.#1000580439

Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-26 14:42:52 +08:00
zhumingze
4c7e409a73 feat(master): Adaptation of the maximum number of large disk node dp. #1000562853
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-26 14:14:25 +08:00
Wu Huocheng
fa86ecde97 feat(cli): add cli for metanode tag.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-22 11:30:50 +08:00
Wu Huocheng
1ff30ad62c fix(master): fix the bug that checking apply id fails.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-22 11:30:50 +08:00
Wu Huocheng
1727527195 chore(master): delete the unused code.#1000511705
The createStoreModeChangePlan is replaced with batchAddLearner.
So delete the unused code that is createStoreModeChangePlan.

Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-22 11:30:50 +08:00
Wu Huocheng
4345b97fbe feat(master): set zone, nodeset, ip list for learner batch api.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-22 11:30:50 +08:00
leonrayang
8ef937dfb8 Optimize lookup process and add attr to the response
close:#1000519910

Signed-off-by: leonrayang <changliang@oppo.com>
2025-12-18 11:38:50 +08:00
zhumingze
bfa9bfd727 fix(master): Add process dp num for manual distribution optimization interface. #1000239501
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-18 11:32:10 +08:00
zhumingze
951db45139 fix(master): Fix the issue where the dp num of vol can't reach 10. #1000541227
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-18 11:31:53 +08:00
Victor1319
fd9183ec06 feat(raft): enhance logging for peer management and learner recovery. #1000560474
- Add logging for peer removal actions in raft state machine
- Update addMetaReplica and addMetaReplicaLearner methods for logging
- Improve logging in partition FSM operations for member changes
- Enhance logging for learner mode recovery checks and actions
- Modify cluster balance logic to include additional parameters for learners

This commit improves the observability of raft operations and learner recovery processes.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-17 17:36:42 +08:00
Victor1319
5baf97ccb5 feat(cluster): implement learner mode recovery timeout configuration. #1000560474
- Add learnerRecoverTimeoutSeconds parameter for cluster settings
- Update command line interface to accept learner recovery timeout
- Enhance API to parse and handle learnerRecoverTimeoutSeconds
- Modify cluster view to display learnerRecoverTimeoutSeconds status
- Implement logic to set learnerRecoverTimeoutSeconds in cluster

This commit enhances cluster management by allowing configuration of
the learner mode recovery timeout, improving the system's resilience
during decommissioning processes.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-17 14:31:54 +08:00
Victor1319
f4511216a8 feat(cluster): add enableMpDecommissionByLearner parameter for cluster management. #1000539820
- Introduce enableMpDecommissionByLearner flag for cluster settings
- Update command line interface to accept new parameter
- Enhance API to parse and handle enableMpDecommissionByLearner
- Modify cluster view to display enableMpDecommissionByLearner status
- Implement logic to set enableMpDecommissionByLearner in cluster

This commit enhances cluster management capabilities by allowing
configuration of the mp decommissioning behavior.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-17 14:29:06 +08:00
Victor1319
ce3c0048df refactor(metanode): replace preparePartitionCheckMultiVersion with getPartitionCheckProxyMultiVersion. #1000348192
- Remove preparePartitionCheckMultiVersion function and its logic
- Update calls to use getPartitionCheckProxyMultiVersion instead
- Simplify partition check handling in multiple operations
- Ensure consistent behavior across metadata manager methods

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-17 14:29:06 +08:00
Wu Huocheng
e03c6f6173 chore(master): Optimize the code.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-16 17:41:19 +08:00
Wu Huocheng
ee32d2271d chore(metanode): Optimize the code.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-16 15:17:48 +08:00
Wu Huocheng
6eb8a24b4f chore(metanode): use GetInodeBuf instead of NewBytesBuffEx.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-16 11:48:34 +08:00
leonrayang
5ee498cc38 fix(metanode): Remove code blocks with extensive duplicated logic.#1000511705
Signed-off-by: leonrayang <changliang@oppo.com>
2025-12-15 19:43:23 +08:00
Wu Huocheng
3a3fcb7d82 chore(metanode): rewrite codes according to comments.#1000511705
refine the code with cursor

Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:39:37 +08:00
Wu Huocheng
c0f10c17af fix(metanode): skip to check the applyid is zero.#1000511705
It reports mpisnull, and the master stops to check it.

Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:25:06 +08:00
Wu Huocheng
42ba9c79d6 chore(master): rewrite the codes to make it better.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:24:01 +08:00
Wu Huocheng
75683b488f fix(metanode): it doesn't need to add 100 to apply id when calculate md5sum.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:23:16 +08:00
Wu Huocheng
d572932a0a fix(master): fix the bug that can't allocate resource for mp plan.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:22:39 +08:00
Wu Huocheng
da23a16b7d feat(master): check metapartition md5sum.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:22:14 +08:00
Wu Huocheng
0817396958 feat(metanode): add api to calculate mp's md5sum.#1000457722
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:21:40 +08:00
Wu Huocheng
d1e0d0da1d fix(metanode): skip to update size of mp if it is not changed.#1000457722
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:21:05 +08:00
Wu Huocheng
a89d685533 feat(metanode): get mp inode and dentry md5 sum.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:20:49 +08:00
Wu Huocheng
3612f75ef4 feat(master): batch create raft learner and promote.#1000511705
Signed-off-by: Wu Huocheng <wuhuocheng@oppo.com>
2025-12-15 16:20:36 +08:00
zhumingze
d4efb36935 feat(master): Add a manual execution interface for distribution optimization.
.#1000239501

Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-15 11:19:47 +08:00
zhumingze
d250ed5b92 fix(sdk): Trash Directory Structure Optimization. #1000528020
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-15 10:27:41 +08:00
leonrayang
8adaad9f90 fix(client): Disable xattr security.capability to avoid useless request to backend #1000519910
Signed-off-by: leonrayang <changliang@oppo.com>
2025-12-15 09:35:39 +08:00
leonrayang
86030fe047 feature(client): Reduce unessaray inode get request if quota not enabled
close:#1000519910

Signed-off-by: leonrayang <changliang@oppo.com>
2025-12-15 09:35:39 +08:00
leonrayang
231e44edcd feature(client): Add Dentry not exist cache
close:#1000519910

Signed-off-by: leonrayang <changliang@oppo.com>
2025-12-15 09:35:39 +08:00
Victor1319
faa1a4bbc7 test(metanode): enhance FreeList tests with edge cases and concurrency. #1000348192
- Rename TestFreeList to TestFreeList_BasicOperations for clarity
- Add tests for edge cases including empty list and invalid inode
- Implement FIFO order tests to ensure correct item retrieval
- Add tests for removing items from different positions
- Introduce concurrent operations tests to validate thread safety
- Enhance mixed operations tests for comprehensive coverage

This commit significantly improves the test coverage for FreeList,
ensuring robustness against edge cases and concurrent operations within the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
80fe126da2 refactor(metanode): remove unused mutex and optimize multipart handling. #1000348192
- Remove unused RWMutex from MetaNodeMetrics struct
- Rename TestMultipartExtend_Bytes to TestMUMultipartExtend_Bytes
- Add comprehensive unit tests for Multipart and Part structs
- Implement tests for edge cases and concurrent access scenarios
- Optimize buffer handling in multipart serialization methods

This commit enhances the performance and readability of the metanode
implementation, ensuring better resource management and test coverage.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
9009567e2c test(metanode): add comprehensive unit tests for Dentry functionality.#1000348192
- Implement helper functions for creating and comparing dentries
- Add tests for marshaling and unmarshaling of dentry items
- Include tests for basic properties and string representation of dentries
- Implement tests for edge cases with long, special, and unicode names
- Add tests for handling empty and zero values in dentries
- Enhance performance benchmarks for dentry operations.

This commit significantly improves the test coverage for Dentry,
ensuring robustness and correctness in the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
2882c8caa3 test(metanode): add comprehensive unit tests for MetaQuotaManager and related structures. #1000348192
- Implement tests for NewQuotaManager initialization and properties
- Add marshal/unmarshal tests for MetaQuotaInode and TxMetaQuotaInode
- Include edge case tests for empty and large quota IDs
- Add concurrent access tests for quota manager methods
- Enhance tests for quota heartbeat info setting and retrieval
- Validate error handling for invalid data during unmarshal operations

This commit significantly improves the test coverage for the MetaQuotaManager,
ensuring robustness and correctness in the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
6e86997b81 test(metanode): enhance buffer tests with comprehensive scenarios #1000348192
- Rename buffer test functions for clarity and consistency
- Add tests for nil pointer handling in Put functions
- Implement buffer reuse tests to ensure proper reset behavior
- Add concurrent access tests for buffer pools
- Introduce tests for buffer capacity limits and various read operations
- Enhance binary serialization tests for sorted extents

This commit significantly improves the test coverage for buffer operations,
ensuring robustness and correctness in the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
f39cf0e5aa fix(metanode): enhance type safety and performance in SortedObjExtents. #1000348192
- Implement safe type assertions for sorted extents to prevent panics
- Increase buffer size in String method for improved performance
- Fix JSON formatting in String method for expiredTime field
- Refactor IsEmpty and Size methods for better readability
- Add comprehensive unit tests for SortedObjExtents, covering edge cases
- Improve concurrent access handling in tests for SortedObjExtents

This commit enhances the type safety and performance of the SortedObjExtents
implementation, ensuring robustness and correctness in the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
6a414a633a test(metanode): enhance uniqChecker tests with edge cases and concurrency. #1000348192
- Rename existing test functions for clarity and consistency
- Add tests for edge cases including zero and large uniqids
- Implement concurrent access tests for legalIn method
- Introduce eviction logic tests for both recent and old operations
- Add tests for marshaling and unmarshaling edge cases
- Improve queue operation tests including scan, clone, and reset

This commit significantly enhances the test coverage for the uniqChecker,
ensuring robustness against edge cases and concurrent operations within the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
b58cfa92f4 test(metanode): add unit tests for DelExtFile functionality . #1000348192
- Implemented unit tests for DelExtFile methods including GetIdx, Len,
  Swap, Less, and Sort.
- Created mockFileInfo type to simulate os.FileInfo for testing.
- Added test cases to validate behavior for valid and invalid file names.
- Ensured proper handling of sorting and filtering of file information.

This commit enhances test coverage for DelExtFile, ensuring the
correctness of its methods within the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
a48c593ecb refactor(raftstore): enhance partition interface and implementation#1000347751
- Changed the return type of Status method to simplify usage
- Added GetID and GetWalPath methods to the Partition interface
- Improved error handling in Delete method for better clarity
- Updated comments for methods to enhance understanding
- Simplified return statements in several methods for consistency

This commit refines the partition interface and its implementation,
improving code readability and maintainability within the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
d146a3945b refactor(monitor): optimize monitoring data structures and improve comments. #1000347751
- Replaced map-based storage with sync.Map for zombie and noLeader durations
- Enhanced comments for better clarity and understanding of the code
- Simplified the logic for monitoring zombie peers and elections
- Improved error handling and logging for raft log directory creation

This commit refines the monitoring implementation and enhances code readability
within the CubeFS system.

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:57:16 +08:00
Victor1319
b9c4a84f9d feat(lcnode): enhance inode retrieval and batch processing for dentries. #1000523711
- Refactor handleFile to utilize cached inode info
- Implement batchGetFileInodeInfo for efficient inode retrieval
- Update ScanDentry struct to include HasInodeInfo and InodeInfo
- Modify handleDirLimitDepthFirst and handleDirLimitBreadthFirst to use batch processing
- Add BatchInodeGet method to MetaWrapper interface

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2025-12-09 14:15:46 +08:00
zhumingze
363222a82f feat(master): Use cursor to optimize some code in the master. #1000389771
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-08 14:39:38 +08:00
zhumingze
415984dd59 feat(data): Use Cursor to optimiza datanode code part3. #1000389775
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-12-08 14:38:58 +08:00