update: update to CubeFS

Signed-off-by: leonrayang <chl696@sina.com>
This commit is contained in:
leonrayang 2022-08-22 12:45:41 +08:00 committed by Victor1319
parent 3df66d786d
commit 9a30b9f884
326 changed files with 417 additions and 417 deletions

8
NOTICE
View File

@ -1,10 +1,10 @@
ChubaoFS
Copyright 2017-present The Chubao Authors.
CubeFS
Copyright 2017-present The CubeFS Authors.
This product includes software devoloped at
The Apache Software Foundation (http://www.apache.org/).
ChubaoFS reused the code from the MinIO project to implement the
CubeFS reused the code from the MinIO project to implement the
signature algorithm V2, signature algorithm V4 and policy-related
logic for S3-compatible object storage interfaces.
@ -16,4 +16,4 @@ This product includes software developed at MinIO, Inc.
The MinIO project contains unmodified/modified subcomponents too with
separate copyright notices and license terms. Your use of the source
code for these subcomponents is subject to the terms and conditions
of Apache License Version 2.0
of Apache License Version 2.0

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2022 The ChubaoFS Authors.
// Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,5 +1,5 @@
// Copyright (C) 2020 Juicefs
// Modified work Copyright 2022 The ChubaoFS Authors.
// Modified work Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2022 The ChubaoFS Authors.
// Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2022 The ChubaoFS Authors.
// Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2022 The ChubaoFS Authors.
// Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2022 The Chubao Authors.
// Copyright 2022 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -222,7 +222,7 @@ func main() {
os.Exit(1)
}
syslog.Printf("Hello, ChubaoFS Storage\n%s\n", Version)
syslog.Printf("Hello, CubeFS Storage\n%s\n", Version)
err = modifyOpenFiles()
if err != nil {
@ -260,7 +260,7 @@ func main() {
err = server.Start(cfg)
if err != nil {
log.LogFlush()
err = errors.NewErrorf("Fatal: failed to start the ChubaoFS %s daemon err %v - ", role, err)
err = errors.NewErrorf("Fatal: failed to start the CubeFS %s daemon err %v - ", role, err)
syslog.Println(err)
daemonize.SignalOutcome(err)
os.Exit(1)

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -28,7 +28,7 @@ const (
cmdClusterShort = "Manage cluster components"
)
func (cmd *ChubaoFSCmd) newClusterCmd(client *master.MasterClient) *cobra.Command {
func (cmd *CubeFSCmd) newClusterCmd(client *master.MasterClient) *cobra.Command {
var clusterCmd = &cobra.Command{
Use: cmdClusterUse,
Short: cmdClusterShort,
@ -123,11 +123,11 @@ func newClusterFreezeCmd(client *master.MasterClient) *cobra.Command {
Short: cmdClusterFreezeShort,
Args: cobra.MinimumNArgs(1),
Long: `Turn on or off the automatic allocation of the data partitions.
If 'freeze=false', ChubaoFS WILL automatically allocate new data partitions for the volume when:
If 'freeze=false', CubeFS WILL automatically allocate new data partitions for the volume when:
1. the used space is below the max capacity,
2. and the number of r&w data partition is less than 20.
If 'freeze=true', ChubaoFS WILL NOT automatically allocate new data partitions `,
If 'freeze=true', CubeFS WILL NOT automatically allocate new data partitions `,
Run: func(cmd *cobra.Command, args []string) {
var (
err error

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2021 The Chubao Authors.
// Copyright 2021 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -26,16 +26,16 @@ import (
)
const (
cmdRootShort = "ChubaoFS Command Line Interface (CLI)"
cmdRootShort = "CubeFS Command Line Interface (CLI)"
)
type ChubaoFSCmd struct {
type CubeFSCmd struct {
CFSCmd *cobra.Command
}
func NewRootCmd(client *master.MasterClient) *ChubaoFSCmd {
func NewRootCmd(client *master.MasterClient) *CubeFSCmd {
var optShowVersion bool
var cmd = &ChubaoFSCmd{
var cmd = &CubeFSCmd{
CFSCmd: &cobra.Command{
Use: path.Base(os.Args[0]),
Short: cmdRootShort,

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -93,7 +93,7 @@ func newUserCreateCmd(client *master.MasterClient) *cobra.Command {
displaySecretKey = optSecretKey
}
var displayUserType = userType.String()
fmt.Printf("Create a new ChubaoFS cluster user\n")
fmt.Printf("Create a new CubeFS cluster user\n")
stdout(" User ID : %v\n", userID)
stdout(" Password : %v\n", displayPassword)
stdout(" Access Key: %v\n", displayAccessKey)
@ -181,7 +181,7 @@ func newUserUpdateCmd(client *master.MasterClient) *cobra.Command {
if optUserType != "" {
displayUserType = optUserType
}
fmt.Printf("Update ChubaoFS cluster user\n")
fmt.Printf("Update CubeFS cluster user\n")
stdout(" User ID : %v\n", userID)
stdout(" Access Key: %v\n", displayAccessKey)
stdout(" Secret Key: %v\n", displaySecretKey)

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -83,8 +83,8 @@ const (
Role = "Client"
DefaultIP = "127.0.0.1"
DynamicUDSNameFormat = "/tmp/ChubaoFS-fdstore-%v.sock"
DefaultUDSName = "/tmp/ChubaoFS-fdstore.sock"
DynamicUDSNameFormat = "/tmp/CubeFS-fdstore-%v.sock"
DefaultUDSName = "/tmp/CubeFS-fdstore.sock"
DefaultLogPath = "/var/log/chubaofs"
)

View File

@ -8,7 +8,7 @@
"192.168.0.12:17010",
"192.168.0.13:17010"
],
"objectNodeDomain": "object.chubao.io",
"objectNodeDomain": "object.cubefs.io",
"monitor_addr": "http://192.168.0.102:9090",
"dashboard_addr": "http://192.168.0.103",
"monitor_app": "cfs",

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -245,7 +245,7 @@ func main() {
os.Exit(1)
}
syslog.Printf("Hello, ChubaoFS Storage\n%s\n", Version)
syslog.Printf("Hello, CubeFS Storage\n%s\n", Version)
err = modifyOpenFiles()
if err != nil {
@ -283,7 +283,7 @@ func main() {
err = server.Start(cfg)
if err != nil {
log.LogFlush()
err = errors.NewErrorf("Fatal: failed to start the ChubaoFS %s daemon err %v - ", role, err)
err = errors.NewErrorf("Fatal: failed to start the CubeFS %s daemon err %v - ", role, err)
syslog.Println(err)
daemonize.SignalOutcome(err)
os.Exit(1)

View File

@ -1,7 +1,7 @@
/*
* MIT License
* Copyright (c) 2016 Samsara Networks Inc.
* Modifications copyright 2020 The ChubaoFS Authors.
* Modifications copyright 2020 The CubeFS Authors.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2018 The Chubao Authors.
// Copyright 2018 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
// Copyright 2021 The Chubao Authors.
// Copyright 2021 The CubeFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.

View File

@ -2,15 +2,15 @@
Version="1.0.0"
if [[ -n "$1" ]] ;then
# docker image tag of ChubaoFS
# docker image tag of CubeFS
Version=$1
fi
RootPath=$(cd $(dirname $0)/..; pwd)
source ${RootPath}/build/build.sh
CfsServer="chubaofs/cfs-server:$Version"
CfsClient="chubaofs/cfs-client:$Version"
CfsServer="cubefs/cfs-server:$Version"
CfsClient="cubefs/cfs-client:$Version"
docker build -t ${CfsServer} -f ${RootPath}/docker/Dockerfile-cfs --target server ${RootPath}
docker build -t ${CfsClient} -f ${RootPath}/docker/Dockerfile-cfs --target client ${RootPath}

View File

@ -10,7 +10,7 @@ url2="http://${host}:${port}"
ds_url="${url2}/api/datasources"
dashboard_url="${url2}/api/dashboard/db"
org="chubaofs.io"
org="cubefs.io"
parse_json(){
echo "$1" | sed "s/.*\"$2\":\([^,}]*\).*/\1/"

View File

@ -1,7 +1,7 @@
apiVersion: 1
providers:
- name: 'ChubaoFS'
- name: 'CubeFS'
orgId: 1
folder: ''
type: file

View File

@ -11,10 +11,10 @@ help() {
Usage: ./run_docker.sh [ -h | --help ] [ -d | --disk </disk/path> ] [ -l | --ltptest ]
-h, --help show help info
-d, --disk </disk/path> set ChubaoFS DataNode local disk path
-b, --build build ChubaoFS server and client
-s, --server start ChubaoFS servers docker image
-c, --client start ChubaoFS client docker image
-d, --disk </disk/path> set CubeFS DataNode local disk path
-b, --build build CubeFS server and client
-s, --server start CubeFS servers docker image
-c, --client start CubeFS client docker image
-m, --monitor start monitor web ui
-l, --ltptest run ltp test
-r, --run run servers, client and monitor

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -1,4 +1,4 @@
# Copyright 2020 The ChubaoFS Authors.
# Copyright 2020 The CubeFS Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

View File

@ -32,7 +32,7 @@ init_cli() {
cd ${conf_path}
${cli} completion &> /dev/null
echo 'source '${conf_path}'/cfs-cli.sh' >> ~/.bashrc
echo -n "Installing ChubaoFS cli tool ... "
echo -n "Installing CubeFS cli tool ... "
echo -e "\033[32mdone\033[0m"
}
check_cluster() {

View File

@ -8,7 +8,7 @@ As the CLI continues to improve, it will eventually achieve 100% coverage of the
Compile and Configure
-----------------------
After downloading the ChubaoFS source code, execute the command ``go build`` in the directory ``chubaofs/cli`` to generate ``cli``.
After downloading the CubeFS source code, execute the command ``go build`` in the directory ``chubaofs/cli`` to generate ``cli``.
At the same time, a configuration file named ``.cfs-cli.json`` will be generated in the directory ``root``, and the master address can be changed to the current cluster master address. You can also get or set the master address by executing the command ``./cli config info`` or ``./cli config set``.

View File

@ -10,7 +10,7 @@ Create
Create a user in the cluster to access object storage service. When the cluster starts, the ``root`` user is automatically created (the value of ``type`` is ``0x1``).
ChubaoFS treats every ``Owner`` of volume as a ``user``. For example, if the value of **Owner** is ``testuser`` when creating a volume, the volume is owned by user ``testuser``.
CubeFS treats every ``Owner`` of volume as a ``user``. For example, if the value of **Owner** is ``testuser`` when creating a volume, the volume is owned by user ``testuser``.
If there is no user ID with the same value as the **Owner** when creating the volume, the user named the value of **Owner** will be automatically created when creating the volume.
@ -18,7 +18,7 @@ If there is no user ID with the same value as the **Owner** when creating the vo
:header: "Key", "Type", "Description", "Range", "Mandatory", "Default"
"id", "string", "user ID", "Consists of letters, numbers and underscores, no more than 20 characters", "Yes", "None"
"pwd", "string", "user's password", "Unlimited", "No", "``ChubaoFSUser``"
"pwd", "string", "user's password", "Unlimited", "No", "``CubeFSUser``"
"ak", "string", "Access Key", "Consists of 16-bits letters and numbers", "No", "Random value"
"sk", "string","Secret Key", "Consists of 32-bits letters and numbers", "No", "Random value"
"type", "int", "user type", "2: [admin] / 3: [normal user]", "Yes", "None"

View File

@ -11,7 +11,7 @@ Create
| Allocate a set of data partition and a meta partition to the user.
| Default create 10 data partition and 3 meta partition when create volume.
| ChubaoFS uses the **Owner** parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as **Owner** will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: `/admin-api/master/user`
| CubeFS uses the **Owner** parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as **Owner** will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: `/admin-api/master/user`
.. csv-table:: Parameters
:header: "Parameter", "Type", "Description", "Mandatory", "Default"

View File

@ -20,7 +20,7 @@
# -- Project information -----------------------------------------------------
project = u'ChubaoFS'
project = u'CubeFS'
copyright = u'2018, The Chubao Authors'
author = u'The Chubao Authors'
@ -104,7 +104,7 @@ html_theme = 'sphinx_rtd_theme'
# Output file base name for HTML help builder.
htmlhelp_basename = 'ChubaoFS Doc'
htmlhelp_basename = 'CubeFS Doc'
@ -113,7 +113,7 @@ htmlhelp_basename = 'ChubaoFS Doc'
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'ChubaoFS', u'ChubaoFS Documentation',
(master_doc, 'CubeFS', u'CubeFS Documentation',
[author], 1)
]
@ -124,8 +124,8 @@ man_pages = [
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'ChubaoFS', u'ChubaoFS Documentation',
author, 'ChubaoFS', 'One line description of project.',
(master_doc, 'CubeFS', u'CubeFS Documentation',
author, 'CubeFS', 'One line description of project.',
'Miscellaneous'),
]

View File

@ -1,4 +1,4 @@
Contributing to ChubaoFS
Contributing to CubeFS
========================
Bug Reports

View File

@ -1,7 +1,7 @@
Authnode
=========
Internet and Intranet are insecure places where malicious hackers usually use tools to "sniff" sensitive information off of the network. Worse yet, client/server are not trusted to be honest about their identities. Therefore, ChubaoFS encounters some common security problems once deployed in the Network.
Internet and Intranet are insecure places where malicious hackers usually use tools to "sniff" sensitive information off of the network. Worse yet, client/server are not trusted to be honest about their identities. Therefore, CubeFS encounters some common security problems once deployed in the Network.
Security Problems
------------------
@ -12,7 +12,7 @@ Security Problems
High Level Architecture
-----------------------
`Authnode` is the security node providing a general Authentication & Authorization framework for `ChubaoFS`. Besides, `Authnode` acts as a centralized key store of both symmetric and asymmetric key. `Authnode` adopts and customizes the idea of authentication from `Kerberos` which is built on top of tickets. Specifically, whenever a client node (`Master`, `Meta`, `Data` or `Client` node) accesses a service, it's firstly required to show the shared secret key for authentication in `Authnode`. If successful, `Authnode` would issue a time-limited ticket specifically for that service. With the purpose of authorization, capabilities are embedded in tickets to indicate `who can do what on what resource`.
`Authnode` is the security node providing a general Authentication & Authorization framework for `CubeFS`. Besides, `Authnode` acts as a centralized key store of both symmetric and asymmetric key. `Authnode` adopts and customizes the idea of authentication from `Kerberos` which is built on top of tickets. Specifically, whenever a client node (`Master`, `Meta`, `Data` or `Client` node) accesses a service, it's firstly required to show the shared secret key for authentication in `Authnode`. If successful, `Authnode` would issue a time-limited ticket specifically for that service. With the purpose of authorization, capabilities are embedded in tickets to indicate `who can do what on what resource`.
.. image:: pic/authflow.png
:align: center
@ -20,7 +20,7 @@ High Level Architecture
:alt: Architecture
In the context of `Authnode`, we define a node as `Client` if it is responsible to initialize a service request while `Server` or `Service` is defined as the node responding that request. In this case, any `ChubaoFS` nodes can be acted as either `Client` or `Server`.
In the context of `Authnode`, we define a node as `Client` if it is responsible to initialize a service request while `Server` or `Service` is defined as the node responding that request. In this case, any `CubeFS` nodes can be acted as either `Client` or `Server`.
The communication between `Client` and `Server` is based on `HTTPS` or `TCP` and the workflow of `Authnode` is depicted in the graph above and briefly described as follows:
@ -61,7 +61,7 @@ If a service request is sent via TCP protocol, it has the following steps:
Future Work
-----------
`Authnode` supports a general authentication and authorization which is an emergent need for `ChubaoFS`. There are two directions of security enhancements for `ChubaoFS` in the future.
`Authnode` supports a general authentication and authorization which is an emergent need for `CubeFS`. There are two directions of security enhancements for `ChubaoFS` in the future.
Feature Enrichment
@ -71,7 +71,7 @@ Current implementation of `Authnode` doesn't support some advanced features:
- Key rotation: Shared secret keys are hardcoded in client and server and would not changed. It increases security risks that attacks break the encryption and find the keys. Rotating keys on a regular basis would help to mitigate such risks.
- Ticket revocation: For performance considerations, ticket would be valid for a while (such as several hours). If a client unfortunately leaks its ticket, malicious parties are able to use the ticket for service request during the period of being valid. Ticket revocation mechanism can prevent such an issue by revoking it once leakage happens.
- HSM support: `Authnode` is the security bottleneck in `ChubaoFS`. Breaking `Authnode` means compromising the whole system since it manages the key store. Hardware Security Module or HSM provides physical safeguards for key management. Having `Authnode` protected by HSM (for example *SGX*) can mitigate the risk of `Authnode` being compromised.
- HSM support: `Authnode` is the security bottleneck in `CubeFS`. Breaking `Authnode` means compromising the whole system since it manages the key store. Hardware Security Module or HSM provides physical safeguards for key management. Having `Authnode` protected by HSM (for example *SGX*) can mitigate the risk of `Authnode` being compromised.
End-to-End Data Encryption

View File

@ -16,9 +16,9 @@ To reduce the communication with the data nodes, the client caches the most rec
Integration with FUSE
-----------------------
The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.
The CubeFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.
Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel's page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as "dirty" and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.
Currently the write-back cache feature does not work well in CubeFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel's page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as "dirty" and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.
Live Upgrade
-----------------------

View File

@ -17,13 +17,13 @@ Writing a new file to the extent store always causes the data to be written at t
- Small File Storage
The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, \textit{fallocate()}\footnote{\url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to \textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~\cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.
The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. CubeFS relies on the punch hole interface, \textit{fallocate()}\footnote{\url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to \textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~\cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.
- Replication
The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.
The replication is performed in terms of partitions during file writes. Depending on the file write pattern, CubeFS adopts different replication strategies.
When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.
When a file is sequentially written into CubeFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.
.. image:: ../pic/workflow-sequential-write.png
:align: center

View File

@ -1,7 +1,7 @@
Resource Manager (Master)
=========================
The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.
The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the CubeFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.
Utilization-Based Distribution/Placement
-----------------------------------------

View File

@ -2,8 +2,8 @@ Object Subsystem (ObjectNode)
=============================
The object subsystem provides S3-compatible object storage interface.
It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible).
So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.
It makes possible for CubeFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible).
So that user can operate files stored in CubeFS by using native Amazon S3 SDKs.
Structure
---------
@ -14,7 +14,7 @@ Structure
The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master).
Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the CubeFS cluster
which it belong to directly without any volume-mount operation.
Features
@ -47,7 +47,7 @@ User
Before using the object storage service, you need to create a user through the Master.
While creating users, *AccessKey* and *SecretKey* will be generated for each user.
ChubaoFS uses the field **Owner** of the volume as the user ID. There are two ways to create users:
CubeFS uses the field **Owner** of the volume as the user ID. There are two ways to create users:
- When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.
- Create a user by calling the user management API of the Master.
@ -56,7 +56,7 @@ Authentication
--------------
The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service.
The authentication consisting of *AccessKey* and *SecretKey* generated by Resource Manager (Master) with user creation, which can be obtained through the Master API.
The *AccessKey* is a 16-character string unique in the entire ChubaoFS cluster.
The *AccessKey* is a 16-character string unique in the entire CubeFS cluster.
The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:
@ -64,7 +64,7 @@ The user has all access permissions to the volume owned by him. Users can grant
- Permission for a single operation, such as *GetObject*, *PutObject*, etc.
- Custom permission.
When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.
When a user uses the object storage service to execute a certain operation, CubeFS will identify whether the user has the corresponding permission.
Invisible Temporary Data
-------------------------
@ -78,9 +78,9 @@ Object Mode Conflict (Important)
The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service.
So that the object with key '*a/b/c*' and the object with key '*a/b*' are different object without any conflict.
But ChubaoFS is based on POSIX design. According to semantic transformation rule, the '*b*' part in key '*a/b/c*' transform to folder '*b*' under the folder '*a*' , and in key '*a/b*' transform to file '*b*' under the folder '*a*'.
But CubeFS is based on POSIX design. According to semantic transformation rule, the '*b*' part in key '*a/b/c*' transform to folder '*b*' under the folder '*a*' , and in key '*a/b*' transform to file '*b*' under the folder '*a*'.
The object key like this is conflict in ChubaoFS.
The object key like this is conflict in CubeFS.
When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a ``409`` status code to the client.

View File

@ -34,7 +34,7 @@ Multi-Zone Deploy
-----------------------
If you want the cluster to support fault tolerance in the computer room, you can deploy a ChubaoFS cluster across computer rooms. At the same time, it should be noted that since the communication delay between computer rooms is higher than that of a single computer room, if the requirements for high availability are greater than low latency, you can choose a cross-computer room deployment solution. If you have higher performance requirements, it is recommended to deploy clusters in a single computer room.
If you want the cluster to support fault tolerance in the computer room, you can deploy a CubeFS cluster across computer rooms. At the same time, it should be noted that since the communication delay between computer rooms is higher than that of a single computer room, if the requirements for high availability are greater than low latency, you can choose a cross-computer room deployment solution. If you have higher performance requirements, it is recommended to deploy clusters in a single computer room.
Configuration scheme: Modify the zoneName parameter in the DataNode/MetaNode configuration file, specify the name of the computer room where you are, and then start the DataNode/MetaNode process, the computer room will be stored and recorded by the Master along with the registration of DataNode/MetaNode.

View File

@ -60,7 +60,7 @@ Small file operation performance and scalability benchmark test by mdtest_.
#!/bin/bash
set -e
TARGET_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume
TARGET_PATH="/mnt/test/mdtest" # mount point of CubeFS volume
for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size
do
mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET_PATH;
@ -303,7 +303,7 @@ Metadata performance and scalability benchmark test by mdtest_.
.. code-block:: bash
#!/bin/bash
TEST_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume
TEST_PATH=/mnt/cfs/mdtest # mount point of CubeFS volume
for CLIENTS in 1 2 4 8 # number of clients
do
mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST_PATH;

View File

@ -1,10 +1,10 @@
Q&A
==============
- If you are new to ChubaoFS and want to start quickly, please refer to :doc:`user-guide/docker`
- If you are interested in ChubaoFS and want to conduct a performance test before applying it in the product environment, please refer to :doc:`evaluation`
- If you have completed the assessment of ChubaoFS and want to put it into product environment, and want to learn how to carry out capacity planning and environment preparing, please refer to :doc:`env`
- If you want to know about best practices of ChubaoFS, please refer to :doc:`use-case`
- If you are new to CubeFS and want to start quickly, please refer to :doc:`user-guide/docker`
- If you are interested in CubeFS and want to conduct a performance test before applying it in the product environment, please refer to :doc:`evaluation`
- If you have completed the assessment of CubeFS and want to put it into product environment, and want to learn how to carry out capacity planning and environment preparing, please refer to :doc:`env`
- If you want to know about best practices of CubeFS, please refer to :doc:`use-case`
- If you encounter some problems in the product environment, the following content may help to solve your problems.
For the convenience of description, we define the following keyword abbreviations
@ -23,7 +23,7 @@ For the convenience of description, we define the following keyword abbreviation
Compile
----------
1. Compile ChubaoFS on one machine, but it cannot be started when deployed to other machines.
1. Compile CubeFS on one machine, but it cannot be started when deployed to other machines.
First please make sure to use the ``PORTABLE=1 make static_lib`` command to compile RocksDB, then use the ``ldd`` command to check whether the dependent libraries are installed on the machine. After installing the missing libraries, execute the ``ldconfig`` command.
@ -120,7 +120,7 @@ It is recommended set `reserved space` in the dn startup json file, it is behind
Performance of Data and Metadata
----------------------------------
1. How does ChubaoFS compare with its alternatives ?
1. How does CubeFS compare with its alternatives ?
- Ceph
@ -130,26 +130,26 @@ Ceph (e.g., various storage backends) also make it very complicated and difficul
- GFS and HDFS
GFS and its open source implementation HDFS (https://github.com/apache/hadoop) are designed for storing large files with sequential access.
Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.
Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, CubeFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.
- Hadoop Ozone
Hadoop Ozone is a scalable distributed object storage system designed for Hadoop. It was originally proposed to solve the problem of HDFS namenode expansion. It reconstructs the namenode metadata management part of hdfs and reuses the datanode of hdfs.
ChubaoFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS's POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.
CubeFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS's POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.
- Haystack
Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata.
ChubaoFS adopts similar ideas by putting the file metadata into the main memory.
CubeFS adopts similar ideas by putting the file metadata into the main memory.
However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory,
and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer.
ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.
CubeFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.
- Public Cloud
Windows Azure Storage (https://azure.microsoft.com/en-us/) is a cloud storage system that provides strong consistency and multi-tenancy to the clients.
Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level.
Different from CubeFS, it builds an extra partition layer to handle random writes before streaming data into the lower level.
AWS EFS (https://aws.amazon.com/efs/) is a cloud storage service that provides scalable and elastic file storage.
Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.
@ -159,7 +159,7 @@ GlusterFS (https://www.gluster.org/) is a scalable distributed file system that
2. If the scale of metadata is huge, how to improve cluster performance?
The metadata of ChubaoFS is stored in the memory. Expanding memory of the mn or expanding the amount of mn horizontally will significantly improve the metadata performance and support a large number of small files.
The metadata of CubeFS is stored in the memory. Expanding memory of the mn or expanding the amount of mn horizontally will significantly improve the metadata performance and support a large number of small files.
3. If a dn/mn is added to the cluster, will it be automatically rebalanced, for example the dp/mp on the old node are migrated to the new node?
@ -197,7 +197,7 @@ Capacity Management
2. How to optimize the read/write performance from the Volume side
The more dp that can be read and written, the better read and write performance of the Volume. ChubaoFS adopts a dynamic space allocation mechanism. After creating a Volume, it will pre-allocate a certain data partition dp for the Volume. When the number of dp that can be read and written is less than 10, the dp number will be automatically expanded by a step of 10. If you want to manually increase the number of readable and writable dp, you can use the following command:
The more dp that can be read and written, the better read and write performance of the Volume. CubeFS adopts a dynamic space allocation mechanism. After creating a Volume, it will pre-allocate a certain data partition dp for the Volume. When the number of dp that can be read and written is less than 10, the dp number will be automatically expanded by a step of 10. If you want to manually increase the number of readable and writable dp, you can use the following command:
.. code-block:: bash
@ -240,7 +240,7 @@ Most of the parameters in the cluster have default values, and the default zone
3. The meaning of NodeSet
Each zone will have several NodeSets, and each NodeSet can carry 18 nodes by default. Because ChubaoFS implements multi-raft, each node starts a raft server process, and each raft server manages m raft instances on the node. If the other replication group members of these raft members are distributed on N nodes, then the raft heartbeat will be transmitted between N nodes. As the cluster scale expands, N will become larger. Through the NodeSet restriction, the heartbeat is relatively independent within the NodeSet, which avoids the heartbeat storm of the cluster dimension. We use the multi raft and nodeset mechanisms together to avoid the problem of the raft heartbeat storm.
Each zone will have several NodeSets, and each NodeSet can carry 18 nodes by default. Because CubeFS implements multi-raft, each node starts a raft server process, and each raft server manages m raft instances on the node. If the other replication group members of these raft members are distributed on N nodes, then the raft heartbeat will be transmitted between N nodes. As the cluster scale expands, N will become larger. Through the NodeSet restriction, the heartbeat is relatively independent within the NodeSet, which avoids the heartbeat storm of the cluster dimension. We use the multi raft and nodeset mechanisms together to avoid the problem of the raft heartbeat storm.
.. image:: pic/nodeset.png
:align: center
@ -307,7 +307,7 @@ Upgrade
1. Steps
a. Download and unzip the latest binary file compression package from ChubaoFS official website https://github.com/cubefs/cubefs/releases
a. Download and unzip the latest binary file compression package from CubeFS official website https://github.com/cubefs/cubefs/releases
b. Freeze the cluster
.. code-block:: bash
@ -592,9 +592,9 @@ Fuse Client
- Because the client reads and writes files through the http protocol, please check whether the network status is healthy
- Check whether there is an overloaded mn, whether the mn process is hanging, you can restart mn, or expand a new mn to the cluster and take the mp on the overloaded mn decommission to relieve the pressure of mn
4. Does ChubaoFS provide strong consistence guarantees?
4. Does CubeFS provide strong consistence guarantees?
No.ChubaoFS has relaxed POSIX consistency semantics, i.e., instead of providing strong consistency guarantees, it only ensures sequential consistency for file/directory operations, and does not have any leasing mechanism to prevent multiple clients writing to the same file/directory. It depends on the upperlevel application to maintain a more restrict consistency level if necessary.
No.CubeFS has relaxed POSIX consistency semantics, i.e., instead of providing strong consistency guarantees, it only ensures sequential consistency for file/directory operations, and does not have any leasing mechanism to prevent multiple clients writing to the same file/directory. It depends on the upperlevel application to maintain a more restrict consistency level if necessary.
5. Is it feasible to kill the client to directly stop the client service

View File

@ -1,4 +1,4 @@
Welcome to ChubaoFS(Chubao File System)
Welcome to CubeFS(Chubao File System)
==============================================
.. toctree::

View File

@ -1,7 +1,7 @@
Introduction
=============
ChubaoFS(Chubao File System) is a distributed file system that is designed to natively support large scale container platforms.
CubeFS(Chubao File System) is a distributed file system that is designed to natively support large scale container platforms.
High Level Architecture
-----------------------
@ -11,16 +11,16 @@ High Level Architecture
:scale: 50 %
:alt: Architecture
ChubaoFS consists of a *metadata subsystem*, a *data subsystem*, and a *resource manager*, and can be accessed by different *clients* (as a set of application processes) hosted on the containers through different file system instances called *volumes*.
CubeFS consists of a *metadata subsystem*, a *data subsystem*, and a *resource manager*, and can be accessed by different *clients* (as a set of application processes) hosted on the containers through different file system instances called *volumes*.
The metadata subsystem stores the file metadata, and consists of a set of *meta nodes*. Each meta node consists of a set of *meta partitions*.
The data subsystem stores the file contents, and consists of a set of *data nodes*. Each data node consists of a set of *data partitions*.
The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume.
The volume is a logical concept in CubeFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume.
From a client's perspective, the volume can be viewed as a file system instance that contains data accessible by the containers.
A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation.
A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.
A CubeFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.
Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.
@ -49,7 +49,7 @@ E-commence venders who move their line of business applications to the cloud usu
Relaxed POSIX Semantics and Metadata Atomicity
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.
In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. CubeFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.

View File

@ -1,7 +1,7 @@
Use Cases
=============
ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.
CubeFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.
Machine Learning
--------------------
@ -12,7 +12,7 @@ Disadvantages of using a local disk to store training data sets:
- Training data sets need to be updated frequently, requiring more disk space.
- Risk of loss of training data set if machine crashes.
The advantages of using ChubaoFS to store training data sets:
The advantages of using CubeFS to store training data sets:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Multiple replicas of data to ensure high data reliability without worrying about losing data.
@ -27,7 +27,7 @@ Using local disks to store data often encounters the following problems:
- Disk usage is uneven and disk IO cannot be fully utilized.
- Local disk space is limited.
The advantages of using ChubaoFS as a backend storage:
The advantages of using CubeFS as a backend storage:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Disk IO usage is uniform and disk IO is fully utilized.
@ -37,32 +37,32 @@ The advantages of using ChubaoFS as a backend storage:
Nginx Log Storage
---------------------
Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space.
Do you often worry about running out of local disk space? With CubeFS, you can store datas in a distributed file system without worrying about running out of disk space.
If you use a local disk to store logs, you may often worry about the following issues:
- Docker local disk space is small.
- If the docker container crashes, the log is lost and unrecoverable.
- The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.
The advantages of using ChubaoFS to store nginx logs are:
The advantages of using CubeFS to store nginx logs are:
- The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.
- Ensure data is highly reliable and do not worry about losing data.
- Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.
- Compatible with the POSIX file system interface, no changes required by the application.
- The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.
- The operation and maintenance of CubeFS are simple, and one person can easily manage the cluster of tens of thousands of machines.
Spark
----------
In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation.
In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to CubeFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation.
The pain points of using local disk to store shuffle intermediate results:
- Insufficient disk space.
- Too many temporary directory files to create new files.
The advantages of using ChubaoFS:
The advantages of using CubeFS:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.
@ -78,7 +78,7 @@ Disadvantages of using OSS(Object Storage Service) to back up MySQL database:
- After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.
- Backup files are processed by multiple layers of services, which affects performance.
Advantages of using ChubaoFS to back up MySQL databases:
Advantages of using CubeFS to back up MySQL databases:
- Easy to use, compatible with POSIX file interface, and can be used as a local file system.
- Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.

Some files were not shown because too many files have changed in this diff Show More