gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 02:00:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

supplementary licence information and documents

Browse Source 
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
This commit is contained in:
Mofei Zhang 2019-12-19 18:33:55 +08:00
parent 22d3aca9a8
commit ebfe9e297c
No known key found for this signature in database
GPG Key ID: 7C19725C197AE672
31 changed files with 525 additions and 192 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
MAINTAINERS.md
View File

@ -12,5 +12,5 @@ The ChubaoFS maintainers are:
* Zhengyi Zhu <zhengyi.zhu.hust@gmail.com> @zhuzhengyi: Monitoring and ObjectNode
* Liying Zhang <zly7755@163.com> @Vivian7755: Product Management & Advocate
* Junyuan Zeng <jzeng04@gmail.com> @jzeng4: Authorization Node
* Xihao Xu<xxscott@163.com> @xxscott: CSI Driver
* Wenjia Wu<buaa1214wwj@126.com> @wenjia322: Authorization Node
* Xihao Xu <xxscott@163.com> @xxscott: CSI Driver
* Wenjia Wu <buaa1214wwj@126.com> @wenjia322: Authorization Node

16
docs/source/admin-api/master/volume.rst
View File

@ -56,12 +56,16 @@ response
.. code-block:: json
{
"Name": "test",
"VolType": "extent",
"MetaPartitions": {},
"DataPartitions": {}
}
{
"Name": "test",
"VolType": "extent",
"MetaPartitions": {},
"DataPartitions": {},
"OSSSecure": {
"AccessKey": "l9aC5S9OzlneHguZ",
"SecretKey": "4sL0xJfHVVHKOC3MDwPfaF1AcGm5a2XL"
}
}
Stat

127
docs/source/design/objectnode.rst Normal file
View File

@ -0,0 +1,127 @@
Object Subsystem (ObjectNode)
=============================
The object subsystem provides S3-compatible object storage interface.
It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible).
So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.
Structure
---------
.. image:: ../pic/cfs-object-subsystem-structure.png
:align: center
The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master).
Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster
which it belong to directly without any volume-mount operation.
Features
--------
- Provides object storage interface compatible with native Amazon S3 SDKs.
- Fusion storage expose two type interface (POSIX and S3-compatible).
- Stateless and high scalability
Semantic Transform
-------------------
Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.
.. csv-table::
:header: "POSIX", "Object Storage"
"Volume", "Bucket"
"File Path", "Key"
**Example:**
.. image:: ../pic/cfs-object-subsystem-semantic.png
:align: center
Put object '*example/a/b.txt*' will be create and write data to file '*/a/b.txt*' in volume '*example*'.
Authentication
--------------
The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service.
The authentication consisting of *AccessKey* and *SecretKey* generated by Resource Manager (Master) with volume creation.
The *AccessKey* is a 16-character string unique in the entire ChubaoFS cluster.
Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master).
User can fetch it by using administration API, see :doc:`/admin-api/master/volume`
Invisible Temporary Data
-------------------------
In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary.
The volume operator in ObjectNode puts file data into temporary which only have '**inode**' without '**dentry**' in metadata.
When all the file data stored successfully, the volume operator create or update '**dentry**' in metadata makes it visible to users.
Object Key Conflict (Important)
--------------------------------
The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service.
So that the object with key '*a/b/c*' and the object with key '*a/b*' are different object without any conflict.
But ChubaoFS is based on POSIX design. According to semantic transformation rule, the '*b*' part in key '*a/b/c*' transform to folder '*b*' under the folder '*a*' , and in key '*a/b*' transform to file '*b*' under the folder '*a*'.
The object key like this is conflict in ChuBaoFS.
Supported S3-Compatible APIs
----------------------------
Bucket APIs
^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"HeadBucket", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html"
"GetBucketLocation", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html"
Object APIs
^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"HeadObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html"
"PutObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html"
"GetObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html"
"ListObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html"
"ListObjectsV2", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html"
"DeleteObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html"
"DeleteObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html"
"CopyObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html"
Multipart Upload APIs
^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"CreateMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html"
"ListMultipartUploads", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html"
"AbortMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html"
"CompleteMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html"
"ListParts", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html"
"UploadPart", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html"
"UploadPartCopy", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html"
Supported SDKs
--------------
Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
.. csv-table::
:header: "Name", "Language", "Link"
"AWS SDK for Java", "Java", "https://aws.amazon.com/sdk-for-java/"
"AWS SDK for JavaScript", "JavaScript", "https://aws.amazon.com/sdk-for-browser/"
"AWS SDK for JavaScript inNode.js", "JavaScript", "https://aws.amazon.com/sdk-for-node-js/"
"AWS SDK for Go", "Go", "https://docs.aws.amazon.com/sdk-for-go/"
"AWS SDK for PHP", "PHP", "https://aws.amazon.com/sdk-for-php/"
"AWS SDK for Ruby", "Ruby", "https://aws.amazon.com/sdk-for-ruby/"
"AWS SDK for .NET", ".NET", "https://aws.amazon.com/sdk-for-net/"
"AWS SDK for C++", "C++", "https://aws.amazon.com/sdk-for-cpp/"
"Boto3", "Python", "https://aws.amazon.com/sdk-for-python/"

1
docs/source/index.rst
View File

@ -15,6 +15,7 @@ Welcome to ChubaoFS(Chubao File System)
design/master
design/metanode
design/datanode
design/objectnode.rst
design/client
.. toctree::

BIN
docs/source/pic/cfs-object-subsystem-semantic.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
docs/source/pic/cfs-object-subsystem-structure.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

30
docs/source/quick-start-guide.rst
View File

@ -149,6 +149,36 @@ Start Datanode
For detailed explanations of *datanode.json*, please refer to :doc:`user-guide/datanode`.
Start ObjectNode
^^^^^^^^^^^^^^^^
.. code-block:: bash
nohup ./cfs-server -c objectnode.json &
Sample *objectnode.json is* shown as follows,
.. code-block:: json
{
"role": "objectnode",
"domains": [
"object.cfs.local"
],
"listen": 80,
"masters": [
"master1.cfs.local:80",
"master2.cfs.local:80",
"master3.cfs.local:80"
],
"logLevel": "info",
"logDir": "/export/Logs/objectnode",
"region": "cn_bj"
}
For detailed explanations of *meta.json*, please refer to :doc:`user-guide/objectnode`.
Create Volume
^^^^^^^^^^^^^

76
docs/source/user-guide/objectnode.rst
View File

@ -1,20 +1,24 @@
Object Storage Service
Object Subsystem (ObjectNode)
==============================
How To Provide Object Storage Service with ObjectNode
-----------------------
How To Provide Object Storage Service with Object Subsystem (ObjectNode)
-------------------------------------------------------------------------
Start a ObjectNode process by execute the server binary of ChubaoFS you built with ``-c`` argument and specify configuration file.
.. code-block:: bash
nohup cfs-server -c s3gateway.json &
nohup cfs-server -c objectnode.json &
Configurations
-----------------------
Object Node using `JSON` format configuration file.
.. csv-table:: Properties
**Properties**
.. csv-table::
:header: "Key", "Type", "Description", "Mandatory"
"role", "string", "Role of process and must be set to *objectnode*", "Yes"
@ -55,3 +59,65 @@ Configurations
"prof": "7013"
}
Fetch Authentication Keys
----------------------------
Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master).
User can fetch it by using administration API, see :doc:`/admin-api/master/volume`
Supported S3-Compatible APIs
----------------------------
Bucket APIs
^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"HeadBucket", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html"
"GetBucketLocation", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html"
Object APIs
^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"HeadObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html"
"PutObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html"
"GetObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html"
"ListObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html"
"ListObjectsV2", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html"
"DeleteObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html"
"DeleteObjects", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html"
"CopyObject", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html"
Multipart Upload APIs
^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:header: "Name", "API Reference"
"CreateMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html"
"ListMultipartUploads", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html"
"AbortMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html"
"CompleteMultipartUpload", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html"
"ListParts", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html"
"UploadPart", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html"
"UploadPartCopy", "https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html"
Supported SDKs
--------------
Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
.. csv-table::
:header: "Name", "Language", "Link"
"AWS SDK for Java", "Java", "https://aws.amazon.com/sdk-for-java/"
"AWS SDK for JavaScript", "JavaScript", "https://aws.amazon.com/sdk-for-browser/"
"AWS SDK for JavaScript inNode.js", "JavaScript", "https://aws.amazon.com/sdk-for-node-js/"
"AWS SDK for Go", "Go", "https://docs.aws.amazon.com/sdk-for-go/"
"AWS SDK for PHP", "PHP", "https://aws.amazon.com/sdk-for-php/"
"AWS SDK for Ruby", "Ruby", "https://aws.amazon.com/sdk-for-ruby/"
"AWS SDK for .NET", ".NET", "https://aws.amazon.com/sdk-for-net/"
"AWS SDK for C++", "C++", "https://aws.amazon.com/sdk-for-cpp/"
"Boto3", "Python", "https://aws.amazon.com/sdk-for-python/"

14
objectnode/acl.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
// https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html

14
objectnode/acl_handler.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
// https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/dev/acl-using-rest-api.html

15
objectnode/acl_test.go
View File

@ -1,2 +1,15 @@
// Package s3 provides ...
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode

15
objectnode/api_handler_bucket.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (
@ -22,7 +36,6 @@ func (o *ObjectNode) listBucketsHandler(w http.ResponseWriter, r *http.Request)
// API reference: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html
func (o *ObjectNode) getBucketLocation(w http.ResponseWriter, r *http.Request) {
log.LogInfof("getBucketLocation: get bucket location: requestID(%v)", RequestIDFromRequest(r))
// TODO: implement 'getBucketLocation'
var output = &GetBucketLocationOutput{
LocationConstraint: o.region,
}

15
objectnode/api_middleware.go
View File

@ -1,4 +1,17 @@
// Package s3 provides ...
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

15
objectnode/auth_signature_v2.go
View File

@ -1,4 +1,17 @@
// Package s3 provides ...
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/auth_signature_v4.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/auth_signature_v4_test.go
View File

@ -1 +1,15 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode

14
objectnode/fs.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/fs_manager.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/fs_store.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
type MetaStore interface {

14
objectnode/fs_store_authnode.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
type authnodeStore struct {

14
objectnode/fs_store_object.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/fs_store_xattr.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

81
objectnode/fs_stream.go
View File

@ -1,81 +0,0 @@
package objectnode
import (
"context"
"io"
"sync"
)
type Stream struct {
mu *sync.Mutex
cond *sync.Cond
buffer []byte
closed bool
ctx context.Context
}
func (s *Stream) Write(p []byte) (n int, err error) {
s.mu.Lock()
defer s.mu.Unlock()
select {
case <-s.ctx.Done():
return 0, s.ctx.Err()
default:
}
if s.closed {
return 0, io.EOF
}
s.buffer = append(s.buffer, p...)
s.cond.L.Lock()
s.cond.Broadcast()
s.cond.L.Unlock()
return len(p), nil
}
func (s *Stream) Close() error {
s.mu.Lock()
defer s.mu.Unlock()
s.closed = true
s.cond.L.Lock()
s.cond.Broadcast()
s.cond.L.Unlock()
return nil
}
func (s *Stream) Read(p []byte) (n int, err error) {
s.mu.Lock()
defer s.mu.Unlock()
select {
case <-s.ctx.Done():
return 0, s.ctx.Err()
default:
}
if len(s.buffer) == 0 && s.closed {
return 0, io.EOF
}
if len(s.buffer) == 0 {
s.mu.Unlock()
s.cond.L.Lock()
s.cond.Wait()
s.cond.L.Unlock()
s.mu.Lock()
if len(s.buffer) == 0 && s.closed {
return 0, io.EOF
}
}
var readN = len(p)
if len(s.buffer) < readN {
readN = len(s.buffer)
}
copy(p, s.buffer[:readN])
s.buffer = s.buffer[readN:]
return readN, nil
}
func NewStream(ctx context.Context) *Stream {
stream := &Stream{}
stream.mu = new(sync.Mutex)
stream.cond = sync.NewCond(new(sync.Mutex))
stream.ctx = ctx
return stream
}

88
objectnode/fs_stream_test.go
View File

@ -1,88 +0,0 @@
package objectnode
import (
"context"
"io"
"reflect"
"sync"
"testing"
"time"
)
func TestNewStream_modify(t *testing.T) {
ctx, _ := context.WithCancel(context.Background())
stream := NewStream(ctx)
var src = []byte("1234567890")
// write to stream
var err error
if _, err = stream.Write(src); err != nil {
t.Fatalf("write fail casue: %v", err)
}
_ = stream.Close()
// modify source
for i := 0; i < len(src); i++ {
src[i] = 0
}
// read from stream
var actual = make([]byte, len(src))
if _, err = stream.Read(actual); err != nil && err != io.EOF {
t.Fatalf("read fail cause: %v", err)
}
if string(actual) != "1234567890" {
t.Fatalf("result mismatch")
}
}
func TestStream_rw(t *testing.T) {
ctx, _ := context.WithCancel(context.Background())
stream := NewStream(ctx)
var src = []byte("1234567890")
asyncWG := new(sync.WaitGroup)
// async write
asyncWG.Add(1)
go func() {
defer asyncWG.Done()
var err error
var n int
for _, b := range src {
n, err = stream.Write([]byte{b})
if err != nil {
t.Fatalf("write fail cause: %v", err)
}
if n != 1 {
t.Fatalf("write bytes count mismatch: expect 10 actual %v", n)
}
time.Sleep(100 * time.Millisecond)
}
_ = stream.Close()
}()
// async read
asyncWG.Add(1)
go func() {
defer asyncWG.Done()
var err error
var n int
tmp := make([]byte, 3)
result := make([]byte, 0)
for {
n, err = stream.Read(tmp)
if err != nil && err != io.EOF {
t.Fatalf("read fail cause: %v", err)
}
if err == io.EOF {
break
}
t.Logf("read: %v", tmp[:n])
result = append(result, tmp[:n]...)
}
if !reflect.DeepEqual(result, src) {
t.Fatalf("result mismatch: expect %v actual %v", src, result)
}
}()
asyncWG.Wait()
}

8
objectnode/fs_volume.go
View File

@ -1269,7 +1269,7 @@ func (v *volume) ListMultipartUploads(prefix, delimiter, keyMarker string, multi
return uploads, NextMarker, NextSessionIdMarker, IsTruncated, prefixes, nil
}
func (v *volume) ListParts(path, sessionId string, maxParts, partNumberMarker uint64) (parts []*FSPart, nextMarker uint64, IsTruncated bool, err error) {
func (v *volume) ListParts(path, sessionId string, maxParts, partNumberMarker uint64) (parts []*FSPart, nextMarker uint64, isTruncated bool, err error) {
var parentId uint64
dirs, _ := splitPath(path)
// process path
@ -1284,10 +1284,10 @@ func (v *volume) ListParts(path, sessionId string, maxParts, partNumberMarker ui
sessionParts := multipartInfo.Parts
resLength := maxParts
IsTruncated = true
isTruncated = true
if (uint64(len(sessionParts)) - partNumberMarker) < maxParts {
resLength = uint64(len(sessionParts)) - partNumberMarker
IsTruncated = false
isTruncated = false
nextMarker = 0
} else {
nextMarker = partNumberMarker + resLength
@ -1304,7 +1304,7 @@ func (v *volume) ListParts(path, sessionId string, maxParts, partNumberMarker ui
parts = append(parts, fsPart)
}
return parts, nextMarker, IsTruncated, nil
return parts, nextMarker, isTruncated, nil
}
func (v *volume) CopyFile(targetPath, sourcePath string) (info *FSFileInfo, err error) {

14
objectnode/fs_volume_test.go
View File

@ -1 +1,15 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode

15
objectnode/policy_handler_test.go
View File

@ -1,2 +1,15 @@
// Package s3 provides ...
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode

15
objectnode/policy_test.go
View File

@ -1,4 +1,17 @@
// Package s3 provides ...
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
/*

14
objectnode/result.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/result_test.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (

14
objectnode/util.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The ChubaoFS Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package objectnode
import (