Commit Graph

48 Commits

Author SHA1 Message Date
chihe
347208f189 refactor(clientmetanode): hybrid cloud check inode/vol storage class when checking vol type
Signed-off-by: chihe <chi.he@oppo.com>
2024-12-26 09:24:22 +08:00
shuqiang-zheng
3f1e45a22a fix(security): slice memory allocation with excessive size value.
Signed-off-by: shuqiang-zheng <zhengshuqiang@oppo.com>
2024-08-12 14:03:06 +08:00
Victor1319
4df91ebc6e refactor(all): reslove merge conflicts between 3.3.x and 3.4.0.
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2024-07-18 16:57:38 +08:00
yhjiango
3797f6b0e8 fix(object): fix request parts check in complete multipart
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-07-18 16:57:38 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
2024-05-14 09:54:13 +08:00
yhjiango
f7e7235694 fix(object): fix request parts check in complete multipart
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-03-13 15:54:54 +08:00
yhjiango
234d30ec11 fix(object): partNumber must not be greater than 10000
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-03-06 15:42:27 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt
Signed-off-by: slasher <shenjie1@oppo.com>
2024-01-12 14:21:16 +08:00
yhjiango
abdb0f2d9d enhance(object): add time-consuming tracking in object access
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-11-22 15:15:06 +08:00
yhjiango
f1cfca0be5 fix(object): partNumber of multipart must be greater than 0
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-11-15 15:18:47 +08:00
tangdeyi
dd46c24873 fix(objectnode): fix read body entirely into memory and without an upper boundary
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-10-20 17:06:26 +08:00
tangdeyi
14f0f67c73 feat(objnode): s3 api qos
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-31 16:36:14 +08:00
yhjiango
4401624f96 feat(objectnode): s3 sts and signature auth
1. sts federation token
2. signature auth code refactor

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
d11e9142a2 feat(objectnode): object lock
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-30 20:05:32 +08:00
yhjiango
13dfb462f3 fix(objectnode): save the ACL of the object uploaded by the non-volume owner
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-09 14:01:52 +08:00
tangdeyi
7613a0fadb refactor(objectnode): refactor some redundant code
Signed-off-by: tangdeyi <deyitang94@gmail.com>
2023-08-09 14:01:52 +08:00
yhjiango
7339d6a6d7 feat(objectnode): reduce acl meta storage space usage
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-09 14:01:52 +08:00
tangdeyi
0bb8a4b6ea feat(objectnode): read/write cold volume buffer pool
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-04 18:08:38 +08:00
tangdeyi
b036d1fa7c fix(objnode): fix code scan error
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-04 18:08:38 +08:00
tangdeyi
1f6c3d6ddc fix(objectnode): uploadPart compares the Content-Md5 with the body checksum
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-06-06 19:22:07 +08:00
tangdeyi
b27a7c4ba0 fix(objectnode): listobject bugfix
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
1434374d51 fix(metanode): The same partInode must not be overwritten
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
2b6b00126a fix(objectnode): error adjustment in complete multipart
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
c5b008ebc8 fix(objectnode): fix the md5 mismatch between the put and get of the multipart object
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
0a4830dd93 fix(objectnode): fix the list multipart uploads and dir object inaccessible
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
a56250817d fix(objectnode): fix the volume load call and error return
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
862fde677a fix(objectnode):
1. MultipartUpload with nonexistent uploadId return NoSuchUploadID
2. NoSuchBucket err handle

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
0affc88413 fix(objectnode): fully compatible with s3
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
831ea00d5d fix(objectnode): enhance s3 compatibility
1.CORS supports preflight request;
2.S3-compatible err reply;
3.Small parts upload bugfix;
4.Specific status code when panic;
5.UploadPartCopy;

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
Patrick Wu
9a18277836 fix: do buffer write without buffer pool when copying object in objectnode
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
c8bf55ae69 S3 bugfix
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
leonrayang
9a30b9f884 update: update to CubeFS
Signed-off-by: leonrayang <chl696@sina.com>
2022-08-22 18:16:09 +08:00
leonrayang
2970e2c3d1 update. go import path change from chubaofs to cubefs
Signed-off-by: leonrayang <chl696@sina.com>
2022-01-25 18:17:02 +08:00
Mofei Zhang
dd45cd8ac5
Enhancement: error handling of ObjectNode
- Response Http400 after got unexpected EOF error when reading data from
  request connection.
- Fix ldflag issue from 'cli/build.sh'.
- Do not write failure status code if request connect closed by peer.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-08-06 12:12:50 +08:00
yuboLee
061a02cd12 handle ump warnning and error log info
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-08-03 10:53:57 +08:00
yuboLee
2775a5e1cc Enhancement: add support for modifying part system metadata, self defined metadata management and tagging management
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-06-11 08:48:53 +08:00
yuboLee
41ae6707a5 Fix: fix the bug of listing multipart uploads, and add check for request part info
Enhancement: add test case for listMultipartUploads and copyObject

Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-06-01 17:56:15 +08:00
Mofei Zhang
df4b7f9a42
Fix: parallel-safe issue in conn pool and metanode conn serve
1. Fix: parallel-safe issue in conn pool and metanode conn serve
2. Fix: nil pointer issue when list parts for non-exist uploadId
3. Enhancement: validate read size while read from connnect.
4. Enhancement: clean up useless data on failure while upload part.
5. Enhancement: generate ETag if do not exist when listing.
6. Enhancement: include error message when formatting packet to string.
7. Enhancement: improve compatile with old version SDK.
8. Enhancement:  success when retry add exist multipart part.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Signed-off-by: 张墨飞 <zhangmofei@jd.com>
2020-05-21 12:28:18 +08:00
Mofei Zhang
323e529ee0
Enhancement: make ETag versioned
When ETag is stored, it is versioned by encoding, which can make
the file system interface and object storage interface more closely
integrated.

Referring to the Amazon AWS S3 Developer Guide:
"Amazon S3 response includes an ETag that uniquely identifies the
combined object data. This ETag will not necessarily be an MD5 hash
of the object data."

When a new object is written to ChubaoFS through the object storage
interface, ChubaoFS calculates the ETag value for it, and uses the
modification time of the object as its version information when
encoding and storing.

If this file is modified by the file system interface, the ETag of
this file will be invalid because the version information will be
inconsistent with modification time.

After the object storage interface finds that the ETag is invalid,
it will automatically generate a new ETag matching its modification
time.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-05-05 12:56:00 +08:00
Mofei Zhang
c9bb74d718
Fix: xml element of DeleteResult
Fix xml element of DeleteResult.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-30 16:46:25 +08:00
Mofei Zhang
7a57837f74
Fix: third-party license information
Adjust the format of the license header information of the third-party
code used to make the format more compliant. The NOTICE file supplements
the third-party code NOTIC information and explains the use of the code.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-14 19:29:39 +08:00
Mervin
c1b2d740be
Merge branch 'master' into parse_part_no_marker 2020-03-31 17:07:02 +08:00
yuboLee
22521d5165 fix:parse param 'part-number-marker' failed when listing parts of one upload id.
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-03-31 16:18:43 +08:00
Mofei Zhang
509d96c3d1
Enhancement: folder operations in S3 APIs
Supports folder operations in the same way as Amazon AWS S3.
  - Get folder meta info by HeadObject interface.
  - Create folder when header specifies Content-Type as
    application/directory and Content-Length as 0 on PutObject interface.
  - Delete an empty folder by DeleteObject interface.
  - The GetObject interface responds normally when the target is an
    existing directory.
  - Differentiate folders and files on the ListObject interface through the
    CommonPrefix property (already supported).

Following object storage interface haven been implemented folder operations:
  - HeadObject
  - GetObject
  - DeleteObject
  - DeleteObjects
  - ListObjects
  - ListObjectsV2

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-30 17:01:45 +08:00
Mervin
fbbe1e0a7e
Refactor: param parse in object storage interface handlers
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 00:49:13 +08:00
Mofei Zhang
4979d728b2 Add: configuration for signature ignore actions
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-02-26 00:00:32 +08:00
Mofei Zhang
b4434486c9
fix: unit test building issues and log format pattern issues.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2019-12-20 18:19:01 +08:00
Mofei Zhang
d609fedb5c feature: object storage interface
Add ObjectNode provides S3-compatibile APIs.
Fusion Storage interface expose two interface (POSIX and S3-compatible) for file operation.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2019-12-17 17:36:53 +08:00