Commit Graph

60 Commits

Author SHA1 Message Date
zhumingze
b74b3fa74e fix(objectnode): fix incorrect statusCode when creating volume in objectnode. #1000146602
Signed-off-by: zhumingze <zhumingze@oppo.com>
2025-08-08 14:59:30 +08:00
tangdeyi
b7cf39a0c3 fix(objectnode): optimize the logic of createBucket
with #22863348

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2024-12-26 09:24:22 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
2024-05-14 09:54:13 +08:00
yhjiango
611ef3263e fix(object): fix tagging valid verification
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-03-13 15:54:54 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt
Signed-off-by: slasher <shenjie1@oppo.com>
2024-01-12 14:21:16 +08:00
tangdeyi
9388fc02b3 feat(objectnode): disable createBucket by s3
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-12-06 14:10:49 +08:00
tangdeyi
0d8662cc9a feat(objectnode): listbucket add authed volumes
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-12-06 14:10:49 +08:00
tangdeyi
b8e3732fa8 fix(object): fix create bucket err
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-11-01 10:33:45 +08:00
tangdeyi
dd46c24873 fix(objectnode): fix read body entirely into memory and without an upper boundary
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-10-20 17:06:26 +08:00
tangdeyi
14f0f67c73 feat(objnode): s3 api qos
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-31 16:36:14 +08:00
yhjiango
4401624f96 feat(objectnode): s3 sts and signature auth
1. sts federation token
2. signature auth code refactor

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
d11e9142a2 feat(objectnode): object lock
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
7613a0fadb refactor(objectnode): refactor some redundant code
Signed-off-by: tangdeyi <deyitang94@gmail.com>
2023-08-09 14:01:52 +08:00
yhjiango
7339d6a6d7 feat(objectnode): reduce acl meta storage space usage
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-09 14:01:52 +08:00
tangdeyi
0bb8a4b6ea feat(objectnode): read/write cold volume buffer pool
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-04 18:08:38 +08:00
tangdeyi
b036d1fa7c fix(objnode): fix code scan error
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-04 18:08:38 +08:00
yhjiango
2040ce00a9 fix(objectnode): fix bucket policy and location processing
1. Fix the problem of returning 500 internal errors in PutBucketPolicy.
2. Add related parameter verification in GetBucketLocation.

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-07-20 10:58:50 +08:00
yhjiango
a56250817d fix(objectnode): fix the volume load call and error return
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
0affc88413 fix(objectnode): fully compatible with s3
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
831ea00d5d fix(objectnode): enhance s3 compatibility
1.CORS supports preflight request;
2.S3-compatible err reply;
3.Small parts upload bugfix;
4.Specific status code when panic;
5.UploadPartCopy;

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
Patrick Wu
c8bf55ae69 S3 bugfix
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
leonrayang
9a30b9f884 update: update to CubeFS
Signed-off-by: leonrayang <chl696@sina.com>
2022-08-22 18:16:09 +08:00
leonrayang
2970e2c3d1 update. go import path change from chubaofs to cubefs
Signed-off-by: leonrayang <chl696@sina.com>
2022-01-25 18:17:02 +08:00
yuboLee
ce4f466f0f Fix: some bugs about object tagging and bucket tagging
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-07-10 12:17:40 +08:00
yuboLee
37b5aa3ede Fix: ListBuckets and headBucket result is empty using java sdk
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-07-07 15:37:33 +08:00
awzhgw
7754a60f2d
Merge pull request #570 from ZhuZhengyi/ob_fix
Fix: immediately return when duplicastedBucket error occur by create โ€ฆ
2020-05-21 16:41:15 +08:00
Mofei Zhang
df4b7f9a42
Fix: parallel-safe issue in conn pool and metanode conn serve
1. Fix: parallel-safe issue in conn pool and metanode conn serve
2. Fix: nil pointer issue when list parts for non-exist uploadId
3. Enhancement: validate read size while read from connnect.
4. Enhancement: clean up useless data on failure while upload part.
5. Enhancement: generate ETag if do not exist when listing.
6. Enhancement: include error message when formatting packet to string.
7. Enhancement: improve compatile with old version SDK.
8. Enhancement:  success when retry add exist multipart part.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Signed-off-by: ๅผ ๅขจ้ฃž <zhangmofei@jd.com>
2020-05-21 12:28:18 +08:00
zhuzhengyi1
4a14460510 Fix: immediately return when duplicastedBucket error occur by create Bucket Handler
Signed-off-by: zhuzhengyi1 <zhuzhengyi@jd.com>
2020-05-20 14:07:38 +08:00
Mofei Zhang
a4ec2c88d5
Enhancement: improve tagging storage and add tests
1. Add tests for tagging operations.
2. Support setup tagging when put an object.
3. Fix routing issue on DeleteObjectTagging interface.
4. Improve tagging encoding for storage.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-05-05 22:38:34 +08:00
Mofei Zhang
9e89d40ddf
Fix: xml response of get bucket location interface
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-05-03 20:36:45 +08:00
Mofei Zhang
0296e5709c
Fix: create time format of list buckets
Fix create time format of list buckets API to
"2006-01-02T15:04:05.000Z" as UTC time.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-25 22:04:05 +08:00
Mofei Zhang
5c7c9980eb
Fix: precondition parsing issue
Fix the parsing issue of the two preconditions of If-Match and
If-None-Match.

We found that the request sent through the AWS S3 SDK included
double quotes for the values of these two preconditions.

This part is not specified in the AWS S3 API Reference.

Affected APIs: HeadObject, GetObject

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-22 17:25:19 +08:00
Mofei Zhang
4a456f6951
Enhancement: add python based test script for s3
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-21 17:17:13 +08:00
Mofei Zhang
7a57837f74
Fix: third-party license information
Adjust the format of the license header information of the third-party
code used to make the format more compliant. The NOTICE file supplements
the third-party code NOTIC information and explains the use of the code.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-14 19:29:39 +08:00
Mofei Zhang
4edef106d5
Enhancement: add routing for unimplemented APIs
In view of the particularity of the AWS S3 protocol rules, additional
routes are implemented for unimplemented APIs to prevent users from
incorrectly routing to other APIs when requesting unimplemented APIs.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-08 11:21:25 +08:00
Mofei Zhang
e37abd0731
Enhancement: reduce blocking under concurrency
The blacklist mechanism has been added to the user store and volume
manager in the object node to solve the adverse effect of a large number
of invalid requests on system performance in a short period of time.

Stream requests to the user store and volume manager in the object node
in a hash bucket manner, reducing the blocking caused by read and write
locks under concurrency.

Abstracted the user store and added an implementation that does not
cache user information. This is enabled through the "strict"
configuration to provide sufficient real-time performance with cluster
information during interface compatibility testing.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-31 19:36:21 +08:00
Mofei Zhang
509d96c3d1
Enhancement: folder operations in S3 APIs
Supports folder operations in the same way as Amazon AWS S3.
  - Get folder meta info by HeadObject interface.
  - Create folder when header specifies Content-Type as
    application/directory and Content-Length as 0 on PutObject interface.
  - Delete an empty folder by DeleteObject interface.
  - The GetObject interface responds normally when the target is an
    existing directory.
  - Differentiate folders and files on the ListObject interface through the
    CommonPrefix property (already supported).

Following object storage interface haven been implemented folder operations:
  - HeadObject
  - GetObject
  - DeleteObject
  - DeleteObjects
  - ListObjects
  - ListObjectsV2

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-30 17:01:45 +08:00
Mofei Zhang
ddfd8b7c3f
Enhancement: improve error parsing for master APIs
Enhancement: improve error parsing for master APIs.
Fix: status code of DelteBucket ans DeleteObjects API.
Fix: url signature parsing of signature algorithm V2.
Add: s3tests framework for compatibility test.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-22 08:31:51 +08:00
wenjia322
19836a57cd fix: nil pointer when release bucket
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-20 00:03:31 +08:00
wenjia322
745728a306 refactor: change index of 'UserInfo' from 'ak' to 'userID'
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-20 00:03:31 +08:00
Mofei Zhang
b49d89c490
Add: list user with filter command for CLI
Add: list user with filter command for CLI
Enhancement: performance of ListBuckets API
Fix: action identify issue

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-17 10:26:41 +08:00
Mofei Zhang
11d0a4ddf0
Refactor: reorganize permission and actions
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-17 00:11:01 +08:00
wenjia322
ecce5dc45f feature: add master api 'listVols'; remove redundant 'deleteVolPolicy' in deleteBucket
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-16 22:41:46 +08:00
wenjia322
82ca381f5a fix: report no error when vol not exists
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-16 11:21:03 +08:00
wenjia322
ceb9c8cbd5 fix: get vol create time when list buckets
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-13 13:20:26 +08:00
wenjia322
e4b8934bdd refactor: improve identifiers' name with better semantics
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-10 15:53:43 +08:00
Mervin
fbbe1e0a7e
Refactor: param parse in object storage interface handlers
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 00:49:13 +08:00
wenjia322
05b064c587 feature: change user information when create or delete volume
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-09 22:40:13 +08:00
Mervin
789454791d
Fix: ignore policy check when action is a signature ignore action
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-09 11:45:55 +08:00
wenjia322
acaf24101d refactor: move util/oss/user.go to proto/oss_proto.go
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-04 14:58:12 +08:00