Commit Graph

58 Commits

Author SHA1 Message Date
tangdeyi
b7cf39a0c3 fix(objectnode): optimize the logic of createBucket
with #22863348

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2024-12-26 09:24:22 +08:00
Victor1319
4df91ebc6e refactor(all): reslove merge conflicts between 3.3.x and 3.4.0.
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2024-07-18 16:57:38 +08:00
yhjiango
e487250c51 fix(object): adjust STS auth process
Signed-off-by: yhjiango <jiangyunhua@oppo.com>

(cherry picked from commit 1f23b3f620)
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-07-18 16:57:38 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
2024-05-14 09:54:13 +08:00
yhjiango
1f23b3f620 fix(object): adjust STS auth process
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2024-03-06 15:42:27 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt
Signed-off-by: slasher <shenjie1@oppo.com>
2024-01-12 14:21:16 +08:00
yhjiango
c8e92fb0f0 feat(objectnode): support local and external audit log
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-11-01 14:46:14 +08:00
yhjiango
4401624f96 feat(objectnode): s3 sts and signature auth
1. sts federation token
2. signature auth code refactor

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
d11e9142a2 feat(objectnode): object lock
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
bf5dad6ee6 fix(objectnode): bucket metadata cache synchronization
Signed-off-by: tangdeyi <deyitang94@gmail.com>
2023-06-16 17:42:02 +08:00
yhjiango
f83ec30d71 style(objectnode): add status code and error message log in the end of traceMiddleware
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
0a4830dd93 fix(objectnode): fix the list multipart uploads and dir object inaccessible
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
3eddf3e623 fix(objectnode): Options CORS
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
ad2f245e76 fix(objectnode): Options CORS
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
yhjiango
0affc88413 fix(objectnode): fully compatible with s3
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
277a988c93 fix(objectnode): enhance s3 compatibility
1.CORS supports preflight request;
2.S3-compatible err reply;
3.Small parts upload bugfix;
4.Specific status code when panic;
5.UploadPartCopy;

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
831ea00d5d fix(objectnode): enhance s3 compatibility
1.CORS supports preflight request;
2.S3-compatible err reply;
3.Small parts upload bugfix;
4.Specific status code when panic;
5.UploadPartCopy;

Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
tangdeyi
9f0d7810a0 feature: bucket policy fully compatible with s3
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-05-18 10:48:57 +08:00
Patrick Wu
c8bf55ae69 S3 bugfix
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
leonrayang
9a30b9f884 update: update to CubeFS
Signed-off-by: leonrayang <chl696@sina.com>
2022-08-22 18:16:09 +08:00
leonrayang
2970e2c3d1 update. go import path change from chubaofs to cubefs
Signed-off-by: leonrayang <chl696@sina.com>
2022-01-25 18:17:02 +08:00
Victor1319
e3012d3f0e fix: defer set(err) will always make err be nil, replaced by defer func(){set(err)}, same as setWithLabels(err, labels)
Signed-off-by: Victor1319 <834863182@qq.com>
2021-02-20 16:04:16 +08:00
Mofei Zhang
dd45cd8ac5
Enhancement: error handling of ObjectNode
- Response Http400 after got unexpected EOF error when reading data from
  request connection.
- Fix ldflag issue from 'cli/build.sh'.
- Do not write failure status code if request connect closed by peer.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-08-06 12:12:50 +08:00
yuboLee
061a02cd12 handle ump warnning and error log info
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-08-03 10:53:57 +08:00
yuboLee
12b9f5385f Enhancement: add ump monitor log
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-07-22 10:36:09 +08:00
awzhgw
36d77c14d2
Merge pull request #729 from wenjia322/no_cache_meta
refactor: whether to cache volume oss meta is determined by the configuration parameter 'strict'
2020-07-06 16:08:51 +08:00
Mofei Zhang
879e041f95
Fix: S3 response when volume do not exists
When initializing the volume, if the master explicitly responds that the
specified volume does not exist, it will not retry.

During the processing of the S3 request, both the signature and the
strategy are checked for the existence of the volume. NoSuchBucket
response is returned when the specified volume does not exist.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-07-03 20:34:06 +08:00
wenjia322
6cc377ae1c refactor: choose different 'MetaLoader' by configuration 'strict'
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-07-03 18:05:43 +08:00
Mofei Zhang
516c09c0b4
Enhancment: optimized response status code monitoring
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-06-12 17:03:23 +08:00
yuboLee
8152e7690c Enhancement: Add exception request monitor via status code, such as 400, 403 and 500
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-06-12 14:20:58 +08:00
Mofei Zhang
4a1f4faf6d
Optimize CORS middleware
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-05-22 15:16:42 +08:00
wenjia322
ae3688cc2a feature: add 'corsMiddleware' and 'optionObject'
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-05-22 15:16:38 +08:00
Mofei Zhang
df4b7f9a42
Fix: parallel-safe issue in conn pool and metanode conn serve
1. Fix: parallel-safe issue in conn pool and metanode conn serve
2. Fix: nil pointer issue when list parts for non-exist uploadId
3. Enhancement: validate read size while read from connnect.
4. Enhancement: clean up useless data on failure while upload part.
5. Enhancement: generate ETag if do not exist when listing.
6. Enhancement: include error message when formatting packet to string.
7. Enhancement: improve compatile with old version SDK.
8. Enhancement:  success when retry add exist multipart part.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Signed-off-by: ๅผ ๅขจ้ฃž <zhangmofei@jd.com>
2020-05-21 12:28:18 +08:00
Mofei Zhang
184ca0b428
Ehancement: improve compatibility with nginx
Http's Expect header is a special header. When nginx is used as the
reverse proxy in the front end of ObjectNode, nginx will process the
Expect header information in advance, send the http status code 100 to
the client, and will not forward this header information to ObjectNode.
At this time, if the client request uses the Expect header when signing,
it will cause the ObjectNode to verify the signature.
A workaround is used here to solve this problem. Add the following
configuration in Nginx:

proxy_set_header X-Forwarded-Expect $ http_Expect

In this way, nginx will not only automatically handle the Expect
handshake, but also send the original value of Expect to the ObjectNode
through X-Forwared-Expect. ObjectNode only needs to use the value of
X-Forwared-Expect.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-30 15:12:53 +08:00
Mofei Zhang
0296e5709c
Fix: create time format of list buckets
Fix create time format of list buckets API to
"2006-01-02T15:04:05.000Z" as UTC time.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-25 22:04:05 +08:00
Mofei Zhang
43460bf5b7
Refactor: wrap and use built-in chunked reader
Wrap and use the chunked reader from the "httputil" package which
provided by Go and provides a close mwthod.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-15 11:11:08 +08:00
Mervin
6f98e0fc1b
Merge branch 'master' into disable_action 2020-04-15 08:44:11 +08:00
Mofei Zhang
8226b367c9
Fix: format pattern issue in logging
Fix format pattern issue in logging and correct variable naming.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-15 08:41:57 +08:00
yuboLee
70daeb47cf Enhancement: add support for disable some actions via configurations in objectnode.json
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-04-14 23:16:17 +08:00
Mofei Zhang
7a57837f74
Fix: third-party license information
Adjust the format of the license header information of the third-party
code used to make the format more compliant. The NOTICE file supplements
the third-party code NOTIC information and explains the use of the code.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-14 19:29:39 +08:00
Mofei Zhang
4ff10e0326
Enhancement: adjust name of the monitoring point
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-09 15:52:19 +08:00
zhuzhengyi1
721b9f3858 Enhancement: add prometheus metrics for objectnode
Signed-off-by: zhuzhengyi1 <zhuzhengyi@jd.com>
2020-04-09 14:11:51 +08:00
Mofei Zhang
509d96c3d1
Enhancement: folder operations in S3 APIs
Supports folder operations in the same way as Amazon AWS S3.
  - Get folder meta info by HeadObject interface.
  - Create folder when header specifies Content-Type as
    application/directory and Content-Length as 0 on PutObject interface.
  - Delete an empty folder by DeleteObject interface.
  - The GetObject interface responds normally when the target is an
    existing directory.
  - Differentiate folders and files on the ListObject interface through the
    CommonPrefix property (already supported).

Following object storage interface haven been implemented folder operations:
  - HeadObject
  - GetObject
  - DeleteObject
  - DeleteObjects
  - ListObjects
  - ListObjectsV2

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-30 17:01:45 +08:00
Mofei Zhang
ddfd8b7c3f
Enhancement: improve error parsing for master APIs
Enhancement: improve error parsing for master APIs.
Fix: status code of DelteBucket ans DeleteObjects API.
Fix: url signature parsing of signature algorithm V2.
Add: s3tests framework for compatibility test.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-22 08:31:51 +08:00
Mofei Zhang
270df7fc4a
Merge branch 'master' into object-store
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-18 09:47:59 +08:00
Mofei Zhang
5bf657f2eb
Fix: error while list objects with multi-level directory prefix
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-17 21:13:14 +08:00
Mofei Zhang
11d0a4ddf0
Refactor: reorganize permission and actions
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-17 00:11:01 +08:00
Mervin
7cbb5a21d9
Enhancement: support CORS request
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 21:46:19 +08:00
Mervin
83b657724e
Fix: xml entity issue in PutXAttrRequest struct
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 14:26:25 +08:00
Mervin
fbbe1e0a7e
Refactor: param parse in object storage interface handlers
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 00:49:13 +08:00