Commit Graph

63 Commits

Author SHA1 Message Date
leonrayang
3396accd80 feat(projectstructure): reorganize the module directory in the root to a suitable subdirectory
close:#22458839

Signed-off-by: leonrayang <chl696@sina.com>
2024-12-26 09:24:22 +08:00
Victor1319
f9c00bb5c8 fix(meta): adjust the call order of the RegistConsul func to prevent concurrent modify to clustername.
#22873788

Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2024-12-26 09:24:22 +08:00
Victor1319
be565f6609 refactor(server): refactor http connect pool when comminute with master.
Signed-off-by: Victor1319 <zengxuewei@oppo.com>
2024-12-26 09:24:22 +08:00
W9068822
d25840b71f feat(metrics): report version of client, metanode, datanode, authnode, lcnode, objectnode
Signed-off-by: W9068822 <v-lijianrong1@oppo.com>
2024-07-22 17:10:59 +08:00
slasher
5c7898354f style(all): format all codes with golangci tools
closes #3371

@formatter:off

Signed-off-by: slasher <shenjie1@oppo.com>
2024-05-14 09:54:13 +08:00
slasher
1d84f74acd chore(format): format all code with gofumpt
Signed-off-by: slasher <shenjie1@oppo.com>
2024-01-12 14:21:16 +08:00
leonrayang
be0ed93edc enhance(httpserver):resolve Potential Slowloris attacks according to ostif security team
Signed-off-by: leonrayang <chl696@sina.com>
2023-12-11 10:09:36 +08:00
tangdeyi
9388fc02b3 feat(objectnode): disable createBucket by s3
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-12-06 14:10:49 +08:00
yhjiango
c8e92fb0f0 feat(objectnode): support local and external audit log
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-11-01 14:46:14 +08:00
tangdeyi
f9c273ac4c feat(objectnode): add some s3qos log
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-09-05 10:28:40 +08:00
tangdeyi
210de3e28c fix(objectnode): fix s3qos zero nodenum
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-31 16:36:14 +08:00
tangdeyi
14f0f67c73 feat(objnode): s3 api qos
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-31 16:36:14 +08:00
tangdeyi
f8dbf7db3b fix(objectnode): dentry/attr cache code refactor and bugfix
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-30 20:05:32 +08:00
yhjiango
4401624f96 feat(objectnode): s3 sts and signature auth
1. sts federation token
2. signature auth code refactor

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-08-30 20:05:32 +08:00
tangdeyi
0bb8a4b6ea feat(objectnode): read/write cold volume buffer pool
Signed-off-by: tangdeyi <tangdeyi@oppo.com>
2023-08-04 18:08:38 +08:00
yhjiango
2040ce00a9 fix(objectnode): fix bucket policy and location processing
1. Fix the problem of returning 500 internal errors in PutBucketPolicy.
2. Add related parameter verification in GetBucketLocation.

Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-07-20 10:58:50 +08:00
leonrayang
d5bb45ea49 docs. update to cubefs
Signed-off-by: leonrayang <chl696@sina.com>
2023-05-26 11:17:10 +08:00
slasher
5361602c10 build: cubefs using local module of blobstore
close: #1527

Signed-off-by: slasher <shenjie1@oppo.com>
2023-05-23 19:09:11 +08:00
yuxiaobo
ac75137295 build: the build script of blobstore is merged into cubefs
1. rocksdb upgrade to 6.3.6
2. optimize the build scripts
3. fix build errors
4. add env variable for blobstore unit test
5. update git ignore file to ignore build and test result
6. modify open files limit for unit test

Signed-off-by: yuxiaobo <yuxiaobo@oppo.com>
Signed-off-by: slasher <shenjie1@oppo.com>
Signed-off-by: Victor1319 <834863182@qq.com>
2023-05-23 15:24:53 +08:00
yhjiango
0affc88413 fix(objectnode): fully compatible with s3
Signed-off-by: yhjiango <jiangyunhua@oppo.com>
2023-05-18 10:48:57 +08:00
leonrayang
6826097467 update: go fmt
Signed-off-by: leonrayang <chl696@sina.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
01de17866e fix: fix appending empty obj extent key bug
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
5f9e775cbe Cold volume support for S3 in objectnode
Signed-off-by: Patrick Wu <wuchunhuan@gmail.com>
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
3d5464c313 add lifecycle to cache item
Signed-off-by: Patrick Wu <wuchunhuan@gmail.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
6dfb9380c6 add meta cache for objectnode
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
5f34fd5102 fix: optimization for dir quota limitation
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
Patrick Wu
f15e2a8d6d feature: add quota limitation for directory children number
Signed-off-by: Patrick Wu <wuchunhuan@oppo.com>
2023-03-04 16:13:35 +08:00
leonrayang
9a30b9f884 update: update to CubeFS
Signed-off-by: leonrayang <chl696@sina.com>
2022-08-22 18:16:09 +08:00
leonrayang
c1adba0775 fix: docker test run error
Signed-off-by: leonrayang <chl696@sina.com>
2022-04-13 15:57:55 +08:00
leonrayang
2970e2c3d1 update. go import path change from chubaofs to cubefs
Signed-off-by: leonrayang <chl696@sina.com>
2022-01-25 18:17:02 +08:00
wenjia322
6cc377ae1c refactor: choose different 'MetaLoader' by configuration 'strict'
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-07-03 18:05:43 +08:00
Mofei Zhang
9e89d40ddf
Fix: xml response of get bucket location interface
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-05-03 20:36:45 +08:00
Mofei Zhang
184ca0b428
Ehancement: improve compatibility with nginx
Http's Expect header is a special header. When nginx is used as the
reverse proxy in the front end of ObjectNode, nginx will process the
Expect header information in advance, send the http status code 100 to
the client, and will not forward this header information to ObjectNode.
At this time, if the client request uses the Expect header when signing,
it will cause the ObjectNode to verify the signature.
A workaround is used here to solve this problem. Add the following
configuration in Nginx:

proxy_set_header X-Forwarded-Expect $ http_Expect

In this way, nginx will not only automatically handle the Expect
handshake, but also send the original value of Expect to the ObjectNode
through X-Forwared-Expect. ObjectNode only needs to use the value of
X-Forwared-Expect.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-30 15:12:53 +08:00
Mervin
6f98e0fc1b
Merge branch 'master' into disable_action 2020-04-15 08:44:11 +08:00
Mofei Zhang
8226b367c9
Fix: format pattern issue in logging
Fix format pattern issue in logging and correct variable naming.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-15 08:41:57 +08:00
yuboLee
70daeb47cf Enhancement: add support for disable some actions via configurations in objectnode.json
Signed-off-by: yuboLee <pangbolee@gmail.com>
2020-04-14 23:16:17 +08:00
Mofei Zhang
7a57837f74
Fix: third-party license information
Adjust the format of the license header information of the third-party
code used to make the format more compliant. The NOTICE file supplements
the third-party code NOTIC information and explains the use of the code.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-04-14 19:29:39 +08:00
zhuzhengyi1
721b9f3858 Enhancement: add prometheus metrics for objectnode
Signed-off-by: zhuzhengyi1 <zhuzhengyi@jd.com>
2020-04-09 14:11:51 +08:00
Mofei Zhang
e37abd0731
Enhancement: reduce blocking under concurrency
The blacklist mechanism has been added to the user store and volume
manager in the object node to solve the adverse effect of a large number
of invalid requests on system performance in a short period of time.

Stream requests to the user store and volume manager in the object node
in a hash bucket manner, reducing the blocking caused by read and write
locks under concurrency.

Abstracted the user store and added an implementation that does not
cache user information. This is enabled through the "strict"
configuration to provide sufficient real-time performance with cluster
information during interface compatibility testing.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-31 19:36:21 +08:00
Mofei Zhang
bdb1aa1ffd
Enhancement: optional not including ETag
When the object storage HeadObject and GetObject have no etag
information in the extended attributes of the file, the ETag header is
not returned. This solves the problem of data verification failure when
reading files written through the file system interface using the AWS S3
SDK through the object storage interface to a certain extent.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-26 19:08:38 +08:00
Mofei Zhang
333cdcb887
Enhancement: unified configuration items
Adjust the service listen configuration format of ObjectNode to the
port value.

Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-23 19:25:42 +08:00
wenjia322
745728a306 refactor: change index of 'UserInfo' from 'ak' to 'userID'
Signed-off-by: wenjia322 <buaa1214wwj@126.com>
2020-03-20 00:03:31 +08:00
Mofei Zhang
11d0a4ddf0
Refactor: reorganize permission and actions
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-17 00:11:01 +08:00
Mofei Zhang
d99f147e6a
Refacrot: move actions to command proto package
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-03-16 08:43:17 +08:00
Mervin
5db80feeee Refactor: http api route in master
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-11 09:47:41 +08:00
Mervin
7cbb5a21d9
Enhancement: support CORS request
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 21:46:19 +08:00
Mervin
fbbe1e0a7e
Refactor: param parse in object storage interface handlers
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-10 00:49:13 +08:00
Mervin
789454791d
Fix: ignore policy check when action is a signature ignore action
Signed-off-by: Mervin <mofei2816@gmail.com>
2020-03-09 11:45:55 +08:00
Mofei Zhang
aec409ffcf Refactor: using middleware for policy valication.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-02-27 20:36:28 +08:00
Mofei Zhang
4979d728b2 Add: configuration for signature ignore actions
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
2020-02-26 00:00:32 +08:00