1. Fix: parallel-safe issue in conn pool and metanode conn serve
2. Fix: nil pointer issue when list parts for non-exist uploadId
3. Enhancement: validate read size while read from connnect.
4. Enhancement: clean up useless data on failure while upload part.
5. Enhancement: generate ETag if do not exist when listing.
6. Enhancement: include error message when formatting packet to string.
7. Enhancement: improve compatile with old version SDK.
8. Enhancement: success when retry add exist multipart part.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Signed-off-by: 张墨飞 <zhangmofei@jd.com>
The PutObject interface supports setting user-defined metadata. The
GetObject and HeadObject interfaces can correctly return the stored
user-defined metadata.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
1. Add tests for tagging operations.
2. Support setup tagging when put an object.
3. Fix routing issue on DeleteObjectTagging interface.
4. Improve tagging encoding for storage.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
When ETag is stored, it is versioned by encoding, which can make
the file system interface and object storage interface more closely
integrated.
Referring to the Amazon AWS S3 Developer Guide:
"Amazon S3 response includes an ETag that uniquely identifies the
combined object data. This ETag will not necessarily be an MD5 hash
of the object data."
When a new object is written to ChubaoFS through the object storage
interface, ChubaoFS calculates the ETag value for it, and uses the
modification time of the object as its version information when
encoding and storing.
If this file is modified by the file system interface, the ETag of
this file will be invalid because the version information will be
inconsistent with modification time.
After the object storage interface finds that the ETag is invalid,
it will automatically generate a new ETag matching its modification
time.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
We found thousands of http transport goroutines in a process that has
been running for several weeks.
The master client creates a new http client instance every time it sends
an http request, and these instances do not call the method provided by
them to release the restricted connection after use. Try to use
singleton http client to solve this problem.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Fix the parsing issue of the two preconditions of If-Match and
If-None-Match.
We found that the request sent through the AWS S3 SDK included
double quotes for the values of these two preconditions.
This part is not specified in the AWS S3 API Reference.
Affected APIs: HeadObject, GetObject
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
During the process of scanning the child nodes of the current
directory, there may be other parallel operations that may
delete the current directory.
If got the syscall.ENOENT error when invoke readdir, it means
that the above situation has occurred.
At this time, stops process and returns success.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Adjust the format of the license header information of the third-party
code used to make the format more compliant. The NOTICE file supplements
the third-party code NOTIC information and explains the use of the code.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Implemented the following conditional parameters in the GetObject
and HeadObject interfaces:
- If-Match
- If-Modified-Since
- If-None-Match
- If-Unmodified-Since
Signed-off-by: yuboLee <pangbolee@gmail.com>
Supports folder operations in the same way as Amazon AWS S3.
- Get folder meta info by HeadObject interface.
- Create folder when header specifies Content-Type as
application/directory and Content-Length as 0 on PutObject interface.
- Delete an empty folder by DeleteObject interface.
- The GetObject interface responds normally when the target is an
existing directory.
- Differentiate folders and files on the ListObject interface through the
CommonPrefix property (already supported).
Following object storage interface haven been implemented folder operations:
- HeadObject
- GetObject
- DeleteObject
- DeleteObjects
- ListObjects
- ListObjectsV2
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
When the object storage HeadObject and GetObject have no etag
information in the extended attributes of the file, the ETag header is
not returned. This solves the problem of data verification failure when
reading files written through the file system interface using the AWS S3
SDK through the object storage interface to a certain extent.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
The list xattrs interface of posix only needs to specify all xattr
keys under the inode, and does not need the values of these keys.
Refactor the list xattr related interface in the metadata and only
transfer keys to optimize performance.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
Enhancement: validate user policy when mount volume.
Enhancement: when an object node loads a volume, different volumes do not block each other.
Enhancement: when checking policies, check user permissions before checking volume policies to reduce the overhead of loading volumes.
Enhancement: release connection pool when releasing meta wrapper resources.
Enhancement: the object node automatically releases the deleted volume.
Fix: response when deleting non-existing files using Object Storage
Interface.
Fix: url parameter parsing issue of signature algorithm V2.
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
