gian/cubefs
1
0
Fork 0
mirror of https://github.com/cubefs/cubefs.git synced 2026-08-02 10:06:14 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

merge with master branch and resolve conflict

Browse Source 
Signed-off-by: Mofei Zhang <mofei2816@gmail.com>
This commit is contained in:
Mofei Zhang 2019-12-25 23:34:07 +08:00
commit ea557ef830
No known key found for this signature in database
GPG Key ID: 7C19725C197AE672
32 changed files with 759 additions and 2125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@ -0,0 +1,25 @@
---
name: Bug report
about: Create a report to help us improve
title: Bug report
labels: bug
assignees: awzhgw
---
**Describe the bug**
A clear and concise description of what the bug is.
**Environment**
Describe your server system version, kernel version
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Additional context**
Add any other context about the problem here.

20
.github/ISSUE_TEMPLATE/feature_request.md vendored Normal file
View File

@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: Feature request
labels: enhancement
assignees: awzhgw
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

15
.github/pull_request_template.md vendored Normal file
View File

@ -0,0 +1,15 @@
<!-- Thanks for sending a pull request! -->
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
```

49
README.md
View File

@ -12,23 +12,28 @@
## Overview
ChubaoFS (储宝文件系统 in Chinese) is a distributed file system and object storage service for cloud native applications.
ChubaoFS (储宝文件系统 in Chinese) is a distributed file system and object storage service for cloud native applications. It is hosted by the [Cloud Native Computing Foundation](https://cncf.io) (CNCF) as a [sandbox](https://www.cncf.io/sandbox-projects/) project.
It has the following key features:
ChubaoFS has been commonly used as the underlying storage infrastructure for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes.
An advantage of doing so is to separate storage from compute - one can scale up or down based on the workload and independent of the other, providing total flexibility in matching resources to the actual storage and compute capacity required at any given time.
- scale-out metadata management
- strong replication consistency for both append and random write
Some key features of ChubaoFS include:
- specific storage optimizations for large and small files
- Scale-out metadata management
- multi-tenancy
- Strong replication consistency
- POSIX-compatible & mountable
- Specific performance optimizations for large/small files and sequential/random writes
- S3-like object storage interface
- Multi-tenancy
In particular, ChubaoFS can be used to decouple computing from storage for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes.
- POSIX-compatible and mountable
- S3-compatible object storage interface
We are committed to making ChubaoFS better and more mature. Please stay tuned.
## Document
@ -38,7 +43,6 @@ https://chubaofs.readthedocs.io/zh_CN/latest/
## Build
Build latest version of chubaofs is simply as following:
```
$ git clone http://github.com/chubaofs/chubaofs.git
@ -46,27 +50,30 @@ $ cd chubaofs
$ make
```
If the build is successful, `cfs-server` and `cfs-client` will be found in directory `build/bin`
If the build succeeds, `cfs-server` and `cfs-client` can be found in `build/bin`
## Docker
Under the docker directory, a helper tool called run_docker.sh is provided to run ChubaoFS with docker-compose.
A helper tool called `run_docker.sh` (under the `docker` directory) has been provided to run ChubaoFS with [docker-compose](https://docs.docker.com/compose/).
To start a minimal ChubaoFS cluster from scratch, note that **/data/disk** is arbitrary, and make sure there are at least 10G available space.
```
$ docker/run_docker.sh -r -d /data/disk
```
If client starts successfully, use `mount` command in client docker shell to check mount status:
Note that **/data/disk** can be any directory but please make sure it has at least 10G available space.
To check the mount status, use the `mount` command in the client docker shell:
```
$ mount | grep chubaofs
```
Open http://127.0.0.1:3000 in browser, login with `admin/123456` to view grafana monitor metrics.
Or run server and client seperately by following commands:
To view grafana monitor metrics, open http://127.0.0.1:3000 in browser and login with `admin/123456`.
To run server and client separately, use the following commands:
```
$ docker/run_docker.sh -b
@ -83,21 +90,19 @@ $ docker/run_docker.sh -h
## License
Licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
ChubaoFS is licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
For detail see [LICENSE](LICENSE) and [NOTICE](NOTICE).
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fchubaofs%2Fcfs.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fchubaofs%2Fcfs?ref=badge_large)
## Reference
Reference to cite when you use the system in a research paper or tech report:
Haifeng Liu, et al., CFS: A Distributed File System for Large Scale Container Platforms. SIGMOD‘19, June 30-July 5, 2019, Amsterdam, Netherlands.
Haifeng Liu, et al., CFS: A Distributed File System for Large Scale Container Platforms. SIGMOD‘19, June 30-July 5, 2019, Amsterdam, Netherlands. https://dl.acm.org/citation.cfm?doid=3299869.3314046
arXiv: https://arxiv.org/abs/1911.03001
For more information, please refer to https://dl.acm.org/citation.cfm?doid=3299869.3314046 and https://arxiv.org/abs/1911.03001
## Community
- Twitter: [@ChubaoFS](https://twitter.com/ChubaoFS)
- Mailing list: chubaofs-maintainers@groups.io
- Mailing list: should use chubaofs-users@groups.io
- Slack: [chubaofs.slack.com](https://chubaofs.slack.com/)

24
client/fs/dio_darwin.go Normal file
View File

@ -0,0 +1,24 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
import (
"bazil.org/fuse"
)
func isDirectIOEnabled(flags fuse.OpenFlags) bool {
// flag 'O_DIRECT' is not supported in Darwin(Apple MacOS).
return false
}

25
client/fs/dio_linux.go Normal file
View File

@ -0,0 +1,25 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
import (
"syscall"
"bazil.org/fuse"
)
func isDirectIOEnabled(flags fuse.OpenFlags) bool {
return (int(flags) & syscall.O_DIRECT) != 0
}

24
client/fs/dio_windows.go Normal file
View File

@ -0,0 +1,24 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
import (
"bazil.org/fuse"
)
func isDirectIOEnabled(flags fuse.OpenFlags) bool {
// flag 'O_DIRECT' is not supported in Microsoft Windows.
return false
}

4
client/fs/dir.go
View File

@ -97,6 +97,8 @@ func (d *Dir) Create(ctx context.Context, req *fuse.CreateRequest, resp *fuse.Cr
resp.Flags |= fuse.OpenKeepCache
}
d.super.ic.Delete(d.inode.ino)
elapsed := time.Since(start)
log.LogDebugf("TRACE Create: parent(%v) req(%v) resp(%v) ino(%v) (%v)ns", d.inode.ino, req, resp, inode.ino, elapsed.Nanoseconds())
return child, child, nil
@ -131,6 +133,8 @@ func (d *Dir) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (fs.Node, error
d.super.nodeCache[inode.ino] = child
d.super.fslock.Unlock()
d.super.ic.Delete(d.inode.ino)
elapsed := time.Since(start)
log.LogDebugf("TRACE Mkdir: parent(%v) req(%v) ino(%v) (%v)ns", d.inode.ino, req, inode.ino, elapsed.Nanoseconds())
return child, nil

6
client/fs/file.go
View File

@ -17,16 +17,16 @@ package fs
import (
"fmt"
"io"
"syscall"
"time"
"bazil.org/fuse"
"bazil.org/fuse/fs"
"golang.org/x/net/context"
"sync"
"github.com/chubaofs/chubaofs/proto"
"github.com/chubaofs/chubaofs/util/log"
"sync"
)
// File defines the structure of a file.
@ -201,7 +201,7 @@ func (f *File) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.Wri
}()
var waitForFlush, enSyncWrite bool
if ((int(req.FileFlags) & syscall.O_DIRECT) != 0) || (req.FileFlags&fuse.OpenSync != 0) {
if isDirectIOEnabled(req.FileFlags) || (req.FileFlags&fuse.OpenSync != 0) {
waitForFlush = true
enSyncWrite = f.super.enSyncWrite
}

20
clientv2/fs/dio_darwin.go Normal file
View File

@ -0,0 +1,20 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
func isDirectIOEnabled(flags int) bool {
// flag 'O_DIRECT' is not supported in Darwin(Apple MacOS).
return false
}

23
clientv2/fs/dio_linux.go Normal file
View File

@ -0,0 +1,23 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
import (
"syscall"
)
func isDirectIOEnabled(flags int) bool {
return (int(flags) & syscall.O_DIRECT) != 0
}

20
clientv2/fs/dio_windows.go Normal file
View File

@ -0,0 +1,20 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package fs
func isDirectIOEnabled(flags int) bool {
// flag 'O_DIRECT' is not supported in Microsoft Windows.
return false
}

5
clientv2/fs/file.go
View File

@ -17,10 +17,11 @@ package fs
import (
"errors"
"fmt"
"golang.org/x/net/context"
"io"
"syscall"
"golang.org/x/net/context"
"github.com/jacobsa/fuse"
"github.com/jacobsa/fuse/fuseops"
@ -133,7 +134,7 @@ func (s *Super) WriteFile(ctx context.Context, op *fuseops.WriteFileOp) error {
}()
var waitForFlush, enSyncWrite bool
if (flags&syscall.O_DIRECT != 0) || (flags&syscall.O_SYNC != 0) {
if isDirectIOEnabled(flags) || (flags&syscall.O_SYNC != 0) {
waitForFlush = true
enSyncWrite = s.enSyncWrite
}

85
docs/source/design/authnode.rst Normal file
View File

@ -0,0 +1,85 @@
Authnode
=========
Internet and Intranet are insecure places where malicious hackers usually use tools to "sniff" sensitive information off of the network. Worse yet, client/server are not trusted to be honest about their identities. Therefore, ChubaoFS encounters some common security problems once deployed in the Network.
Security Problems
------------------
- Unauthorized parties may access the resource, such as restful API, volume information.
- Communication channel is vulnerable to `Man-in-the-middle` (`MITM`) attacks.
High Level Architecture
-----------------------
`Authnode` is the security node providing a general Authentication & Authorization framework for `ChubaoFS`. Besides, `Authnode` acts as a centralized key store of both symmetric and asymmetric key. `Authnode` adopts and customizes the idea of authentication from `Kerberos` which is built on top of tickets. Specifically, whenever a client node (`Master`, `Meta`, `Data` or `Client` node) accesses a service, it's firstly required to show the shared secret key for authentication in `Authnode`. If successful, `Authnode` would issue a time-limited ticket specifically for that service. With the purpose of authorization, capabilities are embedded in tickets to indicate `who can do what on what resource`.
.. image:: pic/authflow.png
:align: center
:scale: 50 %
:alt: Architecture
In the context of `Authnode`, we define a node as `Client` if it is responsible to initialize a service request while `Server` or `Service` is defined as the node responding that request. In this case, any `ChubaoFS` nodes can be acted as either `Client` or `Server`.
The communication between `Client` and `Server` is based on `HTTPS` or `TCP` and the workflow of `Authnode` is depicted in the graph above and briefly described as follows:
Ticket Request (F1)
+++++++++++++++++++
Before any service requests, client holding a secret key *Ckey* is required to get a ticket from `Authnode` for the service (called `target service`).
- C->A: Client sends a request including Client ID (*id*) and Service ID (*sid*) indicating target service.
- C<-A: Server lookups client secret key (*CKey*) and server secret key (*Skey*) from `key store` and responds a *Ckey*-encrypted message mainly including and a session key (*sess_key*) and target service ticket (*ticket*) encrypted with secret key *Skey* of target service if verification succeeds.
After obtaining the ticket and processing some security checks with *Ckey*, client has in its possession *sess_key* and Skey{*ticket*} for future service requests.
Service Request in HTTPS (F2)
+++++++++++++++++++++++++++++
If a service request is sent via HTTPS protocol, it has the following steps:
- C->S: Client sends a request containing SKey {*ticket*}.
- C<-S: Server (1) performs message decryption and get the ticket, (2) verifies its capabilities and (3) responds the *data* which is encrypted with *sess_key* extracted from ticket.
Client uses *sess_key* to decrypt message returned from server and verify its validity.
Service Request in TCP (F3)
+++++++++++++++++++++++++++
If a service request is sent via TCP protocol, it has the following steps:
- C->S: Client sends a request containing SKey {*ticket*}.
- C<-S: Server (1) decrypts the ticket and validate its capabilities, (2) extracts *sess_key*, (3) generates a random number *s_n* and (4) responds this number encrypted with *sess_key*.
- C->S: Client decrypts the replied message and send to server another message including a randomly generated number *s_c* and *s_n* + 1, both of which are encrypted with *sess_key*.
- C<-S: Server verifies whether *s_n* has been increased by one and sends back a message including *s_c* + 1 which is encrypted by *sess_key* if the verification succeeds.
- C<->S: Client verifies whether *s_c* has been increased by one after message decryption. If successful, an authenticated communication channel has been established between client and server. Based on this channel, client and server can perform further communications.
Future Work
-----------
`Authnode` supports a general authentication and authorization which is an emergent need for `ChubaoFS`. There are two directions of security enhancements for `ChubaoFS` in the future.
Feature Enrichment
++++++++++++++++++
Current implementation of `Authnode` doesn't support some advanced features:
- Key rotation: Shared secret keys are hardcoded in client and server and would not changed. It increases security risks that attacks break the encryption and find the keys. Rotating keys on a regular basis would help to mitigate such risks.
- Ticket revocation: For performance considerations, ticket would be valid for a while (such as several hours). If a client unfortunately leaks its ticket, malicious parties are able to use the ticket for service request during the period of being valid. Ticket revocation mechanism can prevent such an issue by revoking it once leakage happens.
- HSM support: `Authnode` is the security bottleneck in `ChubaoFS`. Breaking `Authnode` means compromising the whole system since it manages the key store. Hardware Security Module or HSM provides physical safeguards for key management. Having `Authnode` protected by HSM (for example *SGX*) can mitigate the risk of `Authnode` being compromised.
End-to-End Data Encryption
++++++++++++++++++++++++++
Current implementation of `Authnode` doesn't systematically support encryption for data in transit and at rest even though we may use session key to encrypt data during communication. A more secure way to protect data is to have `End-to-End` `Data` `Encryption`. In particular, encryption keys are managed and distributed by `Authnode` and data are encrypted in client node, sent via network and stored in server. Compared with server side encryption based on existing tools (`fscrypt`, `ecryptfs` and `dm-crypt`), `End-to-End` `Data` `Encryption` has the following advantages at least:
- It mitigates data leakage once data servers (for example `Data Node`) are broken into by attackers since keys of data decoding are stored in `Authnode`.
- It provides a centralized management (rotation, revocation and generation) for encryption key.

BIN
docs/source/design/pic/authflow.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 148 KiB

2
docs/source/index.rst
View File

@ -17,6 +17,7 @@ Welcome to ChubaoFS(Chubao File System)
design/datanode
design/objectnode.rst
design/client
design/authnode
.. toctree::
:maxdepth: 2
@ -27,6 +28,7 @@ Welcome to ChubaoFS(Chubao File System)
user-guide/datanode
user-guide/objectnode
user-guide/client
user-guide/authnode
user-guide/monitor
user-guide/fuse
user-guide/docker

344
docs/source/user-guide/authnode.rst Normal file
View File

@ -0,0 +1,344 @@
Authnode
====================
`Authnode` provides a general authentication & authorization service among `ChubaoFS` nodes. `Client`, `Master`, `Meta` and `Data` node are required to be authenticated and authorized before any resource access in another node.
Initially, each node (`Auth`, `Client`, `Master`, `Meta` or `Data` node) is launched with a secure key which is distributed by a authenticated person (for instance, cluster admin). With a valid key, a node can be identified in `Authnode` service and granted a ticket for resource access.
The overall work flow is: key creation and distribution --> ticket retrieval with key --> resource access with ticket.
Concepts
----------
- Key: a bit of secret shared data between a node and `Authnode` that asserts identity of a client.
- Ticket: a bit of data that cryptographically asserts identity and authorization (through a list of capabilities) for a service for a period of time.
- Capability: a capability is defined in the format of `node:object:action` where `node` refers to a service node (such as `auth`, `master`, `meta` or `data`), `object` refers to the resource and `action` refers to permitted activities (such as read, write and access). See examples below.
.. csv-table:: Capability Example
:header: "Capability", "Specifications"
"auth:createkey:access", "Access permission for createkey in Authnode"
"master:\*:\*", "Any permissions for any objects in Master node"
"\*:\*:\*", "Any permissions for any objects in any nodes"
Client Tool
------------
cfs-authtool is a client-side utility of `Authnode`, providing key managements (creation, view and modification) and ticket retrieval in and from `Authnode` keystore.
In particular, Each key is associated with an entity name or id, secret key string, creation time, role and capability specification.
.. csv-table:: Key structure
:header: "Key", "Type", "Description"
"id", "string", "Unique key identifier composed of letters and digits"
"key", "string", "Base64 encoded secret key"
"role", "string", "The role of the key (either client or service)"
"caps", "string", "The capabilities of the key"
Build
~~~~~~~~
Use the following commands to build client side tool for `Authnode`:
.. code-block:: bash
$ git clone http://github.com/chubaofs/chubaofs.git
$ cd chubaofs
$ make build
If successful, the tool `cfs-authtool` can be found in `build/bin`.
Synopsis
~~~~~~~~~~~
cfs-authtool ticket -host=AuthNodeAddress [-keyfile=Keyfile] [-output=TicketOutput] [-https=TrueOrFalse -certfile=AuthNodeCertfile] TicketService Service
cfs-authtool api -host=AuthNodeAddress -ticketfile=TicketFile [-data=RequestDataFile] [-output=KeyFile] [-https=TrueOrFalse -certfile=AuthNodeCertfile] Service Request
cfs-authtool authkey [-keylen=KeyLength]
TicketService := [getticket]
Service := [AuthService | MasterService | MetaService | DataService]
Request := [createkey | deletekey | getkey | addcaps | deletecaps | getcaps | addraftnode | removeraftnode]
AuthNode Configurations
------------------------
`Authnode` use **JSON** as configuration file format.
.. csv-table:: Properties
:header: "Key", "Type", "Description", "Mandatory"
"role", "string", "Role of process and must be set to master", "Yes"
"ip", "string", "host ip", "Yes"
"port", "string", "Http port which api service listen on", "Yes"
"prof", "string", "golang pprof port", "Yes"
"id", "string", "identy different master node", "Yes"
"peers", "string", "the member information of raft group", "Yes"
"logDir", "string", "Path for log file storage", "Yes"
"logLevel", "string", "Level operation for logging. Default is *error*.", "No"
"retainLogs", "string", "the number of raft logs will be retain.", "Yes"
"walDir", "string", "Path for raft log file storage.", "Yes"
"storeDir", "string", "Path for RocksDB file storage,path must be exist", "Yes"
"clusterName", "string", "The cluster identifier", "Yes"
"exporterPort", "int", "The prometheus exporter port", "No"
"authServiceKey", "string", "The secret key used for authentication of AuthNode", "Yes"
"authRootKey", "string", "The secret key used for key derivation (session and client secret key)", "Yes"
"enableHTTPS", "bool", "Option whether enable HTTPS protocol", "No"
**Example:**
.. code-block:: json
{
"role": "authnode",
"ip": "192.168.0.14",
"port": "8080",
"prof":"10088",
"id":"1",
"peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082",
"logDir": "/export/Logs/authnode",
"logLevel":"info",
"retainLogs":"100",
"walDir":"/export/Data/authnode/raft",
"storeDir":"/export/Data/authnode/rocksdbstore",
"exporterPort": 9510,
"consulAddr": "http://consul.prometheus-cfs.local",
"clusterName":"test",
"authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
"authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=",
"enableHTTPS":false
}
Steps for Starting ChubaoFS with AuthNode
------------------------------------------
Create Authnode key
~~~~~~~~~~~~~~~~~~~~~~
Run the command:
.. code-block:: bash
$ ./cfs-authtool authkey
If successful, two key files can be generated ``authroot.json`` and ``authservice.json`` under current directory.
They represent `authServiceKey` and `authRootKey` respectively.
 
example ``authservice.json`` :
.. code-block:: json
{
"id": "AuthService",
"key": "9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
"create_ts": 1573801212,
"role": "AuthService",
"caps": "{\"*\"}"
}
Edit ``authnode.json`` in `docker/conf` as following:
- ``authRootKey``: use the value of ``key`` in ``authroot.json``
- ``authServiceKey``: use the value of ``key`` in ``authService.json``
  example ``authnode.json`` :
.. code-block:: json
{
"role": "authnode",
"ip": "192.168.0.14",
"port": "8080",
"prof":"10088",
"id":"1",
"peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082",
"retainLogs":"2",
"logDir": "/export/Logs/authnode",
"logLevel":"info",
"walDir":"/export/Data/authnode/raft",
"storeDir":"/export/Data/authnode/rocksdbstore",
"exporterPort": 9510,
"consulAddr": "http://consul.prometheus-cfs.local",
"clusterName":"test",
"authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
"authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=",
"enableHTTPS":false
}
Start Authnode Cluster
~~~~~~~~~~~~~~~~~~~~~~~~~
In directory `docker/authnode`, run the following command to start a `Authnode` cluster.
.. code-block:: bash
$ docker-compose up -d
Create `admin` in Authnode
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Get `Authnode` ticket using `authServiceKey`:
.. code-block:: bash
$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=authservice.json -output=ticket_auth.json getticket AuthService
   
    example ``ticket_auth.json`` :
.. code-block:: json
{
"id": "AuthService",
"session_key": "A9CSOGEN9CFYhnFnGwSMd4WFDBVbGmRNjaqGOhOinJE=",
"service_id": "AuthService",
"ticket": "RDzEiRLX1xjoUyp2TDFviE/eQzXGlPO83siNJ3QguUrtpwiHIA3PLv4edyKzZdKcEb3wikni8UxBoIJRhKzS00+nB7/9CjRToAJdT9Glhr24RyzoN8psBAk82KEDWJhnl+Y785Av3f8CkNpKv+kvNjYVnNKxs7f3x+Ze7glCPlQjyGSxqARyLisoXoXbiE6gXR1KRT44u7ENKcUjWZ2ZqKEBML9U4h0o58d3IWT+n4atWKtfaIdp6zBIqnInq0iUueRzrRlFEhzyrvi0vErw+iU8w3oPXgTi+um/PpUyto20c1NQ3XbnkWZb/1ccx4U0"
}
Create `admin` using `Authnode` ticket:
.. code-block:: bash
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_auth.json -data=data_admin.json -output=key_admin.json AuthService createkey
    example ``data_admin.json`` :
.. code-block:: json
{
"id": "admin",
"role": "service",
"caps": "{\"API\":[\"*:*:*\"]}"
}
Create key for ChubaoFS cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Get `Authnode` ticket using `admin` key:
.. code-block:: bash
$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=key_admin.json -output=ticket_admin.json getticket AuthService
- Create key for Master
.. code-block:: bash
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_master.json -output=key_master.json AuthService createkey
    example ``data_master.json`` :
.. code-block:: json
{
"id": "MasterService",
"role": "service",
"caps": "{\"API\":[\"*:*:*\"]}"
}
Specifications:
id: will set `Client` ID
role: will set the role of id
caps: will set the capabilities of id
Edit ``master.json`` as following:
- ``masterServiceKey``: use the value of ``key`` in ``key_master.json``
- Create key for Client
.. code-block:: bash
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_client.json -output=key_client.json AuthService createkey
example ``data_client.json``:
.. code-block:: json
{
"id": "ltptest",
"role": "client",
"caps": "{\"API\":[\"*:*:*\"], \"Vol\":[\"*:*:*\"]}"
}
Edit ``client.json`` as following:
``clientKey``: use the value of ``key`` in ``key_client.json``
example ``client.json`` ：
.. code-block:: json
{
"masterAddr": "192.168.0.11:17010,192.168.0.12:17010,192.168.0.13:17010",
"mountPoint": "/cfs/mnt",
"volName": "ltptest",
"owner": "ltptest",
"logDir": "/cfs/log",
"logLevel": "info",
"consulAddr": "http://192.168.0.100:8500",
"exporterPort": 9500,
"profPort": "17410",
"authenticate": true,
"ticketHost": "192.168.0.14:8080,192.168.0.15:8081,192.168.0.16:8082",
"clientKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=",
"enableHTTPS": "false"
}
Specifications:
authenticate: will enable authentication flow if set true.
ticketHost: will set the IP/URL of `Authnode` cluster.
clientKey: will set the key generated by `Authnode`
enableHTTPS: will enable HTTPS if set true.
Start ChubaoFS cluster
~~~~~~~~~~~~~~~~~~~~~~~
Run the following to launch ChubaoFS cluster with `AuthNode` enabled:
.. code-block:: bash
$ docker/run_docker.sh -r -d /data/disk
Generate Certificate
---------------------
To prevent `MITM` (Man In The Middle) attacks, `HTTPS` is required for the communication between client and service.
The following steps show the generation of self-sign a certificate with a private (`.key`) and public key.
- Generating Key and Self Signed Cert:
.. code-block:: bash
$ openssl req \
-x509 \
-nodes \
-newkey rsa:2048 \
-keyout server.key \
-out server.crt \
-days 3650 \
-subj "/C=GB/ST=China/L=Beijing/O=jd.com/OU=Infra/CN=*"
- `server.crt`: `AuthNode` public certificate needed to be sent to `Client`
- `server.key`: `AuthNode` private key needed to be securely placed in `/app` folder in `Authnode`
For easy deployment, current implementation of `AuthNode` uses TLS option `insecure_skip_verify` and `tls.RequireAndVerifyClientCert`, which would skip secure verification of both client and server.
For environment with high security command, these options should be turned off.

4
metanode/inode.go
View File

@ -374,13 +374,16 @@ func (i *Inode) ExtentsTruncate(exts []BtreeItem, length uint64, ct int64) {
// IncNLink increases the nLink value by one.
func (i *Inode) IncNLink() {
mtime := Now.GetCurrentTime().Unix()
i.Lock()
i.NLink++
i.ModifyTime = mtime
i.Unlock()
}
// DecNLink decreases the nLink value by one.
func (i *Inode) DecNLink() {
mtime := Now.GetCurrentTime().Unix()
i.Lock()
if proto.IsDir(i.Type) && i.NLink == 2 {
i.NLink--
@ -388,6 +391,7 @@ func (i *Inode) DecNLink() {
if i.NLink > 0 {
i.NLink--
}
i.ModifyTime = mtime
i.Unlock()
}

5
storage/extent.go
View File

@ -17,8 +17,6 @@ package storage
import (
"encoding/binary"
"fmt"
"github.com/chubaofs/chubaofs/util"
"github.com/chubaofs/chubaofs/util/log"
"hash/crc32"
"io"
"math"
@ -28,6 +26,9 @@ import (
"sync/atomic"
"syscall"
"time"
"github.com/chubaofs/chubaofs/util"
"github.com/chubaofs/chubaofs/util/log"
)
const (

25
storage/fallocate_darwin.go
View File

@ -1,6 +1,27 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package storage
import (
"syscall"
"github.com/chubaofs/chubaofs/util/log"
)
func fallocate(fd int, mode uint32, off int64, len int64) (err error) {
panic("Fallocate is required")
return nil
// system call 'fallocate' is not supported in Darwin(Apple MacOS).
log.LogWarnf("fallocate: not supported in Darwin(Apple MacOS) operating system")
return syscall.ENOSYS
}

14
storage/fallocate_linux.go
View File

@ -1,3 +1,17 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package storage
import "syscall"

25
storage/fallocate_windows.go
View File

@ -1,6 +1,27 @@
// Copyright 2018 The Chubao Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
package storage
import (
"syscall"
"github.com/chubaofs/chubaofs/util/log"
)
func fallocate(fd int, mode uint32, off int64, len int64) (err error) {
panic("Fallocate is required")
return nil
// system call 'fallocate' is not supported in Microsoft Windows.
log.LogWarnf("fallocate: not supported in Microsoft Windows operating system")
return syscall.ENOSYS
}

3
storage/persistence_crc.go
View File

@ -16,9 +16,10 @@ package storage
import (
"encoding/binary"
"sync/atomic"
"github.com/chubaofs/chubaofs/util"
"github.com/chubaofs/chubaofs/util/log"
"sync/atomic"
)
type BlockCrc struct {

37
util/exporter/consul_register.go
View File

@ -15,13 +15,15 @@
package exporter
import (
"bytes"
"encoding/json"
"fmt"
"net"
"net/http"
"os"
"time"
"github.com/chubaofs/chubaofs/util/log"
"github.com/parnurzeal/gorequest"
)
const (
@ -34,11 +36,11 @@ const (
* optional for user when set prometheus exporter
*/
type ConsulRegisterInfo struct {
Name string
ID string
Address string
Port int64
Tags []string
Name string `json:"Name"`
ID string `json:"ID"`
Address string `json:"Address"`
Port int64 `json:"Port"`
Tags []string `json:"Tags"`
}
// get consul id
@ -50,7 +52,7 @@ func RegisterConsul(addr, app, role, cluster string, port int64) {
if len(addr) <= 0 {
return
}
log.LogInfo("consul register enable ", addr)
log.LogDebugf("consul register enable %v", addr)
ticker := time.NewTicker(RegisterPeriod)
defer func() {
if err := recover(); err != nil {
@ -95,7 +97,7 @@ func SendRegisterReq(addr string, app string, role string, cluster string, port
}
id := GetConsulId(app, role, host, port)
url := addr + RegisterPath
resp, body, errs := gorequest.New().Put(url).SendMap(ConsulRegisterInfo{
cInfo := &ConsulRegisterInfo{
Name: app,
ID: id,
Address: host,
@ -105,8 +107,21 @@ func SendRegisterReq(addr string, app string, role string, cluster string, port
"role=" + role,
"cluster=" + cluster,
},
}).End()
if errs != nil {
log.LogErrorf("Error on register consul resp: %v, body: %v", body, resp)
}
client := &http.Client{}
cInfoBytes, err1 := json.Marshal(cInfo)
if err1 != nil {
log.LogErrorf("marshal error, %v", err1.Error())
return
}
req, err2 := http.NewRequest(http.MethodPut, url, bytes.NewBuffer(cInfoBytes))
if err2 != nil {
log.LogErrorf("new request error, %v", err2.Error())
return
}
req.Header.Set("Content-Type", "application/json; charset=utf-8")
_, err3 := client.Do(req)
if err3 != nil {
log.LogErrorf("Error on register consul resp: %v, ", err3.Error())
}
}

24
vendor/github.com/parnurzeal/gorequest/.gitignore generated vendored
View File

@ -1,24 +0,0 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
.idea
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
tags

15
vendor/github.com/parnurzeal/gorequest/.travis.yml generated vendored
View File

@ -1,15 +0,0 @@
language: go
go:
- 1.7
- 1.6
- 1.5
- 1.4
- 1.3
- 1.2
install:
- go get -t -v ./...
notifications:
email:
recipients: parnurzeal@gmail.com
on_success: change
on_failure: always

123
vendor/github.com/parnurzeal/gorequest/CHANGELOG generated vendored
View File

@ -1,123 +0,0 @@
Changelog
=========
v0.2.15 (2016-08-30)
Features
* Allow float and boolean in Query()'s queryStruct @davyzhang
* Support Map in Query() @yangmls
* Support Map in Send() @longlongh4
* Document RedirectPolicy @codegoalie
* Enable Debug mode by ENV variable @heytitle
* Add Retry() @xild
Bug/Fixes
* Allow bodies with all methods @pkopac
* Change std "errors" pkg to "github.com/pkg/errors" @pkopac
v0.2.14 (2016-08-30)
Features
* Support multipart @fraenky8
* Support OPTIONS request @coderhaoxin
* Add EndStruct method @franciscocpg
* Add AsCurlCommand @jaytaylor
* Add CustomMethod @WaveCutz
* Add MakeRequest @quangbuule
Bug/Fixes
* Disable keep alive by default
v0.2.13 (2015-11-21)
Features
* Add DisableTransportSwap to stop gorequest from modifying Transport settings.
Note that this will effect many functions that modify gorequest's
tranport. (So, use with caution.) (Bug #47, PR #59 by @piotrmiskiewicz)
v0.2.12 (2015-11-21)
Features
* Add SetCurlCommand for printing comparable CURL command of the request
(PR #60 by @QuentinPerez)
v0.2.11 (2015-10-24)
Bug/Fixes
* Add support to Slice data structure (Bug #40, #42)
* Refactor sendStruct to be public function SendStruct
v0.2.10 (2015-10-24)
Bug/Fixes
* Fix incorrect text output in tests (PR #52 by @QuentinPerez)
* Fix Panic and runtime error properly (PR #53 by @QuentinPerez)
* Add support for "text/plain" and "application/xml" (PR #51 by
@smallnest)
* Content-Type header is also equivalent with Type function to identify
supported Gorequest's Target Type
v0.2.9 (2015-08-16)
Bug/Fixes
* ParseQuery accepts ; as a synonym for &. thus Gorequest Query won't
accept ; as in a query string. We add additional Param to solve this (PR
#43 by @6david9)
* AddCookies for bulk adding cookies (PR #46 by @pencil001)
v0.2.8 (2015-08-10)
Bug/Fixes
* Added SetDebug and SetLogger for debug mode (PR #28 by @dafang)
* Ensure the response Body is reusable (PR #37 by alaingilbert)
v0.2.7 (2015-07-11)
Bug/Fixes
* Incorrectly reset "Authentication" header (Hot fix by @na-ga PR #38 & Issue #39)
v0.2.6 (2015-07-10)
Features
* Added EndBytes (PR #30 by @jaytaylor)
v0.2.5 (2015-07-01)
Features
* Added Basic Auth support (pull request #24 by @dickeyxxx)
Bug/Fixes
* Fix #31 incorrect number conversion (PR #34 by @killix)
v0.2.4 (2015-04-13)
Features
* Query() now supports Struct as same as Send() (pull request #25 by @figlief)
v0.2.3 (2015-02-08)
Features
* Added Patch HTTP Method
Bug/Fixes
* Refactored testing code
v0.2.2 (2015-01-03)
Features
* Added TLSClientConfig for better control over tls
* Added AddCookie func to set "Cookie" field in request (pull request #17 by @austinov) - Issue #7
* Added CookieJar (pull request #15 by @kemadz)
v0.2.1 (2014-07-06)
Features
* Implemented timeout test
Bugs/Fixes
* Improved timeout feature by control over both dial + read/write timeout compared to previously controlling only dial connection timeout.
v0.2.0 (2014-06-13) - Send is now supporting Struct type as a parameter
v0.1.0 (2014-04-14) - Finished release with enough rich functions to do get, post, json and redirectpolicy

73
vendor/github.com/parnurzeal/gorequest/CONTRIBUTING.md generated vendored
View File

@ -1,73 +0,0 @@
# Contributing to GoRequest
Thanks for taking the time to contribute!!
GoRequest welcomes any kind of contributions including documentation, bug reports,
issues, feature requests, feature implementations, pull requests, helping to manage and answer issues, etc.
### Code Guidelines
To make the contribution process as seamless as possible, we ask for the following:
* Go ahead and fork the project and make your changes. We encourage pull requests to allow for review and discussion of code changes.
* When you’re ready to create a pull request, be sure to:
* Have test cases for the new code.
* Follow [GoDoc](https://blog.golang.org/godoc-documenting-go-code) guideline and always add documentation for new function/variable definitions.
* Run `go fmt`.
* Additonally, add documentation to README.md if you are adding new features or changing functionality.
* Squash your commits into a single commit. `git rebase -i`. It’s okay to force update your pull request with `git push -f`.
* Make sure `go test ./...` passes, and `go build` completes.
* Follow the **Git Commit Message Guidelines** below.
### Writing Commit Message
Follow this [blog article](http://chris.beams.io/posts/git-commit/). It is a good resource for learning how to write good commit messages,
the most important part being that each commit message should have a title/subject in imperative mood starting with a capital letter and no trailing period:
*"Return error when sending incorrect JSON format"*, **NOT** *"returning some error."*
Also, if your commit references one or more GitHub issues, always end your commit message body with *See #1234* or *Fixes #1234*.
Replace *1234* with the GitHub issue ID. The last example will close the issue when the commit is merged into *master*.
### Sending a Pull Request
Due to the way Go handles package imports, the best approach for working on a
fork is to use Git Remotes. You can follow the instructions below:
1. Get the latest sources:
```
go get -u -t github.com/parnurzeal/gorequest/...
```
1. Change to the GoRequest source directory:
```
cd $GOPATH/src/github.com/parnurzeal/gorequest
```
1. Create a new branch for your changes (the branch name is arbitrary):
```
git checkout -b issue_1234
```
1. After making your changes, commit them to your new branch:
```
git commit -a -v
```
1. Fork GoRequest in Github.
1. Add your fork as a new remote (the remote name, "fork" in this example, is arbitrary):
```
git remote add fork git://github.com/USERNAME/gorequest.git
```
1. Push the changes to your new remote:
```
git push --set-upstream fork issue_1234
```
1. You're now ready to submit a PR based upon the new branch in your forked repository.

20
vendor/github.com/parnurzeal/gorequest/LICENSE generated vendored
View File

@ -1,20 +0,0 @@
The MIT License (MIT)
Copyright (c) 2014 Theeraphol Wattanavekin
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

328
vendor/github.com/parnurzeal/gorequest/README.md generated vendored
View File

@ -1,328 +0,0 @@
GoRequest
=========
GoRequest -- Simplified HTTP client ( inspired by famous SuperAgent lib in Node.js )
![GopherGoRequest](https://raw.githubusercontent.com/parnurzeal/gorequest/gh-pages/images/Gopher_GoRequest_400x300.jpg)
#### "Shooting Requests like a Machine Gun" - Gopher
Sending request would never been fun and easier than this. It comes with lots of feature:
* Get/Post/Put/Head/Delete/Patch/Options
* Set - simple header setting
* JSON - made it simple with JSON string as a parameter
* Multipart-Support - send data and files as multipart request
* Proxy - sending request via proxy
* Timeout - setting timeout for a request
* TLSClientConfig - taking control over tls where at least you can disable security check for https
* RedirectPolicy
* Cookie - setting cookies for your request
* CookieJar - automatic in-memory cookiejar
* BasicAuth - setting basic authentication header
* more to come..
## Installation
```bash
$ go get github.com/parnurzeal/gorequest
```
## Documentation
See [Go Doc](http://godoc.org/github.com/parnurzeal/gorequest) or [Go Walker](http://gowalker.org/github.com/parnurzeal/gorequest) for usage and details.
## Status
[![Drone Build Status](https://drone.io/github.com/jmcvetta/restclient/status.png)](https://drone.io/github.com/parnurzeal/gorequest/latest)
[![Travis Build Status](https://travis-ci.org/parnurzeal/gorequest.svg?branch=master)](https://travis-ci.org/parnurzeal/gorequest)
## Why should you use GoRequest?
GoRequest makes thing much more simple for you, making http client more awesome and fun like SuperAgent + golang style usage.
This is what you normally do for a simple GET without GoRequest:
```go
resp, err := http.Get("http://example.com/")
```
With GoRequest:
```go
request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").End()
```
Or below if you don't want to reuse it for other requests.
```go
resp, body, errs := gorequest.New().Get("http://example.com/").End()
```
How about getting control over HTTP client headers, redirect policy, and etc. Things is getting more complicated in golang. You need to create a Client, setting header in different command, ... to do just only one __GET__
```go
client := &http.Client{
CheckRedirect: redirectPolicyFunc,
}
req, err := http.NewRequest("GET", "http://example.com", nil)
req.Header.Add("If-None-Match", `W/"wyzzy"`)
resp, err := client.Do(req)
```
Why making things ugly while you can just do as follows:
```go
request := gorequest.New()
resp, body, errs := request.Get("http://example.com").
RedirectPolicy(redirectPolicyFunc).
Set("If-None-Match", `W/"wyzzy"`).
End()
```
__DELETE__, __HEAD__, __POST__, __PUT__, __PATCH__ are now supported and can be used the same way as __GET__:
```go
request := gorequest.New()
resp, body, errs := request.Post("http://example.com").End()
// PUT -> request.Put("http://example.com").End()
// DELETE -> request.Delete("http://example.com").End()
// HEAD -> request.Head("http://example.com").End()
// ANYTHING -> request.CustomMethod("TRACE", "http://example.com").End()
```
### JSON
For a __JSON POST__ with standard libraries, you might need to marshal map data structure to json format, setting header to 'application/json' (and other headers if you need to) and declare http.Client. So, you code become longer and hard to maintain:
```go
m := map[string]interface{}{
"name": "backy",
"species": "dog",
}
mJson, _ := json.Marshal(m)
contentReader := bytes.NewReader(mJson)
req, _ := http.NewRequest("POST", "http://example.com", contentReader)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Notes","GoRequest is coming!")
client := &http.Client{}
resp, _ := client.Do(req)
```
Compared to our GoRequest version, JSON is for sure a default. So, it turns out to be just one simple line!:
```go
request := gorequest.New()
resp, body, errs := request.Post("http://example.com").
Set("Notes","gorequst is coming!").
Send(`{"name":"backy", "species":"dog"}`).
End()
```
Moreover, it also supports struct type. So, you can have a fun __Mix & Match__ sending the different data types for your request:
```go
type BrowserVersionSupport struct {
Chrome string
Firefox string
}
ver := BrowserVersionSupport{ Chrome: "37.0.2041.6", Firefox: "30.0" }
request := gorequest.New()
resp, body, errs := request.Post("http://version.com/update").
Send(ver).
Send(`{"Safari":"5.1.10"}`).
End()
```
Not only for Send() but Query() is also supported. Just give it a try! :)
## Callback
Moreover, GoRequest also supports callback function. This gives you much more flexibility on using it. You can use it any way to match your own style!
Let's see a bit of callback example:
```go
func printStatus(resp gorequest.Response, body string, errs []error){
fmt.Println(resp.Status)
}
gorequest.New().Get("http://example.com").End(printStatus)
```
## Multipart/Form-Data
You can specify the content-type of the request to type `multipart` to send all data as `multipart/form-data`. This feature also allows you to send (multiple) files! Check the examples below!
```go
gorequest.New().Post("http://example.com/").
Type("multipart").
Send(`{"query1":"test"}`).
End()
```
The `SendFile` function accepts `strings` as path to a file, `[]byte` slice or even a `os.File`! You can also combine them to send multiple files with either custom name and/or custom fieldname:
```go
f, _ := filepath.Abs("./file2.txt")
bytesOfFile, _ := ioutil.ReadFile(f)
gorequest.New().Post("http://example.com/").
Type("multipart").
SendFile("./file1.txt").
SendFile(bytesOfFile, "file2.txt", "my_file_fieldname").
End()
```
Check the docs for `SendFile` to get more information about the types of arguments.
## Proxy
In the case when you are behind proxy, GoRequest can handle it easily with Proxy func:
```go
request := gorequest.New().Proxy("http://proxy:999")
resp, body, errs := request.Get("http://example-proxy.com").End()
// To reuse same client with no_proxy, use empty string:
resp, body, errs = request.Proxy("").Get("http://example-no-proxy.com").End()
```
## Basic Authentication
To add a basic authentication header:
```go
request := gorequest.New().SetBasicAuth("username", "password")
resp, body, errs := request.Get("http://example-proxy.com").End()
```
## Timeout
Timeout can be set in any time duration using time package:
```go
request := gorequest.New().Timeout(2*time.Millisecond)
resp, body, errs:= request.Get("http://example.com").End()
```
Timeout func defines both dial + read/write timeout to the specified time parameter.
## EndBytes
Thanks to @jaytaylor, we now have EndBytes to use when you want the body as bytes.
The callbacks work the same way as with `End`, except that a byte array is used instead of a string.
```go
resp, bodyBytes, errs := gorequest.New().Get("http://example.com/").EndBytes()
```
## EndStruct
We now have EndStruct to use when you want the body as struct.
The callbacks work the same way as with `End`, except that a struct is used instead of a string.
Supposing the URL **http://example.com/** returns the body `{"hey":"you"}`
```go
heyYou struct {
Hey string `json:"hey"`
}
var heyYou heyYou
resp, _, errs := gorequest.New().Get("http://example.com/").EndStruct(&heyYou)
```
## Retry
Supposing you need retry 3 times, with 5 seconds between each attempt when gets a BadRequest or a InternalServerError
```go
request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").
Retry(3, 5 * time.seconds, http.StatusBadRequest, http.StatusInternalServerError).
End()
```
## Handling Redirects
Redirects can be handled with RedirectPolicy which behaves similarly to
net/http Client's [CheckRedirect
function](https://golang.org/pkg/net/http#Client). Simply specify a function
which takes the Request about to be made and a slice of previous Requests in
order of oldest first. When this function returns an error, the Request is not
made.
For example to redirect only to https endpoints:
```go
request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").
RedirectPolicy(func(req Request, via []*Request) error {
if req.URL.Scheme != "https" {
return http.ErrUseLastResponse
}
}).
End()
```
## Debug
For debugging, GoRequest leverages `httputil` to dump details of every request/response. (Thanks to @dafang)
You can just use `SetDebug` or environment variable `GOREQUEST_DEBUG=0|1` to enable/disable debug mode and `SetLogger` to set your own choice of logger.
Thanks to @QuentinPerez, we can see even how gorequest is compared to CURL by using `SetCurlCommand`.
## Noted
As the underlying gorequest is based on http.Client in most use cases, gorequest.New() should be called once and reuse gorequest as much as possible.
## Contributing to GoRequest:
If you find any improvement or issue you want to fix, feel free to send me a pull request with testing.
Thanks to all contributors thus far:
| Contributors |
|---------------------------------------|
| https://github.com/alaingilbert |
| https://github.com/austinov |
| https://github.com/coderhaoxin |
| https://github.com/codegoalie |
| https://github.com/dafang |
| https://github.com/davyzhang |
| https://github.com/dickeyxxx |
| https://github.com/figlief |
| https://github.com/fraenky8 |
| https://github.com/franciscocpg |
| https://github.com/heytitle |
| https://github.com/hownowstephen |
| https://github.com/kemadz |
| https://github.com/killix |
| https://github.com/jaytaylor |
| https://github.com/na-ga |
| https://github.com/piotrmiskiewicz |
| https://github.com/pencil001 |
| https://github.com/pkopac |
| https://github.com/quangbuule |
| https://github.com/QuentinPerez |
| https://github.com/smallnest |
| https://github.com/WaveCutz |
| https://github.com/xild |
| https://github.com/yangmls |
| https://github.com/6david9 |
Also, co-maintainer is needed here. If anyone is interested, please email me (parnurzeal at gmail.com)
## Credits
* Renee French - the creator of Gopher mascot
* [Wisi Mongkhonsrisawat](https://www.facebook.com/puairw) for providing an awesome GoRequest's Gopher image :)
## License
GoRequest is MIT License.

1238
vendor/github.com/parnurzeal/gorequest/gorequest.go generated vendored
View File

File diff suppressed because it is too large Load Diff

259
vendor/github.com/prometheus/procfs/ipvs.go generated vendored
View File

@ -1,259 +0,0 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package procfs
import (
"bufio"
"encoding/hex"
"errors"
"fmt"
"io"
"io/ioutil"
"net"
"os"
"strconv"
"strings"
)
// IPVSStats holds IPVS statistics, as exposed by the kernel in `/proc/net/ip_vs_stats`.
type IPVSStats struct {
// Total count of connections.
Connections uint64
// Total incoming packages processed.
IncomingPackets uint64
// Total outgoing packages processed.
OutgoingPackets uint64
// Total incoming traffic.
IncomingBytes uint64
// Total outgoing traffic.
OutgoingBytes uint64
}
// IPVSBackendStatus holds current metrics of one virtual / real address pair.
type IPVSBackendStatus struct {
// The local (virtual) IP address.
LocalAddress net.IP
// The remote (real) IP address.
RemoteAddress net.IP
// The local (virtual) port.
LocalPort uint16
// The remote (real) port.
RemotePort uint16
// The local firewall mark
LocalMark string
// The transport protocol (TCP, UDP).
Proto string
// The current number of active connections for this virtual/real address pair.
ActiveConn uint64
// The current number of inactive connections for this virtual/real address pair.
InactConn uint64
// The current weight of this virtual/real address pair.
Weight uint64
}
// NewIPVSStats reads the IPVS statistics.
func NewIPVSStats() (IPVSStats, error) {
fs, err := NewFS(DefaultMountPoint)
if err != nil {
return IPVSStats{}, err
}
return fs.NewIPVSStats()
}
// NewIPVSStats reads the IPVS statistics from the specified `proc` filesystem.
func (fs FS) NewIPVSStats() (IPVSStats, error) {
file, err := os.Open(fs.Path("net/ip_vs_stats"))
if err != nil {
return IPVSStats{}, err
}
defer file.Close()
return parseIPVSStats(file)
}
// parseIPVSStats performs the actual parsing of `ip_vs_stats`.
func parseIPVSStats(file io.Reader) (IPVSStats, error) {
var (
statContent []byte
statLines []string
statFields []string
stats IPVSStats
)
statContent, err := ioutil.ReadAll(file)
if err != nil {
return IPVSStats{}, err
}
statLines = strings.SplitN(string(statContent), "\n", 4)
if len(statLines) != 4 {
return IPVSStats{}, errors.New("ip_vs_stats corrupt: too short")
}
statFields = strings.Fields(statLines[2])
if len(statFields) != 5 {
return IPVSStats{}, errors.New("ip_vs_stats corrupt: unexpected number of fields")
}
stats.Connections, err = strconv.ParseUint(statFields[0], 16, 64)
if err != nil {
return IPVSStats{}, err
}
stats.IncomingPackets, err = strconv.ParseUint(statFields[1], 16, 64)
if err != nil {
return IPVSStats{}, err
}
stats.OutgoingPackets, err = strconv.ParseUint(statFields[2], 16, 64)
if err != nil {
return IPVSStats{}, err
}
stats.IncomingBytes, err = strconv.ParseUint(statFields[3], 16, 64)
if err != nil {
return IPVSStats{}, err
}
stats.OutgoingBytes, err = strconv.ParseUint(statFields[4], 16, 64)
if err != nil {
return IPVSStats{}, err
}
return stats, nil
}
// NewIPVSBackendStatus reads and returns the status of all (virtual,real) server pairs.
func NewIPVSBackendStatus() ([]IPVSBackendStatus, error) {
fs, err := NewFS(DefaultMountPoint)
if err != nil {
return []IPVSBackendStatus{}, err
}
return fs.NewIPVSBackendStatus()
}
// NewIPVSBackendStatus reads and returns the status of all (virtual,real) server pairs from the specified `proc` filesystem.
func (fs FS) NewIPVSBackendStatus() ([]IPVSBackendStatus, error) {
file, err := os.Open(fs.Path("net/ip_vs"))
if err != nil {
return nil, err
}
defer file.Close()
return parseIPVSBackendStatus(file)
}
func parseIPVSBackendStatus(file io.Reader) ([]IPVSBackendStatus, error) {
var (
status []IPVSBackendStatus
scanner = bufio.NewScanner(file)
proto string
localMark string
localAddress net.IP
localPort uint16
err error
)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) == 0 {
continue
}
switch {
case fields[0] == "IP" || fields[0] == "Prot" || fields[1] == "RemoteAddress:Port":
continue
case fields[0] == "TCP" || fields[0] == "UDP":
if len(fields) < 2 {
continue
}
proto = fields[0]
localMark = ""
localAddress, localPort, err = parseIPPort(fields[1])
if err != nil {
return nil, err
}
case fields[0] == "FWM":
if len(fields) < 2 {
continue
}
proto = fields[0]
localMark = fields[1]
localAddress = nil
localPort = 0
case fields[0] == "->":
if len(fields) < 6 {
continue
}
remoteAddress, remotePort, err := parseIPPort(fields[1])
if err != nil {
return nil, err
}
weight, err := strconv.ParseUint(fields[3], 10, 64)
if err != nil {
return nil, err
}
activeConn, err := strconv.ParseUint(fields[4], 10, 64)
if err != nil {
return nil, err
}
inactConn, err := strconv.ParseUint(fields[5], 10, 64)
if err != nil {
return nil, err
}
status = append(status, IPVSBackendStatus{
LocalAddress: localAddress,
LocalPort: localPort,
LocalMark: localMark,
RemoteAddress: remoteAddress,
RemotePort: remotePort,
Proto: proto,
Weight: weight,
ActiveConn: activeConn,
InactConn: inactConn,
})
}
}
return status, nil
}
func parseIPPort(s string) (net.IP, uint16, error) {
var (
ip net.IP
err error
)
switch len(s) {
case 13:
ip, err = hex.DecodeString(s[0:8])
if err != nil {
return nil, 0, err
}
case 46:
ip = net.ParseIP(s[1:40])
if ip == nil {
return nil, 0, fmt.Errorf("invalid IPv6 address: %s", s[1:40])
}
default:
return nil, 0, fmt.Errorf("unexpected IP:Port: %s", s)
}
portString := s[len(s)-4:]
if len(portString) != 4 {
return nil, 0, fmt.Errorf("unexpected port string format: %s", portString)
}
port, err := strconv.ParseUint(portString, 16, 16)
if err != nil {
return nil, 0, err
}
return ip, uint16(port), nil
}