mgr/dashboard/api: generate offline API docs

Generate Sphinx documentation from OpenAPI YAML spec:
- Fixed Docs controller doc generation
- Now dashboard Python doesn't fails if dashboard/frontend/dist doesn't exist
- OpenAPI added to @ceph/api CODEOWNERS
- Renamed Ceph-Dashboard API to Ceph REST or RESTful API.
- OpenAPI Docs: fixed decorators and docstrings.
- Sphinx Docs: updated dashboard and dev ones.

Co-authored-by: Ernesto Puerta <epuertat@redhat.com>
Fixes: https://tracker.ceph.com/issues/45863
Signed-off-by: Courtney Caldwell <ccaldwel@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
This commit is contained in:
Courtney Caldwell 2020-09-08 11:06:33 -04:00 committed by Ernesto Puerta
parent fa83dbd76e
commit ca2261781e
No known key found for this signature in database
GPG Key ID: B4F6FFCB7A8384ED
23 changed files with 9345 additions and 63 deletions

1
.github/CODEOWNERS vendored
View File

@ -8,6 +8,7 @@
/doc/mgr/dashboard.rst @ceph/dashboard
# Dashboard API team
/src/pybind/mgr/dashboard/openapi.yaml @ceph/api
/src/pybind/mgr/dashboard/controllers @ceph/api
/src/pybind/mgr/dashboard/frontend/src/app/shared/api @ceph/api
/src/pybind/mgr/dashboard/run-backend-api-tests.sh @ceph/api

View File

@ -8,3 +8,4 @@ sphinx-autodoc-typehints
sphinx-prompt
Sphinx-Substitution-Extensions
typed-ast
sphinxcontrib-openapi

View File

@ -2,6 +2,12 @@
API Documentation
===================
Ceph RESTful API
================
See `Ceph REST API`_.
.. _Ceph REST API: ../mgr/ceph_api/
Ceph Storage Cluster APIs
=========================

View File

@ -60,6 +60,7 @@ extensions = [
'breathe',
'edit_on_github',
'ceph_releases',
'sphinxcontrib.openapi'
]
ditaa = shutil.which("ditaa")

View File

@ -1996,9 +1996,30 @@ Usage example:
REST API documentation
~~~~~~~~~~~~~~~~~~~~~~
There is an automatically generated Swagger UI page for documentation of the REST
API endpoints.However, by default it is not very detailed. There are two
decorators that can be used to add more information:
Ceph-Dashboard provides two types of documentation for the **Ceph RESTful API**:
* **Static documentation**: available at :ref:`mgr-ceph-api`. This comes from a versioned specification located at ``src/pybind/mgr/dashboard/openapi.yaml``.
* **Interactive documentation**: available from a running Ceph-Dashboard instance (top-right ``?`` icon > API Docs).
If changes are made to the ``controllers/`` directory, it's very likely that
they will result in changes to the generated OpenAPI specification. For that
reason, a checker has been implemented to block unintended changes. This check
is automatically triggered by the Pull Request CI (``make check``) and can be
also manually invoked: ``tox -e openapi-check``.
If that checker failed, it means that the current Pull Request is modifying the
Ceph API and therefore:
#. The versioned OpenAPI specification should be updated explicitly: ``tox -e openapi-fix``.
#. The team @ceph/api will be requested for reviews (this is automated via Github CODEOWNERS), in order to asses the impact of changes.
Additionally, Sphinx documentation can be generated from the OpenAPI
specification with ``tox -e openapi-doc``.
The Ceph RESTful OpenAPI specification is dynamically generated from the
``Controllers`` in ``controllers/`` directory. However, by default it is not
very detailed, so there are two decorators that can and should be used to add
more information:
* ``@EndpointDoc()`` for documentation of endpoints. It has four optional arguments
(explained below): ``description``, ``group``, ``parameters`` and
@ -2312,5 +2333,3 @@ plugin could be rewritten like this:
return self.mute
return [MuteController]

View File

@ -0,0 +1,90 @@
.. _mgr-ceph-api:
================
Ceph RESTful API
================
Introduction
============
The **Ceph RESTful API** (henceforth **Ceph API**) is provided by the
:ref:`mgr-dashboard` module. The Ceph API
service is available at the same URL as the regular Ceph Dashboard, under the
``/api`` base path (please refer to :ref:`dashboard-host-name-and-port`)::
http://<server_addr>:<server_port>/api
or, if HTTPS is enabled (please refer to :ref:`dashboard-ssl-tls-support`)::
https://<server_addr>:<ssl_server_port>/api
The Ceph API leverages the following standards:
* `HTTP 1.1 <https://tools.ietf.org/html/rfc7231>`_ for API syntax and semantics,
* `JSON <https://tools.ietf.org/html/rfc8259>`_ for content encoding,
* `HTTP Content Negotiation <https://tools.ietf.org/html/rfc2295>`_ and `MIME <https://tools.ietf.org/html/rfc2045>`_ for versioning,
* `OAuth 2.0 <https://tools.ietf.org/html/rfc6750>`_ and `JWT <https://tools.ietf.org/html/rfc7519>`_ for authentication and authorization.
.. warning::
Some endpoints are still under active development, and should be carefully
used since new Ceph releases could bring backward incompatible changes.
Authentication and Authorization
================================
Requests to the Ceph API pass through two access control checkpoints:
* **Authentication**: ensures that the request is performed on behalf of an existing and valid user account.
* **Authorization**: ensures that the previously authenticated user can in fact perform a specific action (create, read, update or delete) on the target endpoint.
So, prior to start consuming the Ceph API, a valid JSON Web Token (JWT) has to
be obtained, and it may then be reused for subsequent requests. The
``/api/auth`` endpoint will provide the valid token:
.. code-block:: sh
$ curl -X POST "https://example.com:8443/api/auth" \
-H "Accept: application/vnd.ceph.api.v1.0+json" \
-H "Content-Type: application/json" \
-d '{"username": <username>, "password": <password>}'
{ "token": "<redacted_token>", ...}
The token obtained must be passed together with every API request in the
``Authorization`` HTTP header::
curl -H "Authorization: Bearer <token>" ...
Authentication and authorization can be further configured from the
Ceph CLI, the Ceph-Dashboard UI and the Ceph API itself (please refer to
:ref:`dashboard-user-role-management`).
Versioning
==========
One of the main goals of the Ceph API is to keep a stable interface. For this
purpose, Ceph API is built upon the following principles:
* **Mandatory**: in order to avoid implicit defaults, all endpoints require an explicit default version (starting with ``1.0``).
* **Per-endpoint**: as this API wraps many different Ceph components, this allows for a finer-grained change control.
* **Content/MIME Type**: the version expected from a specific endpoint is stated by the ``Accept: application/vnd.ceph.api.v<major>.<minor>+json`` HTTP header. If the current Ceph API server is not able to address that specific major version, a `415 - Unsupported Media Type <https://tools.ietf.org/html/rfc7231#section-6.5.13>`_ response will be returned.
* **Semantic Versioning**: with a ``major.minor`` version:
* Major changes are backward incompatible: they might result in non-additive changes to the request and/or response formats of a specific endpoint.
* Minor changes are backward/forward compatible: they basically consists of additive changes to the request or response formats of a specific endpoint.
An example:
.. code-block:: bash
$ curl -X GET "https://example.com:8443/api/osd" \
-H "Accept: application/vnd.ceph.api.v1.0+json" \
-H "Authorization: Bearer <token>"
Specification
=============
.. openapi:: ../../../src/pybind/mgr/dashboard/openapi.yaml
:group:
:examples:
:encoding: utf-8

View File

@ -149,7 +149,7 @@ Status
subpage containing a list of all OSDs and related management actions.
* **Managers**: Displays the total number of active and standby Ceph Manager
daemons (ceph-mgr) running alongside monitor daemons.
* **Object Gateway**: Displays the total number of active object gateways and
* **Object Gateway**: Displays the total number of active object gateways and
provides a link to the subpage displaying a list of all object gateway daemons.
* **Metadata Servers**: Displays total number of active and standby metadata
servers daemons for CephFS (ceph-mds).
@ -290,6 +290,8 @@ wanted or required. See :ref:`dashboard-proxy-configuration` for more details.
$ ceph mgr module disable dashboard
$ ceph mgr module enable dashboard
.. _dashboard-host-name-and-port:
Host Name and Port
^^^^^^^^^^^^^^^^^^

View File

@ -29,6 +29,7 @@ sensible.
Writing modules <modules>
Writing orchestrator plugins <orchestrator_modules>
Dashboard module <dashboard>
Ceph RESTful API <ceph_api/index>
Alerts module <alerts>
DiskPrediction module <diskprediction>
Local pool module <localpool>

View File

@ -107,5 +107,5 @@ add_dependencies(tests mgr-dashboard-frontend-build)
if(WITH_TESTS)
include(AddCephTest)
add_tox_test(mgr-dashboard TOX_ENVS lint check)
add_tox_test(mgr-dashboard TOX_ENVS lint check openapi-check)
endif()

View File

@ -37,6 +37,11 @@ else:
logging.root.handlers[0].setLevel(logging.DEBUG)
os.environ['PATH'] = '{}:{}'.format(os.path.abspath('../../../../build/bin'),
os.environ['PATH'])
import sys
# Used to allow the running of a tox-based yml doc generator from the dashboard directory
if os.path.abspath(sys.path[0]) == os.getcwd():
sys.path.pop(0)
from tests import mock # type: ignore

View File

@ -188,6 +188,11 @@ class UiApiController(Controller):
security_scope=security_scope,
secure=secure)
def __call__(self, cls):
cls = super(UiApiController, self).__call__(cls)
cls._api_endpoint = False
return cls
def Endpoint(method=None, path=None, path_params=None, query_params=None, # noqa: N802
json_response=True, proxy=False, xml=False):
@ -582,7 +587,8 @@ class BaseController(object):
@property
def is_api(self):
return hasattr(self.ctrl, '_api_endpoint')
# changed from hasattr to getattr: some ui-based api inherit _api_endpoint
return getattr(self.ctrl, '_api_endpoint', False)
@property
def is_secure(self):

View File

@ -359,7 +359,7 @@ class CephFS(RESTController):
List directories of specified path.
:param fs_id: The filesystem identifier.
:param path: The path where to start listing the directory content.
Defaults to '/' if not set.
Defaults to '/' if not set.
:type path: str | bytes
:param depth: The number of steps to go down the directory tree.
:type depth: int | str
@ -378,7 +378,7 @@ class CephFS(RESTController):
"""
Transforms input path parameter of ls_dir methods (api and ui-api).
:param path: The path where to start listing the directory content.
Defaults to '/' if not set.
Defaults to '/' if not set.
:type path: str | bytes
:return: Normalized path or root path
:return: str
@ -436,7 +436,7 @@ class CephFS(RESTController):
:param fs_id: The filesystem identifier.
:param path: The path of the directory/file.
:return: Returns a dictionary containing 'max_bytes'
and 'max_files'.
and 'max_files'.
:rtype: dict
"""
cfs = self._cephfs_instance(fs_id)
@ -449,9 +449,8 @@ class CephFS(RESTController):
Create a snapshot.
:param fs_id: The filesystem identifier.
:param path: The path of the directory.
:param name: The name of the snapshot. If not specified,
a name using the current time in RFC3339 UTC format
will be generated.
:param name: The name of the snapshot. If not specified, a name using the
current time in RFC3339 UTC format will be generated.
:return: The name of the snapshot.
:rtype: str
"""
@ -517,7 +516,7 @@ class CephFsUi(CephFS):
:param fs_id: The filesystem identifier.
:type fs_id: int | str
:param path: The path where to start listing the directory content.
Defaults to '/' if not set.
Defaults to '/' if not set.
:type path: str | bytes
:param depth: The number of steps to go down the directory tree.
:type depth: int | str

View File

@ -9,8 +9,7 @@ from . import Controller, BaseController, Endpoint, ENDPOINT_MAP, \
allow_empty_body
from .. import mgr
from ..tools import str_to_bool
NO_DESCRIPTION_AVAILABLE = "*No description available*"
logger = logging.getLogger('controllers.docs')
@ -44,7 +43,7 @@ class Docs(BaseController):
if tag_name not in tag_map or not tag_map[tag_name]:
tag_map[tag_name] = tag_descr
tags = [{'name': k, 'description': v if v else "*No description available*"}
tags = [{'name': k, 'description': v if v else NO_DESCRIPTION_AVAILABLE}
for k, v in tag_map.items()]
tags.sort(key=lambda e: e['name'])
return tags
@ -231,18 +230,15 @@ class Docs(BaseController):
_type = cls._type_to_str(param['type'])
else:
_type = cls._gen_type(param)
if 'description' in param:
descr = param['description']
else:
descr = "*No description available*"
res = {
'name': param['name'],
'in': location,
'schema': {
'type': _type
},
'description': descr
}
if param.get('description'):
res['description'] = param['description']
if param['required']:
res['required'] = True
elif param['default'] is None:
@ -273,7 +269,7 @@ class Docs(BaseController):
method = endpoint.method
func = endpoint.func
summary = "No description available"
summary = ''
resp = {}
p_info = []
if hasattr(func, 'doc_info'):
@ -293,11 +289,12 @@ class Docs(BaseController):
methods[method.lower()] = {
'tags': [cls._get_tag(endpoint)],
'summary': summary,
'description': func.__doc__,
'parameters': params,
'responses': cls._gen_responses(method, resp)
}
if summary:
methods[method.lower()]['summary'] = summary
if method.lower() in ['post', 'put']:
if endpoint.body_params:
@ -322,40 +319,34 @@ class Docs(BaseController):
return paths
def _gen_spec(self, all_endpoints=False, base_url=""):
@classmethod
def _gen_spec(cls, all_endpoints=False, base_url="", offline=False):
if all_endpoints:
base_url = ""
host = cherrypy.request.base
host = host[host.index(':')+3:]
host = cherrypy.request.base.split('://', 1)[1] if not offline else 'example.com'
logger.debug("Host: %s", host)
paths = self._gen_paths(all_endpoints)
paths = cls._gen_paths(all_endpoints)
if not base_url:
base_url = "/"
scheme = 'https'
ssl = str_to_bool(mgr.get_localized_module_option('ssl', True))
if not ssl:
scheme = 'http'
scheme = 'https' if offline or mgr.get_localized_module_option('ssl') else 'http'
spec = {
'openapi': "3.0.0",
'info': {
'description': "Please note that this API is not an official "
"Ceph REST API to be used by third-party "
"applications. It's primary purpose is to serve"
" the requirements of the Ceph Dashboard and is"
" subject to change at any time. Use at your "
"own risk.",
'description': "This is the official Ceph REST API",
'version': "v1",
'title': "Ceph-Dashboard REST API"
'title': "Ceph REST API"
},
'host': host,
'basePath': base_url,
'servers': [{'url': "{}{}".format(cherrypy.request.base, base_url)}],
'tags': self._gen_tags(all_endpoints),
'servers': [{'url': "{}{}".format(
cherrypy.request.base if not offline else '',
base_url)}],
'tags': cls._gen_tags(all_endpoints),
'schemes': [scheme],
'paths': paths,
'components': {
@ -461,3 +452,30 @@ class Docs(BaseController):
@allow_empty_body
def _with_token(self, token, all_endpoints=False):
return self._swagger_ui_page(all_endpoints, token)
if __name__ == "__main__":
import sys
import yaml
from . import generate_routes
def fix_null_descr(obj):
"""
A hot fix for errors caused by null description values when generating
static documentation: better fix would be default values in source
to be 'None' strings: however, decorator changes didn't resolve
"""
return {k: fix_null_descr(v) for k, v in obj.items() if v is not None} \
if isinstance(obj, dict) else obj
generate_routes("/api")
try:
with open(sys.argv[1], 'w') as f:
# pylint: disable=protected-access
yaml.dump(
fix_null_descr(Docs._gen_spec(all_endpoints=False, base_url="/", offline=True)),
f)
except IndexError:
sys.exit("Output file name missing; correct syntax is: `cmd <file.yml>`")
except IsADirectoryError:
sys.exit("Specified output is a directory; correct syntax is: `cmd <file.yml>`")

View File

@ -9,7 +9,6 @@ try:
from functools import lru_cache
except ImportError:
from ..plugins.lru_cache import lru_cache
import cherrypy
from cherrypy.lib.static import serve_file
@ -22,11 +21,16 @@ logger = logging.getLogger("controllers.home")
class LanguageMixin(object):
def __init__(self):
self.LANGUAGES = {
f
for f in os.listdir(mgr.get_frontend_path())
if os.path.isdir(os.path.join(mgr.get_frontend_path(), f))
}
try:
self.LANGUAGES = {
f
for f in os.listdir(mgr.get_frontend_path())
if os.path.isdir(os.path.join(mgr.get_frontend_path(), f))
}
except FileNotFoundError:
logger.exception("Build directory missing")
self.LANGUAGES = {}
self.LANGUAGES_PATH_MAP = {
f.lower(): {
'lang': f,
@ -41,7 +45,7 @@ class LanguageMixin(object):
'lang': self.LANGUAGES_PATH_MAP[lang]['lang'],
'path': self.LANGUAGES_PATH_MAP[lang]['path']
}
with open("{}/../package.json".format(mgr.get_frontend_path()),
with open(os.path.normpath("{}/../package.json".format(mgr.get_frontend_path())),
"r") as f:
config = json.load(f)
self.DEFAULT_LANGUAGE = config['config']['locale']

View File

@ -103,7 +103,7 @@ class Orchestrator(RESTController):
Identify a device by switching on the device light for N seconds.
:param hostname: The hostname of the device to process.
:param device: The device identifier to process, e.g. ``/dev/dm-0`` or
``ABC1234DEF567-1R1234_ABC8DE0Q``.
``ABC1234DEF567-1R1234_ABC8DE0Q``.
:param duration: The duration in seconds how long the LED should flash.
"""
orch = OrchClient.instance()

View File

@ -344,7 +344,10 @@ class RbdSnapshot(RESTController):
@ControllerDoc("RBD Trash Management API", "RbdTrash")
class RbdTrash(RESTController):
RESOURCE_ID = "image_id_spec"
rbd_inst = rbd.RBD()
def __init__(self):
super().__init__()
self.rbd_inst = rbd.RBD()
@ViewCache()
def _trash_pool_list(self, pool_name):
@ -432,7 +435,10 @@ class RbdTrash(RESTController):
@ApiController('/block/pool/{pool_name}/namespace', Scope.RBD_IMAGE)
@ControllerDoc("RBD Namespace Management API", "RbdNamespace")
class RbdNamespace(RESTController):
rbd_inst = rbd.RBD()
def __init__(self):
super().__init__()
self.rbd_inst = rbd.RBD()
def create(self, pool_name, namespace):
with mgr.rados.open_ioctx(pool_name) as ioctx:

View File

@ -54,7 +54,7 @@ class Settings(RESTController):
"""
Get the list of available options.
:param names: A comma separated list of option names that should
be processed. Defaults to ``None``.
be processed. Defaults to ``None``.
:type names: None|str
:return: A list of available options.
:rtype: list[dict]
@ -83,7 +83,7 @@ class Settings(RESTController):
Get the given option.
:param name: The name of the option.
:return: Returns a dict containing the name, type,
default value and current value of the given option.
default value and current value of the given option.
:rtype: dict
"""
return self._get(name)
@ -108,7 +108,7 @@ class StandardSettings(RESTController):
"""
Get various Dashboard related settings.
:return: Returns a dictionary containing various Dashboard
settings.
settings.
:rtype: dict
"""
return { # pragma: no cover - no complexity there

View File

@ -223,8 +223,8 @@ class Telemetry(RESTController):
:param enable: Enable or disable sending data
:type enable: bool
:param license_name: License string e.g. 'sharing-1-0' to
make sure the user is aware of and accepts the license
for sharing Telemetry data.
make sure the user is aware of and accepts the license
for sharing Telemetry data.
:type license_name: string
"""
if enable:

View File

@ -168,9 +168,9 @@ class UserPasswordPolicy(RESTController):
:param password: The password to validate.
:param username: The name of the user (optional).
:param old_password: The old password (optional).
:return: An object with the properties valid, credits and valuation.
'credits' contains the password complexity credits and
'valuation' the textual summary of the validation.
:return: An object with properties valid, credits and valuation.
'credits' contains the password complexity credits and
'valuation' the textual summary of the validation.
"""
result = {'valid': False, 'credits': 0, 'valuation': None}
try:

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1 @@
python3-saml

View File

@ -4,7 +4,6 @@ enum34; python_version<'3.4'
more-itertools
PyJWT
pyopenssl
python3-saml
requests
Routes
-e ../../../python-common

View File

@ -5,6 +5,8 @@ envlist =
fix,
check,
run,
apidocs,
openapi-{check,fix, doc}
skipsdist = true
minversion = 2.9.1
@ -32,6 +34,7 @@ deps =
deps =
{[base]deps}
{[base-test]deps}
-rrequirements-extra.txt
setenv =
CFLAGS = -DXMLSEC_NO_SIZE_T
PYTHONUNBUFFERED=1
@ -47,8 +50,7 @@ deps =
{[base]deps}
{[base-test]deps}
{[base-lint]deps}
# TODO: replace with allowlist_external tox=>16.1 (https://github.com/tox-dev/tox/pull/1601)
whitelist_externals = *
allowlist_externals = *
commands = {posargs}
[flake8]
@ -140,3 +142,27 @@ commands =
[testenv:check]
commands =
python ci/check_grafana_uids.py frontend/src/app ../../../../monitoring/grafana/dashboards
[testenv:openapi-{check,fix}]
basepython = python3
allowlist_externals = diff
description =
check: Ensure that auto-generated OpenAPI Specification matches the current version
fix: Update auto-generated OpenAPI Specification with the latest changes
deps =
{[base]deps}
{[base-test]deps}
setenv =
UNITTEST = true
PYTHONPATH=$PYTHONPATH:..
OPENAPI_FILE=openapi.yaml
check: OPENAPI_FILE_TMP={envtmpdir}/{env:OPENAPI_FILE}
commands =
python3 -m dashboard.controllers.docs {env:OPENAPI_FILE_TMP:{env:OPENAPI_FILE}}
check: diff {env:OPENAPI_FILE} {env:OPENAPI_FILE_TMP}
[testenv:openapi-doc]
description = Generate Sphinx documentation from OpenAPI specification
deps = -r../../../../admin/doc-requirements.txt
changedir = ../../../../doc
commands = sphinx-build -W -b html -c . -D suppress_warnings=ref.* -d {envtmpdir}/doctrees mgr/ceph_api {envtmpdir}/html