* Apple FR66 - Host Control Plane Status
* Apple FR66 - use Offline instead of Disconnected
* Apple FR66: fix smoke test test_router_host_control_state
* Apple FR66: reorder import
* Apple FR66: revert DetailsTab.vue and apply new changes
* Update PR: update en.json
* Update PR: add hostcontrolstate to routers/systemvms
* Update PR: test stop/start cloudstack-agent in smoke test
* Update PR: fix UI build error (The template root requires exactly one element vue/valid-template-root)
* Update PR: update message on ui
* Update PR: Disable rebootVM and create vm/volume snapshot for KVM VMs
* Update PR: add more unit tests
* UserData as first clas resource
* Few fixes
* Added userdata id in deploy VM flow
* Fixed userdata append scenario between template userdata and user provided userdata
* UI: added a new section for userdata
* Added userdata details to the deployVM cmd flow
* Write userdata details into metadata VR
* Updated template response and views
* UI: added userdata id to deloyvm wizard
* Added userdata list to deploy VM form
* Added userdata params to registeruserdata UI form
* Small fixes and added userdata to updateVM flow
* Fixed unit tests and imports
* Userdata Navigation in template view
* Added userdata denyoverride flag and userdata params section
* Added ToolTips and fixes deploy VM, register userdata form
* added userdata policy list to register template form
* Allow override append of userdata in deploy VM flow
* update userdata linking to template
* Few UI fixes in deploy VM and edit template form
* fixes in deploy VM form to support deny userdata policy
* Added unit tests for userdata
* Added unit tests for linking userdata to template
* Remove unused imports
* Move test file to proper files
* Fix unused imports
* Fix Userdata delete flow
* Few improvements in the code
* Adding marvin tests for userdata
* Fixed marvin tests for registerd userdata
* Added few more marvin tests for userdata
* Few code fixes
* Few more code fixes
* Added userdata details to register and upload iso forms
* Added userdata selection in deploy VM form for ISOs
* Add comments section to userdata
* Added new API to reset UserData of a VM along with UpdateVM API
* Added new UnitTests for resetVMUserData
* Added resetVMUserdata in UI for stopped VM
* Added blank values for userdata in edit template form
* Added ISO id to the linkuserdatatotemplate API
* Added validation to userdata params so that it wont contain any VR metadafile names
* Removed required param for iso id in linkuserdatatotemplate cmd
* Added length to userdata param
* remove delete cascade on user_vm and vm_template tables for userdata id foreign key reference.
* Fix custom userdata params for config drive by adding userdata file name and value metadata json
* Fix marvin test case
* added comments to marvin test cases
* Fix document link in UI
* Added a check while deleting the userdata, to see if no VMs are using the userdata
* Added unit tests
* Removed labels added during merge
* added success message for userdata registration
* Added Schema changes to 4160 to 4161 upgrade path
* Fixed imports and some errors
* Fix service offering uuid in mysql view
* UI changes wrt to 4.16 branch mostly related to vue2
* Some UI fixes
* Register userdata and update template form fixes
* Fixed compute.js related to userdata
* UI fixes and user_vm_view wrt sshkey
* Fixed update template form
* Fix deploy VM and userdata reset forms
* Fixed Register and upload template and ISO forms
* Fixed getting params for userdata from template in instance creation form
* Removed CloudZonesNetworkElement.java as part of rebase, which is actually removed in https://github.com/shapeblue/cloudstack-apple/pull/191
* Fixed userdata selection in deploy vm and reset userdata vm forms
* Fixed method calls after rebase
Add resource ID and resource type to event.
In UI, adds Events tab in resource view for the supporting resources.
Following SQL changes needed to support events with resource details in DB,
```
-- Alter event table to add resource_id and resource_type
ALTER TABLE `cloud`.`event`
ADD COLUMN `resource_id` bigint unsigned COMMENT 'ID of the resource associated with the even' AFTER `domain_id`,
ADD COLUMN `resource_type` varchar(32) COMMENT 'Account role in the project (Owner or Regular)' AFTER `resource_id`;
DROP VIEW IF EXISTS `cloud`.`event_view`;
CREATE VIEW `cloud`.`event_view` AS
SELECT
event.id,
event.uuid,
event.type,
event.state,
event.description,
event.resource_id,
event.resource_type,
event.created,
event.level,
event.parameters,
event.start_id,
eve.uuid start_uuid,
event.user_id,
event.archived,
event.display,
user.username user_name,
account.id account_id,
account.uuid account_uuid,
account.account_name account_name,
account.type account_type,
domain.id domain_id,
domain.uuid domain_uuid,
domain.name domain_name,
domain.path domain_path,
projects.id project_id,
projects.uuid project_uuid,
projects.name project_name
FROM
`cloud`.`event`
INNER JOIN
`cloud`.`account` ON event.account_id = account.id
INNER JOIN
`cloud`.`domain` ON event.domain_id = domain.id
INNER JOIN
`cloud`.`user` ON event.user_id = user.id
LEFT JOIN
`cloud`.`projects` ON projects.project_account_id = event.account_id
LEFT JOIN
`cloud`.`event` eve ON event.start_id = eve.id;
```
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* prevent role access escallation
* hierarchy issue fixed
* create api list in account manager for checking new account access
* full api list check
* strange role restriction removed for BareMetal
* add role check on upfdate account as well
* more selective use of api checkers
* error msg and var name
Co-authored-by: Daan Hoogland <dahn@onecht.net>
* Add the list of supported namespaces per document and refactor the disks extraction by using the namespaces
* Refactor matching the default OVF schema
* Move parser methods to a new class and refactor
* Fix import, unit tests
* Reduce indentation
* Address review comments
This adds a volume(primary) storage plugin for the Linstor SDS.
Currently it can create/delete/migrate volumes, snapshots should be possible,
but currently don't work for RAW volume types in cloudstack.
* plugin-storage-volume-linstor: notify libvirt guests about the resize
Added support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack (for KVM hypervisor) and enabled VM/Volume operations on that pool (using pool tag).
Please find more details in the FS here:
https://cwiki.apache.org/confluence/x/cDl4CQ
Documentation PR: apache/cloudstack-documentation#169
This enables support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack
Other improvements addressed in addition to PowerFlex/ScaleIO support:
- Added support for config drives in host cache for KVM
=> Changed configuration "vm.configdrive.primarypool.enabled" scope from Global to Zone level
=> Introduced new zone level configuration "vm.configdrive.force.host.cache.use" (default: false) to force host cache for config drives
=> Introduced new zone level configuration "vm.configdrive.use.host.cache.on.unsupported.pool" (default: true) to use host cache for config drives when storage pool doesn't support config drive
=> Added new parameter "host.cache.location" (default: /var/cache/cloud) in KVM agent.properties for specifying the host cache path and create config drives on the "/config" directory on the host cache path
=> Maintain the config drive location and use it when required on any config drive operation (migrate, delete)
- Detect virtual size from the template URL while registering direct download qcow2 (of KVM hypervisor) templates
- Updated full deployment destination for preparing the network(s) on VM start
- Propagate the direct download certificates uploaded to the newly added KVM hosts
- Discover the template size for direct download templates using any available host from the zones specified on template registration
=> When zones are not specified while registering template, template size discovery is performed using any available host, which is picked up randomly from one of the available zones
- Release the VM resources when VM is sync-ed to Stopped state on PowerReportMissing (after graceful period)
- Retry VM deployment/start when the host cannot grant access to volume/template
- Mark never-used or downloaded templates as Destroyed on deletion, without sending any DeleteCommand
=> Do not trigger any DeleteCommand for never-used or downloaded templates as these doesn't exist and cannot be deleted from the datastore
- Check the router filesystem is writable or not, before performing health checks
=> Introduce a new test "filesystem.writable.test" to check the filesystem is writable or not
=> The router health checks keeps the config info at "/var/cache/cloud" and updates the monitor results at "/root" for health checks, both are different partitions. So, test at both the locations.
=> Added new script: "filesystem_writable_check.py" at /opt/cloud/bin/ to check the filesystem is writable or not
- Fixed NPE issue, template is null for DATA disks. Copy template to target storage for ROOT disk (with template id), skip DATA disk(s)
* Addressed some issues for few operations on PowerFlex storage pool.
- Updated migration volume operation to sync the status and wait for migration to complete.
- Updated VM Snapshot naming, for uniqueness in ScaleIO volume name when more than one volume exists in the VM.
- Added sync lock while spooling managed storage template before volume creation from the template (non-direct download).
- Updated resize volume error message string.
- Blocked the below operations on PowerFlex storage pool:
-> Extract Volume
-> Create Snapshot for VMSnapshot
* Added the PowerFlex/ScaleIO client connection pool to manage the ScaleIO gateway clients, which uses a single gateway client per Powerflex/ScaleIO storage pool and renews it when the session token expires.
- The token is valid for 8 hours from the time it was created, unless there has been no activity for 10 minutes.
Reference: https://cpsdocs.dellemc.com/bundle/PF_REST_API_RG/page/GUID-92430F19-9F44-42B6-B898-87D5307AE59B.html
Other fixes included:
- Fail the VM deployment when the host specified in the deployVirtualMachine cmd is not in the right state (i.e. either Resource State is not Enabled or Status is not Up)
- Use the physical file size of the template to check the free space availability on the host, while downloading the direct download templates.
- Perform basic tests (for connectivity and file system) on router before updating the health check config data
=> Validate the basic tests (connectivity and file system check) on router
=> Cleanup the health check results when router is destroyed
* Updated PowerFlex/ScaleIO storage plugin version to 4.16.0.0
* UI Changes to support storage plugin for PowerFlex/ScaleIO storage pool.
- PowerFlex pool URL generated from the UI inputs(Gateway, Username, Password, Storage Pool) when adding "PowerFlex" Primary Storage
- Updated protocol to "custom" for PowerFlex provider
- Allow VM Snapshot for stopped VM on KVM hypervisor and PowerFlex/ScaleIO storage pool
and Minor improvements in PowerFlex/ScaleIO storage plugin code
* Added support for PowerFlex/ScaleIO volume migration across different PowerFlex storage instances.
- findStoragePoolsForMigration API returns PowerFlex pool(s) of different instance as suitable pool(s), for volume(s) on PowerFlex storage pool.
- Volume(s) with snapshots are not allowed to migrate to different PowerFlex instance.
- Volume(s) of running VM are not allowed to migrate to other PowerFlex storage pools.
- Volume migration from PowerFlex pool to Non-PowerFlex pool, and vice versa are not supported.
* Fixed change service offering smoke tests in test_service_offerings.py, test_vm_snapshots.py
* Added the PowerFlex/ScaleIO volume/snapshot name to the paths of respective CloudStack resources (Templates, Volumes, Snapshots and VM Snapshots)
* Added new response parameter “supportsStorageSnapshot” (true/false) to volume response, and Updated UI to hide the async backup option while taking snapshot for volume(s) with storage snapshot support.
* Fix to remove the duplicate zone wide pools listed while finding storage pools for migration
* Updated PowerFlex/ScaleIO volume migration checks and rollback migration on failure
* Fixed the PowerFlex/ScaleIO volume name inconsistency issue in the volume path after migration, due to rename failure
- Create a role from any of the existing role, using new parameter roleid in createRole API
- Import a role with its rules, using a new importRole API
- New default roles for Read-Only and Support Admin & User
- No modifications allowed for Default roles
- Cleaned up old NetApp APIs from role_permissions table.
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Refactor: Cleanup duplicate code
Make use of Java 8 default implementation in interfaces,
to remove code duplication between XxxCmd and XxxCmdAsAdmin.
Refactor checkFormat by pre-calculating the supported
extensions. Also make use of this in ImageStoreUtil.
Makes it easier to add new file and compression formats.
Problem: In Vmware, appliances that have options that are required to be answered before deployments are configurable through vSphere vCenter user interface but it is not possible from the CloudStack user interface.
Root cause: CloudStack does not handle vApp configuration options during deployments if the appliance contains configurable options. These configurations are mandatory for VM deployment from the appliance on Vmware vSphere vCenter. As shown in the image below, Vmware detects there are mandatory configurations that the administrator must set before deploy the VM from the appliance (in red on the image below):
Solution:
On template registration, after it is downloaded to secondary storage, the OVF file is examined and OVF properties are extracted from the file when available.
OVF properties extracted from templates after being downloaded to secondary storage are stored on the new table 'template_ovf_properties'.
A new optional section is added to the VM deployment wizard in the UI:
If the selected template does not contain OVF properties, then the optional section is not displayed on the wizard.
If the selected template contains OVF properties, then the optional new section is displayed. Each OVF property is displayed and the user must complete every property before proceeding to the next section.
If any configuration property is empty, then a dialog is displayed indicating that there are empty properties which must be set before proceeding
image
The specific OVF properties set on deployment are stored on the 'user_vm_details' table with the prefix: 'ovfproperties-'.
The VM is configured with the vApp configuration section containing the values that the user provided on the wizard.
This reverts commit 7a27e35a61.
We're near 4.13 RC1, we've low confidence if the changes from #3152
would cause other regressions so reverting this. The author may send a
PR again towards 4.14.
Regressions found are all related to template and iso registration and
upload.
Make use of Java 8 default implementation in interfaces,
to remove code duplication between XxxCmd and XxxCmdAsAdmin.
Refactor checkFormat by pre-calculating the supported
extensions. Also make use of this in ImageStoreUtil.
Makes it easier to add new file and compression formats.
Problem: Currently tags cannot be applied to snapshot when it is being created but through separate “create tags” API calls. For snapshot policies tags cannot be set either at creation or through “create tags” API.
Root Cause: The “create snapshots” API does not support adding tags during creation and it can only be done through “create tags” API. Snapshot policy as a resource does not support tags and no tags can be set for them through any API.
Solution: Tag support for snapshot policy has been added. Snapshot policy with tags when executed will produce snapshots containing the same tags from snapshot policy.
Following APIs have been updated:
Both “create snapshotpolicy” and “create snapshot” now accepts “tags” as a new parameter. The expected format for “tags” parameter is similar to parameter “tags” in “create tags“ API.
Deletion support for tags associated with snapshots policy has been added to “delete snapshotpolicies” API.
Tags set for snapshot policies are added to the Response of “list snapshotpolicies“ API.
UI support for setting tags to snapshots and snapshot policy is provided through the corresponding menus with a new section in each form to set tags.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* Rename ListUsageRecords API command file name to ListUsageRecordsCmd
* Refactor to use APINAME variable and remove unused s_logger field
* Remove unused import
* - Offline VM and Volume migration on Vmware hypervisor hosts
- Also add VM disk consolidation call on successful VM migrations
* Fix indentation of marvin test file and reformat against PEP8
* * Fix few comment typos
* Refactor debug messages to use String.format() when debug log level is enabled.
* Send list of commands returned by hypervisor Guru instead of explicitly selecting the first one
* Fix unhandled NPE during VM migration
* Revert back to distinct event descriptions for VM to host or storage pool migration
* Reformat test_primary_storage file against PEP-8 and Remove unused imports
* Revert back the deprecation messages in the custom StringUtils class to favour the use of the ApacheUtils
* [CLOUDSTACK-10240] ACS cannot migrate a volume from local to shared storage.
CloudStack is logically restricting the migration of local storages to shared storage and vice versa. This restriction is a logical one and can be removed for XenServer deployments. Therefore, we will enable migration of volumes between local-shared storages in XenServers independently of their service offering. This will work as an override mechanism to the disk offering used by volumes. If administrators want to migrate local volumes to a shared storage, they should be able to do so (the hypervisor already allows that). The same the other way around.
* Cleanups implemented while working on [CLOUDSTACK-10240]
* Fix test case test_03_migrate_options_storage_tags
The changes applied were:
- When loading hypervisors capabilities we must use "default" instead of nulls
- "Enable" storage migration for simulator hypervisor
- Remove restriction on "ClusterScopeStoragePoolAllocator" to find shared pools
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.
- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml
Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
