This PR introduces and wires a new extension model for external network orchestration in CloudStack, centered on a new extension type: NetworkOrchestrator.
It extends the extension lifecycle from cluster-only registration to physical-network registration, adds API support for updating registered extension metadata, and enables automatic offering creation (network and VPC) based on provider-declared supported services and capabilities.
It also adds smoke coverage (KVM-only) using a Linux network namespace based implementation.
Doc PR: apache/cloudstack-documentation#649
What's new
1) New extension type: NetworkOrchestrator
Adds support for creating extensions of type NetworkOrchestrator.
Intended to back CloudStack network/VPC operations via an external orchestrator/provider.
2) Register extension with PhysicalNetwork (in addition to Cluster)
Extensions can now be registered against a PhysicalNetwork resource.
This enables network service provider behavior at physical-network scope, not only cluster scope.
3) Physical network registration details support
Registered extension details for PhysicalNetwork are handled similarly to cluster registration details.
Supports storing/updating external access metadata (credentials/endpoints/config details).
4) New API: update registered extension
Adds API support to update extension registration metadata after registration.
Useful for rotating credentials, updating endpoints, and changing external connection properties without re-registering.
5) Offering automation from external provider capabilities
Network/VPC offerings can be created with the external network provider using:
provider supportedservices
per-service service capabilities
This allows CloudStack offerings to align with what the external provider actually supports.
6) Network support via generated offerings
Using offerings backed by the external provider, networks can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
Source NAT
Static NAT
Port Forwarding
Firewall
Load Balancing
DHCP
DNS
UserData
7) VPC support via generated offerings
Using VPC offerings backed by the external provider, VPCs and tiers can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
VPC tier creation/implementation
Source NAT in VPC context
Static NAT / Port Forwarding / LB on VPC tiers
Network ACL association and ACL rule apply paths
Related lifecycle/restart/reapply operations
8) Linux network namespace based external implementation
Adds/uses a network extension implementation based on Linux network namespaces.
Reference implementation:
https://github.com/apache/cloudstack-extensions/tree/network-namespace/Network-Namespace
9) Smoke test coverage (KVM-only)
Adds smoke tests using the namespace-based extension implementation.
Scope includes provider lifecycle, offering creation, network/VPC flows, and key network services.
Applicable hypervisor for this smoke suite: KVM.
* Add domain_id to oauth_provider table and VO
* Add domain-aware methods to OauthProviderDao
* Add domainId parameter to OAuth provider API commands and response
* Add domain support to OAuth2AuthManager
* Add domain-aware OAuth verification
* Add domain support to ListOAuthProvidersCmd and update related tests
* fix domain path issue
* Add domainId support to OAuth provider
* Return domain name and UUID in OAuth provider API responses using ApiDBUtils
* Refactor domain ID resolution in VerifyOAuthCodeAndGetUserCmd to improve code clarity
* Enhance OAuth2 plugin support for domain-level configuration and authentication checks
* Update OAuth2 tests and VerifyOAuthCodeAndGetUserCmdTest
* Add method to find OAuth provider by domain with global fallback
* Update OAuth provider configuration to use 'domain' instead of 'domainid' in columns and details
* Refactor OAuth provider methods to support domain-level queries and enhance user verification
* Add caching for access token retrieval in GithubOAuth2Provider
* Refactor access token checks in GithubOAuth2Provider to use StringUtils for improved readability and consistency
* Refactor null checks to use utility for improved readability and consistency
* Update OAuth2UserAuthenticatorTest to include domainId in user verification method
* Remove unnecessary blank line and unused imports in OAuth provider command classes
* Refactor and cleanup
* Remove unnecessary blank lines
* Enhance RegisterOAuthProviderCmdTest with additional provider mock data
* Remove startup gate from OAuth plugin initialization to support dynamic config toggling
* Add strictScope to ConfigKey to disable global fallback for domain-scoped oauth2.enabled
* Add domain-scoped provider filtering to listOauthProvider and centralize domain resolution in OAuth2AuthManager
* Add External OAuth tab with domain-scoped provider selection to login page
* code cleanup
* test fix
* Handle login page provider visibility
* UI cleanup
* UI Cleanup
* Keep text color consistent
* add unit tests
* Add Multiple-domain OAuth tests
* Refactor as per PR comments
* Use idempotent DDL helpers for oauth_provider schema migration
* Use global config check for global providers and extract oauthEnabled variable
* Make strictScope return null when id is null
* Rename verification methods to use 'verifySecretCodeAndFetchEmail' for consistency
* Refactor domain handling in OAuth2AuthManagerImpl to use DomainService instead of DomainDao
* Enhance domain ID descriptions in OAuth command classes for clarity
* Add domain path handling to OAuth provider commands and improve descriptions
* Update domain path description in VerifyOAuthCodeAndGetUserCmd to clarify behavior with Domain ID
* Replace remove method with expunge in deleteOauthProvider and add corresponding unit test
* Add external login label to Login.vue and update i18n locale handling
* Fix stale value issue in updateConfiguration response handling in ConfigurationValue.vue
* Enhance OAuth login error handling and add unit test for missing parameters
* Add validation to reject enabling OAuth provider when plugin is disabled at domain scope
* Add domain reassignment support to UpdateOAuthProviderCmd and enhance validation in OAuth2AuthManagerImpl
* Add domain ID to OAuth provider arguments in config
* Fix condition for OAuth verification URL handling in router
* Add domain path to OauthProviderResponse and update UI config to display it
* Update config to remove 'secretkey' from columns and details
* Add secretkey to details in config and display in DetailsTab
* Implement normalization of ROOT domain to null for global OAuth provider handling and add corresponding unit tests
* Refactor OAuth plugin domain scope handling to use a centralized method for enabling checks
* Add strict scope handling to ConfigKey and update OAuth2AuthManager usage
* Implement domain removal listener to clean up OAuth providers on domain deletion
* Enhance OAuth tab icons with disabled state styling for better UX
* Add domain-specific provider prompt and update OAuth provider handling
---------
Co-authored-by: Daman Arora <daman.arora@shapeblue.com>
Simulator CI / test (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Blocked by required conditions
Simulator CI / test (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Blocked by required conditions
Simulator CI / test (component/test_affinity_groups_projects component/test_allocation_states component/test_assign_vm) (push) Blocked by required conditions
Simulator CI / test (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Blocked by required conditions
Simulator CI / test (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Blocked by required conditions
Simulator CI / test (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Blocked by required conditions
Simulator CI / test (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Blocked by required conditions
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Waiting to run
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Waiting to run
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Waiting to run
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Waiting to run
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Waiting to run
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Waiting to run
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Waiting to run
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Waiting to run
* linstor: update java-linstor to 0.7.0 to support auth token api
With Linstor 1.34.0 a new authentication mode is supported:
* Bearer token
To support that it had to be implemented in the java-linstor library
and we need to store the auth token per storage pool.
Also per default with this auth mode Linstor will run with
HTTPS enabled, so we also have to support that.
* ui: seed default-on toggles in zone wizard while hidden
StaticInputsForm.fillValue() only seeded defaults for currently-displayed
fields, so a display-gated switch with checked:true bound to an undefined
value and rendered as off once revealed. Seed checked switch/checkbox
fields even while hidden, so the Linstor 'Allow self-signed certificate'
toggle defaults on in the zone creation wizard.
Simulator CI / test (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / test (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / test (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Has been cancelled
Simulator CI / test (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / test (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / test (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / test (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / test (component/test_regions_accounts component/test_routers component/test_snapshots component/test_stopped_vm component/test_tags component/test_templates component/test_updateResourceCount component/test_update_vm) (push) Has been cancelled
Simulator CI / test (component/test_volumes component/test_vpc component/test_vpc_distributed_routing_offering component/test_vpc_network component/test_vpc_offerings component/test_vpc_routers component/test_vpn_users component/test_vpc_network_lbrules) (push) Has been cancelled
Simulator CI / test (smoke/test_accounts smoke/test_account_access smoke/test_affinity_groups smoke/test_affinity_groups_projects smoke/test_annotations smoke/test_async_job smoke/test_attach_multiple_volumes smoke/test_backup_recovery_dummy smoke/test_certauthority_… (push) Has been cancelled
Simulator CI / test (smoke/test_cluster_drs smoke/test_dynamicroles smoke/test_enable_account_settings_for_domain smoke/test_enable_role_based_users_in_projects smoke/test_events_resource smoke/test_global_settings smoke/test_guest_vlan_range smoke/test_host_maintena… (push) Has been cancelled
Simulator CI / test (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
Simulator CI / test (smoke/test_network smoke/test_network_acl smoke/test_network_ipv6 smoke/test_network_permissions smoke/test_nic smoke/test_nic_adapter_type smoke/test_non_contigiousvlan smoke/test_object_stores smoke/test_outofbandmanagement smoke/test_outofband… (push) Has been cancelled
Simulator CI / test (smoke/test_router_dhcphosts smoke/test_router_dns smoke/test_router_dnsservice smoke/test_routers smoke/test_routers_iptables_default_policy smoke/test_routers_network_ops smoke/test_scale_vm smoke/test_secondary_storage smoke/test_service_offeri… (push) Has been cancelled
* refactor vmschedule to make it more generic and add new commands for it
* fixup
* update marvin tests to use the new APIs
* Update UI to use new APIs
* minor fixups
* Add support scheduling min & max for autoscaling groups
* Add marvin test for ASG
* fixup
* fixup
* fixup
* Fixup
Simulator CI / test (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Blocked by required conditions
Simulator CI / test (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Blocked by required conditions
Simulator CI / test (component/test_affinity_groups_projects component/test_allocation_states component/test_assign_vm) (push) Blocked by required conditions
Simulator CI / test (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Blocked by required conditions
Simulator CI / test (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Blocked by required conditions
Simulator CI / test (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Blocked by required conditions
Simulator CI / test (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Blocked by required conditions
* Addition of description field for NIC's secondary IP addresses
* Address copilot's review
* Add newline at the end of file
Co-authored-by: dahn <daan.hoogland@gmail.com>
* Address Wei's label changes
* Remove colon from label
* Apply suggestions from code review
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
* Fix missing newline at end of SQL file
---------
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
* 4.22:
VM Deployment using snapshot in new zone (#13178)
Change exception treatment on incremental snapshot wait (#12665)
Move checkRoleEscalation outside DB transaction in createAccount (#13044)
Fix/flasharray delete rename destroy patch conflict (#13049)
Fix VPC network offerings listing in isolated network creation form (#12645)
systemvm: accept ipv6 established/related return traffic (#13173)
update debian change log
Updating pom.xml version numbers for release 4.22.2.0-SNAPSHOT
Updating pom.xml version numbers for release 4.22.1.0
Update suse15 packaging spec, use qemu-ovmf-x86_64 package instead of edk2-ovmf for agent (#13133)
Change disk-only VM snapshot removal message (#11182)
Update mysql java connector version to 8.4.0 (matching version for MySQL 8.4) (#12640)
adaptive: honor user-provided capacityBytes when provider stats are unavailable (#13059)
Flexibilize public IP selection (#11076)
* 4.20:
Fix/flasharray delete rename destroy patch conflict (#13049)
Fix VPC network offerings listing in isolated network creation form (#12645)
Update mysql java connector version to 8.4.0 (matching version for MySQL 8.4) (#12640)
adaptive: honor user-provided capacityBytes when provider stats are unavailable (#13059)
Flexibilize public IP selection (#11076)
* Add support to clone existing offerings and update them
* add support for vpc & backup offerings to be cloned
* fix capability list and mapping of params
* Add support to clone network and vpc offering with the right parameters
* make fields non mandatory for clone offerings APIs
* Add UI support for cloning Compute and System Service offerings
* remove unnecessary changes
* fix license and pre-ccommit issues
* Add UI support to clone disk and network offering
* vpc & backup offering clone api
* add unit tests
* fix pre-commit checks
* increase test coverage
* combine add/clone disk/compute offering forms
* update license
* fix unit tests
* fix test failures
* fix test failure - unnecessary stubbings
* pre-commit check failure
* add recently added domain id for bkp offering to be inherited in clone operation
* extract common code wrt service capability in network & vpc offering in add/clone operations
* add some checks to prevent networkmode change when provider is nsx/netris from the source networkmode
* address copilot comments
* address comments
* combine check
* use appropriate zoneId during clone bkp offering
* add check
* fix issue with test
* remove unused imports
* prevent creating a bkp offering of a bkp repo that already exists
* extend clone disk and service offerings to domain admins
Adds a new request parameter for create/updateExtension API to allow
operator to provide detail names for the extension resources which will be reserved to be used by the extension. The end user won't be able to view or add details with these details names for the resource.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* API modifications for passwordchangerequired
* ui login flow for passwordchangerequired
* add passwordchangerequired in listUsers API response, it will be used in UI to render reset password form
* cleanup redundant LOGIN_SOURCE and limiting apis for first time login
* address copilot comments
* allow enforcing password change for all role types and update reset pwd flow for passwordchangerequired
* address review comments
* add unit tests
* cleanup ispasswordchangerequired from user_view
* address review comments
* 1. Allow enforcing password change while creating user
2. Admin can enforce password change on next login with out resetting password
* address review comment, add unit test
* improve code coverage
* fix pre-commit license issue
* 1. allow enter key to submit change password form
2. hide force password reset for disabled/locked user in ui
* 1. throw exception when force reset password is done for locked/disabled user/account
2. ui validation on current and new password being same
3. allow enforce change password for add user until saml is not enabled
* allow oauth login to skip force password change
