The project-wide security threat model merged (apache/cloudstack#13293)
as draft-THREAT-MODEL.md, but the canonical discoverability name that
scanners and satellite-repo SECURITY.md pointers follow is THREAT_MODEL.md.
This renames the file to THREAT_MODEL.md and updates the in-repo
SECURITY.md reference, making the pointer in apache/cloudstack-cloudmonkey
(which already targets .../blob/main/THREAT_MODEL.md) resolve. The
document's own review-status wording is unchanged.
Generated-by: Claude Opus 4.8 (1M context)
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Has been cancelled
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
qemu-img 10.1.0 changed the "qemu-img --help" supported-formats header
from "Supported formats:" to "Supported image formats:". The regex in
QemuImg.helpSupportsImageFormat() only matched the old header, so
hostSupportsVolumeEncryption() returned false on affected hosts even
though cryptsetup and the luks format were both available, blocking
encrypted offerings.
Make the "image" keyword optional in the regex so it matches both the
legacy and current qemu-img help output.
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic) (push) Has been cancelled
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Has been cancelled
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Waiting to run
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Waiting to run
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic) (push) Waiting to run
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Waiting to run
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Waiting to run
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Waiting to run
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Waiting to run
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Waiting to run
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Waiting to run
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Waiting to run
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Waiting to run
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Waiting to run
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Waiting to run
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Waiting to run
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Waiting to run
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Waiting to run
* debian/rules: remove requiretty from /etc/sudoers.d/cloudstack
* fix scripts/storage/secondary/setup-sysvm-tmplt
/tmp is a tmpfs type of filesystem
```
root@mgmt1:~# /usr/share/cloudstack-common/scripts/storage/secondary/setup-sysvm-tmplt -u 002c593f-eac9-4fee-85dd-44887a3f2042 -f /usr/share/cloudstack-m
+anagement/templates/systemvm/systemvmtemplate-4.22.0-x86_64-kvm.qcow2.bz2 -h kvm -d /tmp/tmp5782242889444060623/template/tmpl/1/3
....
Insufficient free disk space for target folder /tmp/tmp5782242889444060623/template/tmpl/1/3: avail=1739872k req=2120000k
root@mgmt1:~# mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64,usrquota)
```
* marvin: revert mysql-connector-python to 8.0.30
* Apply "mysql-connector-python >= 8.4.0",
* register systemvm template via /var/tmp in java too
* Revert "register systemvm template via /var/tmp in java too"
This reverts commit c0147c5e90.
* mgmt: add -Djava.io.tmpdir=/var/tmp to JAVA_OPTS
* fix scripts/storage/secondary/createtmplt.sh
Encrypted Linstor volumes use a LUKS layer inside the DRBD stack, so
the storage-layer snapshot device holds ciphertext while the DRBD
device CloudStack restores to is the decrypted view. Backing up the
raw snapshot and writing it back to the decrypted device corrupted the
volume (different data, unbootable root).
Back up encrypted snapshots from the decrypted DRBD device (forcing
the temporary-resource path) and store them as a LUKS-encrypted qcow2
using the volume passphrase, so snapshots are not kept in clear text
on secondary storage. On revert, decrypt the qcow2 and write plaintext
to the DRBD device; the LUKS layer re-encrypts it. The qemu-img shrink
is skipped for encrypted volumes (the DRBD device is already
net-sized).
Add an integration test (test_linstor_encrypted_snapshots.py): the
encrypted-root snapshot revert round-trip, that
create-volume-from-encrypted-snapshot is rejected by CloudStack core,
and a best-effort check that the backed-up qcow2 is LUKS-encrypted at
rest.
* KVM: add configurable MAC/IP script hook for static ARP/NDP and routes
Introduces a new agent.properties option `vm.network.macip.static`
(false by default) that makes BridgeVifDriver invoke on modifymacip.sh
on every NIC plug (VM start) and unplug (VM stop).
This is very useful in EVPN+VXLAN environments as it can reduce BUM
traffic. By setting static ARP/NDP entries bridges can be configured
using 'neigh_suppress on' as the ARP/NDP entries are already set
statically by CloudStack.
Setting 'neigh_suppress on' requires a manual change in the modifyvxlan.sh
script as this is not the default behavior.
* vxlan: In EVPN mode, disable ARP/NDP learning
FRR populates the FDB via BGP EVPN, so kernel data-plane learning is
redundant and counterproductive.
Static ARP (IPv4) and NDP (IPv6) entries are added on startup of the
Instance and remove on shutdown.
FRR populates the FDB/neighbor table via control plane, and neigh_suppress
tells the kernel bridge to use that information instead of flooding.
This will vastly reduce BUM traffic with static ARP/NDP entries.
This PR introduces and wires a new extension model for external network orchestration in CloudStack, centered on a new extension type: NetworkOrchestrator.
It extends the extension lifecycle from cluster-only registration to physical-network registration, adds API support for updating registered extension metadata, and enables automatic offering creation (network and VPC) based on provider-declared supported services and capabilities.
It also adds smoke coverage (KVM-only) using a Linux network namespace based implementation.
Doc PR: apache/cloudstack-documentation#649
What's new
1) New extension type: NetworkOrchestrator
Adds support for creating extensions of type NetworkOrchestrator.
Intended to back CloudStack network/VPC operations via an external orchestrator/provider.
2) Register extension with PhysicalNetwork (in addition to Cluster)
Extensions can now be registered against a PhysicalNetwork resource.
This enables network service provider behavior at physical-network scope, not only cluster scope.
3) Physical network registration details support
Registered extension details for PhysicalNetwork are handled similarly to cluster registration details.
Supports storing/updating external access metadata (credentials/endpoints/config details).
4) New API: update registered extension
Adds API support to update extension registration metadata after registration.
Useful for rotating credentials, updating endpoints, and changing external connection properties without re-registering.
5) Offering automation from external provider capabilities
Network/VPC offerings can be created with the external network provider using:
provider supportedservices
per-service service capabilities
This allows CloudStack offerings to align with what the external provider actually supports.
6) Network support via generated offerings
Using offerings backed by the external provider, networks can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
Source NAT
Static NAT
Port Forwarding
Firewall
Load Balancing
DHCP
DNS
UserData
7) VPC support via generated offerings
Using VPC offerings backed by the external provider, VPCs and tiers can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
VPC tier creation/implementation
Source NAT in VPC context
Static NAT / Port Forwarding / LB on VPC tiers
Network ACL association and ACL rule apply paths
Related lifecycle/restart/reapply operations
8) Linux network namespace based external implementation
Adds/uses a network extension implementation based on Linux network namespaces.
Reference implementation:
https://github.com/apache/cloudstack-extensions/tree/network-namespace/Network-Namespace
9) Smoke test coverage (KVM-only)
Adds smoke tests using the namespace-based extension implementation.
Scope includes provider lifecycle, offering creation, network/VPC flows, and key network services.
Applicable hypervisor for this smoke suite: KVM.
* Avoid concurrent deletes modifying the same backup chain in parallel - Backup deletes are serialized by using a per-VM GlobalLock
Backup delete checks if it has a live dependent backup not just a live immediate child.
* Enhance DHCP functionality by adding lease time support across various components. Updated DhcpEntryCommand to include lease time, modified VmDhcpConfig to handle lease time, and adjusted related scripts and configurations to accommodate this new parameter. This allows for configurable DHCP lease durations, improving flexibility in network management.
* Move dhcp lease timeout to ConfigKey, set it to 'zone' scope, and add some tests.
* Refactor DHCP lease timeout tests to pass code standards/remove white space.
* add DhcpLeaseTimeout in getConfigKeys() method
* Apply suggestion from @weizhouapache
---------
Co-authored-by: Wei Zhou <weizhou@apache.org>
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic) (push) Has been cancelled
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
* Add domain_id to oauth_provider table and VO
* Add domain-aware methods to OauthProviderDao
* Add domainId parameter to OAuth provider API commands and response
* Add domain support to OAuth2AuthManager
* Add domain-aware OAuth verification
* Add domain support to ListOAuthProvidersCmd and update related tests
* fix domain path issue
* Add domainId support to OAuth provider
* Return domain name and UUID in OAuth provider API responses using ApiDBUtils
* Refactor domain ID resolution in VerifyOAuthCodeAndGetUserCmd to improve code clarity
* Enhance OAuth2 plugin support for domain-level configuration and authentication checks
* Update OAuth2 tests and VerifyOAuthCodeAndGetUserCmdTest
* Add method to find OAuth provider by domain with global fallback
* Update OAuth provider configuration to use 'domain' instead of 'domainid' in columns and details
* Refactor OAuth provider methods to support domain-level queries and enhance user verification
* Add caching for access token retrieval in GithubOAuth2Provider
* Refactor access token checks in GithubOAuth2Provider to use StringUtils for improved readability and consistency
* Refactor null checks to use utility for improved readability and consistency
* Update OAuth2UserAuthenticatorTest to include domainId in user verification method
* Remove unnecessary blank line and unused imports in OAuth provider command classes
* Refactor and cleanup
* Remove unnecessary blank lines
* Enhance RegisterOAuthProviderCmdTest with additional provider mock data
* Remove startup gate from OAuth plugin initialization to support dynamic config toggling
* Add strictScope to ConfigKey to disable global fallback for domain-scoped oauth2.enabled
* Add domain-scoped provider filtering to listOauthProvider and centralize domain resolution in OAuth2AuthManager
* Add External OAuth tab with domain-scoped provider selection to login page
* code cleanup
* test fix
* Handle login page provider visibility
* UI cleanup
* UI Cleanup
* Keep text color consistent
* add unit tests
* Add Multiple-domain OAuth tests
* Refactor as per PR comments
* Use idempotent DDL helpers for oauth_provider schema migration
* Use global config check for global providers and extract oauthEnabled variable
* Make strictScope return null when id is null
* Rename verification methods to use 'verifySecretCodeAndFetchEmail' for consistency
* Refactor domain handling in OAuth2AuthManagerImpl to use DomainService instead of DomainDao
* Enhance domain ID descriptions in OAuth command classes for clarity
* Add domain path handling to OAuth provider commands and improve descriptions
* Update domain path description in VerifyOAuthCodeAndGetUserCmd to clarify behavior with Domain ID
* Replace remove method with expunge in deleteOauthProvider and add corresponding unit test
* Add external login label to Login.vue and update i18n locale handling
* Fix stale value issue in updateConfiguration response handling in ConfigurationValue.vue
* Enhance OAuth login error handling and add unit test for missing parameters
* Add validation to reject enabling OAuth provider when plugin is disabled at domain scope
* Add domain reassignment support to UpdateOAuthProviderCmd and enhance validation in OAuth2AuthManagerImpl
* Add domain ID to OAuth provider arguments in config
* Fix condition for OAuth verification URL handling in router
* Add domain path to OauthProviderResponse and update UI config to display it
* Update config to remove 'secretkey' from columns and details
* Add secretkey to details in config and display in DetailsTab
* Implement normalization of ROOT domain to null for global OAuth provider handling and add corresponding unit tests
* Refactor OAuth plugin domain scope handling to use a centralized method for enabling checks
* Add strict scope handling to ConfigKey and update OAuth2AuthManager usage
* Implement domain removal listener to clean up OAuth providers on domain deletion
* Enhance OAuth tab icons with disabled state styling for better UX
* Add domain-specific provider prompt and update OAuth provider handling
---------
Co-authored-by: Daman Arora <daman.arora@shapeblue.com>
* Add xenserver.create.full.clone global setting
Adds a StoragePool-scoped boolean ConfigKey mirroring vmware.create.full.clone
so XenServer-backed VMs can be deployed as full VDI copies (VDI.copy) instead
of always using linked clones (VDI.clone). Default false preserves today's
behavior.
The per-pool flag flows into the existing PrimaryDataStoreTO.fullCloneFlag
through a new dispatch method addFullCloneAndDiskprovisiongStrictnessFlagOnDest
that switches on hypervisor type.
* Fix HypervisorType dispatch in addFullClone flag helper
Replace invalid switch on Hypervisor.HypervisorType (not a Java enum) with
equality checks so cloud-engine-storage-datamotion compiles.
---------
Co-authored-by: Erki Märks <erma07@users.noreply.github.com>
Implements incremental backup support for the NAS backup provider on KVM, using QEMU dirty bitmaps and libvirt's backup-begin API. RFC: apache/cloudstack#12899.
For large VMs this reduces daily backup storage 80–95% and shortens backup windows from hours to minutes (e.g. a 500 GB VM with moderate writes goes from ~500 GB/day to ~5–15 GB/day after the initial full backup).
Signed-off-by: James Peru <jmsperu@gmail.com>
Co-authored-by: jmsperu <jmsperu@users.noreply.github.com>
Co-authored-by: Abhisar Sinha <63767682+abh1sar@users.noreply.github.com>
`reboot.host.and.alert.management.on.heartbeat.timeout` has to be
disabled. Even the high availability isn't enabled when there is an
issue with a storage CloudStack will reboot the host
* NAS backup: resume paused VM on backup failure and fix missing exit
When a NAS backup job fails (e.g. due to backup storage being full or
I/O errors), the VM may remain indefinitely paused because:
1. The cleanup() function never checks or resumes the VM's paused state
that was set by virsh backup-begin during the push backup operation.
2. The 'Failed' case in the backup job monitoring loop calls cleanup()
but lacks an 'exit' statement, causing an infinite loop where the
script repeatedly detects the failed job and calls cleanup().
3. Similarly, backup_stopped_vm() calls cleanup() on qemu-img convert
failure but does not exit, allowing the loop to continue with
subsequent disks despite the failure.
This fix:
- Adds VM state detection and resume to cleanup(), ensuring the VM is
always resumed if found in a paused state during error handling
- Adds missing 'exit 1' after cleanup() in the Failed backup job case
to prevent the infinite monitoring loop
- Adds missing 'exit 1' after cleanup() in backup_stopped_vm() on
qemu-img convert failure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: retrigger workflow (flaky/stale shards)
* nasbackup.sh: keep cleanup() best-effort if domstate fails
With set -eo pipefail, a non-zero virsh domstate (libvirt unavailable or the
domain gone) in the vm_state assignment would abort cleanup() before the
rm/umount/rmdir. Append '|| true' so cleanup always runs to completion.
Addresses the review suggestion (Copilot, endorsed by @abh1sar and @weizhouapache).
Signed-off-by: James Peru <jmsperu@gmail.com>
---------
Signed-off-by: James Peru <jmsperu@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jmsperu <jmsperu@users.noreply.github.com>
PR #13254 added an 'allocated' case to Status.vue getText() that calls
$t('label.allocated'), but did not update the test mock data, causing
two Allocated badge tests to fail in CI.
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Has been cancelled
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Has been cancelled
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Has been cancelled
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Has been cancelled
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Has been cancelled
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Has been cancelled
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Has been cancelled
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Has been cancelled
Simulator CI / build (component/find_hosts_for_migration component/test_acl_isolatednetwork component/test_acl_isolatednetwork_delete component/test_acl_listsnapshot) (push) Waiting to run
Simulator CI / build (component/test_acl_sharednetwork component/test_acl_sharednetwork_deployVM-impersonation component/test_user_private_gateway component/test_user_shared_network) (push) Waiting to run
Simulator CI / build (component/test_concurrent_snapshots_limit component/test_cpu_domain_limits component/test_cpu_limits component/test_cpu_max_limits component/test_cpu_project_limits component/test_deploy_vm_userdata_multi_nic component/test_deploy_vm_lease) (push) Waiting to run
Simulator CI / build (component/test_lb_secondary_ip component/test_list_nics component/test_list_pod component/test_memory_limits) (push) Waiting to run
Simulator CI / build (component/test_mm_domain_limits component/test_mm_max_limits component/test_mm_project_limits component/test_network_offering component/test_non_contiguous_vlan) (push) Waiting to run
Simulator CI / build (component/test_persistent_networks component/test_project_configs component/test_project_limits component/test_project_resources) (push) Waiting to run
Simulator CI / build (component/test_project_usage component/test_protocol_number_security_group component/test_public_ip component/test_resource_limits component/test_resource_limit_tags) (push) Waiting to run
Simulator CI / build (smoke/test_list_accounts smoke/test_list_disk_offerings smoke/test_list_domains smoke/test_list_hosts smoke/test_list_service_offerings smoke/test_list_storage_pools smoke/test_list_volumes) (push) Waiting to run
* Async event bus publishing in AsyncJobManagerImpl to reduce API thread contention
publishOnEventBus() was calling _messageBus.publish() synchronously on the
request thread, which blocks on MessageBusBase$Gate (an exclusive mutex).
JFR analysis showed this causing up to 107ms waits on Jetty request threads,
contributing to 502 errors from the upstream load balancer.
Move event bus publishing to a dedicated single-threaded executor so API
request threads are no longer blocked by Gate contention. Event ordering
is preserved by the single-threaded executor.
* Wrap async event bus publish in ManagedContextRunnable for DB connection safety
The sole subscriber (ApiServer.handleAsyncJobPublishEvent) performs DAO reads
(getUserIncludingRemoved, getAccount, findById) inside its callback.
Without ManagedContextRunnable, the EventBus thread would not have proper
TransactionLegacy lifecycle management, risking DB connection leaks.
* logger fix
* review comments
* review changes
---------
Co-authored-by: Aaron Chung <aaron_chung@apple.com>
