api,server,ui: snapshot copy, multi-zone replica #7873

shwstppr · 2023-08-16T09:09:35Z

Description

This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - copySnapshot has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the copyTemplate API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - zoneids has been added to createSnapshot and createSnapshotPolicy APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter zoneid has been added to delete the snapshot copy on a specific zone.

listSnapshots API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - showUnique, locationType have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

listSnapshotPolicies and createSnapshotPolicy APIs will return zone details of the zones in which backup will be scheduled for the policy.

New API added
copySnapshot

Request and response params updated for APIs

- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy

UI updated for

Snapshot detail view
Create snapshot form
Create snapshot policy form
Create volume (from snapshot) form
Create template (from snapshot) form

Doc PR: apache/cloudstack-documentation#344

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Manual snapshot operations in a multi-zone environment

Smoke test added:

[root@ref-trl-5659-k-M7-abhishek-kumar-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./ref-trl-5659-k-M7-abhishek-kumar-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/component/test_snapshot_copy.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/Sep_13_2023_22_14_19_WGW1DW All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_take_snapshot_multi_zone | Status : SUCCESS ===

=== TestName: test_02_copy_snapshot_multi_zone | Status : SUCCESS ===

=== TestName: test_03_take_snapshot_multi_zone_delete_single_zone | Status : SUCCESS ===

=== TestName: test_04_copy_snapshot_multi_zone_delete_all | Status : SUCCESS ===

=== TestName: test_05_take_snapshot_multi_zone_create_volume_additional_zone | Status : SUCCESS ===

=== TestName: test_06_take_snapshot_multi_zone_create_template_additional_zone | Status : SUCCESS ===

=== Final results are now copied to: /marvin//MarvinLogs/test_snapshot_copy_WS0393 ===

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2023-09-04T04:32:47Z

@blueorangutan package

blueorangutan · 2023-09-04T04:34:03Z

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-09-04T05:25:55Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6972

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2023-09-04T18:25:30Z

@blueorangutan package

blueorangutan · 2023-09-04T18:26:03Z

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2023-09-04T18:47:10Z

Codecov Report

Merging #7873 (ba05e7c) into main (c9d2303) will increase coverage by 10.10%.
The diff coverage is 22.67%.

@@              Coverage Diff              @@
##               main    #7873       +/-   ##
=============================================
+ Coverage     19.07%   29.17%   +10.10%     
- Complexity    18250    30616    +12366     
=============================================
  Files          4967     5111      +144     
  Lines        336878   360624    +23746     
  Branches      48395    52697     +4302     
=============================================
+ Hits          64246   105222    +40976     
+ Misses       263345   240946    -22399     
- Partials       9287    14456     +5169

Flag	Coverage Δ
simulator-marvin-tests	`25.13% <17.53%> (+4.88%)`	⬆️
uitests	`4.79% <0.00%> (-0.06%)`	⬇️
unit-tests	`14.52% <15.17%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
api/src/main/java/com/cloud/event/EventTypes.java	`95.88% <ø> (ø)`
.../main/java/com/cloud/storage/VolumeApiService.java	`100.00% <ø> (ø)`
...n/java/org/apache/cloudstack/api/ApiConstants.java	`40.90% <ø> (ø)`
...k/api/command/user/snapshot/CreateSnapshotCmd.java	`45.67% <100.00%> (+3.17%)`	⬆️
...k/api/command/user/snapshot/DeleteSnapshotCmd.java	`80.00% <100.00%> (+1.05%)`	⬆️
...k/api/command/user/template/CreateTemplateCmd.java	`59.74% <100.00%> (+7.10%)`	⬆️
...cloudstack/api/command/user/zone/ListZonesCmd.java	`93.75% <100.00%> (+0.41%)`	⬆️
...g/apache/cloudstack/api/response/ZoneResponse.java	`92.04% <100.00%> (-0.98%)`	⬇️
...java/org/apache/cloudstack/query/QueryService.java	`100.00% <ø> (ø)`
...ack/engine/subsystem/api/storage/SnapshotInfo.java	`100.00% <ø> (ø)`
... and 75 more

... and 1638 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

blueorangutan · 2023-09-04T19:26:32Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6974

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2023-10-19T04:53:54Z

[LL]Trillian test result (tid-6861)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48294 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t6861-vmware-70u3.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-10-19T07:55:03Z

@DaanHoogland, I have to run a few more tests with the latest changes

ok, we have an interest in putting this in 4.19. Can you give an E.T.A. on these, @slavkap ?

shwstppr · 2023-10-21T10:25:24Z

@blueorangutan package

blueorangutan · 2023-10-21T10:26:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-10-21T11:20:33Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7456

shwstppr · 2023-10-21T12:17:31Z

@blueorangutan test matrix

blueorangutan · 2023-10-21T12:22:03Z

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-10-21T23:50:57Z

[SF] Trillian test result (tid-8046)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39837 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8046-kvm-centos7.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-10-22T02:17:07Z

[SF] Trillian test result (tid-8045)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48666 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8045-vmware-67u3.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	588.34	test_kubernetes_clusters.py

shwstppr · 2023-10-22T18:45:25Z

@blueorangutan test centos7 xs71

blueorangutan · 2023-10-22T18:46:03Z

@shwstppr [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

shwstppr · 2023-10-22T18:57:45Z

@blueorangutan test centos7 xenserver-71

blueorangutan · 2023-10-22T18:58:03Z

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + xenserver-71) has been kicked to run smoke tests

slavkap

CLGTM, tested with StorPool and NFS primary storage

blueorangutan · 2023-10-23T06:16:52Z

[SF] Trillian test result (tid-8051)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39412 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8051-xenserver-71.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2023-10-23T07:00:43Z

Merging based on approvals, QA testing.

great work @shwstppr !

Using XenServer as the hypervisor, when deleting a snapshot that has a parent, that parent will also get erased on storage, causing data loss. This behavior was introduced with #7873, where the list of snapshot states that can be deleted was changed to add BackedUp snapshots. This PR changes the states list back to the original list, and swaps the while loop for a do while loop to account for the changes in #7873. Fixes #9446

* server, api, ui: access improvements and assorted fixes Fixes domain-admin access check to prevent unauthorized access. Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com> Co-authored-by: nvazquez <nicovazquez90@gmail.com> Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * Revert "server: refactor listNetworks api database retrievals (#9184)" This reverts commit c7f1ba5. * Fix snapshot chain being deleted on XenServer (#9447) Using XenServer as the hypervisor, when deleting a snapshot that has a parent, that parent will also get erased on storage, causing data loss. This behavior was introduced with #7873, where the list of snapshot states that can be deleted was changed to add BackedUp snapshots. This PR changes the states list back to the original list, and swaps the while loop for a do while loop to account for the changes in #7873. Fixes #9446 * UI: Display Firewall, LB and Port Forwading rules tab for CKS clusters deployed on isolated networks (#9458) --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com> Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com> Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

Using XenServer as the hypervisor, when deleting a snapshot that has a parent, that parent will also get erased on storage, causing data loss. This behavior was introduced with apache#7873, where the list of snapshot states that can be deleted was changed to add BackedUp snapshots. This PR changes the states list back to the original list, and swaps the while loop for a do while loop to account for the changes in apache#7873. Fixes apache#9446

* server, api, ui: access improvements and assorted fixes Fixes domain-admin access check to prevent unauthorized access. Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com> Co-authored-by: nvazquez <nicovazquez90@gmail.com> Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * Revert "server: refactor listNetworks api database retrievals (apache#9184)" This reverts commit c7f1ba5. * Fix snapshot chain being deleted on XenServer (apache#9447) Using XenServer as the hypervisor, when deleting a snapshot that has a parent, that parent will also get erased on storage, causing data loss. This behavior was introduced with apache#7873, where the list of snapshot states that can be deleted was changed to add BackedUp snapshots. This PR changes the states list back to the original list, and swaps the while loop for a do while loop to account for the changes in apache#7873. Fixes apache#9446 * UI: Display Firewall, LB and Port Forwading rules tab for CKS clusters deployed on isolated networks (apache#9458) --------- Co-authored-by: nvazquez <nicovazquez90@gmail.com> Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com> Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com> Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

shwstppr added 7 commits July 27, 2023 18:54

server,api,ui: snapshot copy and zone selection

c8c95be

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

70648af

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

f556cb1

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

de0de2a

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

6328b08

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

3630bfd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Merge branch 'main' into snapshot-copy

8cb4f54

boring-cyborg bot added component:api component:orchestration component:storage labels Aug 16, 2023

shwstppr added 8 commits August 21, 2023 12:25

nl

8e4be3b

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

download percent for multi-file download

2c9bf2f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

35b5157

changes

cd49933

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

caa4ff8

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

99e6134

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

106fd8d

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

09e3359

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

882ce93

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 4 commits September 5, 2023 01:55

change

9801e5c

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fixes

527e5f0

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fixes

548c236

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

missing file

a2668dd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

slavkap approved these changes Oct 23, 2023

View reviewed changes

weizhouapache merged commit 543c54c into apache:main Oct 23, 2023

DaanHoogland deleted the snapshot-copy branch October 24, 2023 08:41

This was referenced Jul 25, 2024

Fix snapshot chain being deleted on XenServer #9447

Merged

Whole snapshot chain is lost when the latest snapshot is deleted #9446

Closed

sureshanaparti mentioned this pull request Jan 24, 2025

[UI] Allow quiescevm and asyncbackup flags while taking volume snapshot from UI when these are supported for the volume #10265

Merged

14 tasks

JoaoJandre mentioned this pull request Feb 25, 2025

KVM Incremental Snapshots/Backups #8907

Closed

JoaoJandre mentioned this pull request May 14, 2025

KVM incremental snapshot feature #9270

Merged

13 tasks

This was referenced Aug 14, 2025

Volume Snapshot Size Not displaying #11442

Closed

Fix snapshot physical size listing #11448

Merged

api,server,ui: snapshot copy, multi-zone replica #7873

api,server,ui: snapshot copy, multi-zone replica #7873

Uh oh!

Conversation

shwstppr commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

shwstppr commented Sep 4, 2023

Uh oh!

blueorangutan commented Sep 4, 2023

Uh oh!

blueorangutan commented Sep 4, 2023

Uh oh!

shwstppr commented Sep 4, 2023

Uh oh!

blueorangutan commented Sep 4, 2023

Uh oh!

codecov bot commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Sep 4, 2023

Uh oh!

blueorangutan commented Oct 19, 2023

Uh oh!

DaanHoogland commented Oct 19, 2023

Uh oh!

shwstppr commented Oct 21, 2023

Uh oh!

blueorangutan commented Oct 21, 2023

Uh oh!

blueorangutan commented Oct 21, 2023

Uh oh!

shwstppr commented Oct 21, 2023

Uh oh!

blueorangutan commented Oct 21, 2023

Uh oh!

blueorangutan commented Oct 21, 2023

Uh oh!

blueorangutan commented Oct 22, 2023

Uh oh!

shwstppr commented Oct 22, 2023

Uh oh!

blueorangutan commented Oct 22, 2023

Uh oh!

shwstppr commented Oct 22, 2023

Uh oh!

blueorangutan commented Oct 22, 2023

Uh oh!

slavkap left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Oct 23, 2023

Uh oh!

weizhouapache commented Oct 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

shwstppr commented Aug 16, 2023 •

edited

Loading

codecov bot commented Sep 4, 2023 •

edited

Loading