Codecov Report

❌ Patch coverage is 22.77301% with 3147 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.18%. Comparing base (3166e64) to head (82c9a0c).

@@             Coverage Diff              @@
##               main   #12758      +/-   ##
============================================
+ Coverage     18.01%   18.18%   +0.17%     
- Complexity    16607    16966     +359     
============================================
  Files          6029     6091      +62     
  Lines        542160   547494    +5334     
  Branches      66451    66972     +521     
============================================
+ Hits          97682    99578    +1896     
- Misses       433461   436781    +3320     
- Partials      11017    11135     +118     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JoaoJandre just heads up - my colleagues have been working on an incremental backup feature for NAS B&R (using nbd/qemu bitmap tracking & checkpoints). We're also working on a new Veeam-KVM integration for CloudStack whose PR may may be out soon. My colleagues can further help review and advise on this.

/cc @weizhouapache @abh1sar @shwstppr @sureshanaparti @DaanHoogland @harikrishna-patnala

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc. Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc.

At the time of writing that part, I believe it was only NFS that was supported. I'll update the relevant part.

Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

The secondary storage selector feature (introduced in 2023 by #7659) allows you to specialize secondary storages. This PR extended the feature so that you may also create selectors for backups.

Hi Joao,

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?

   • I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

     This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.

     The backup repository feature also comes with a separate capacity tracking and email alerts.

   • If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

2. About Qemu compression

   • Have you measured / compared the performance of qemu-img compression with other compression methods?
   • As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.
   • I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

3. Do we need a separate backup offering table and api?

   • Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?

   • Did you face any issues with checkpoints and bitmaps?

Hi Joao,

Hello, @abh1sar

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?
   
   * I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

     This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.
     The backup repository feature also comes with a separate capacity tracking and email alerts.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

   * If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

The secondary storage selectors feature (introduced in 2023 through #7659) allows you to specialize secondary storages. Quoting from the PR description: "This PR aims to add the possibility to direct resources (Volumes, Templates, Snapshots and ISOs) to a specific secondary storage through rules written in JavaScript that will only affect new allocated resources". For a few years it has been possible to have secondary storages that only receive snapshots or templates for example. This PR introduces the possibility to add selectors for backups, so that you have secondary storages that are specific for backups.

Furthermore, my colleagues are working on a feature to allow using alternative secondary storage solutions, such as CephFS, iSCSI and S3, while preserving compatibility with features destined to NFS storages. This feature may be extended in the future to allow essentially any type of secondary storage. Thus, the flexibility for secondary storages will soon grow.

2. About Qemu compression
   
   * Have you measured / compared the performance of qemu-img compression with other compression methods?

Using any other type of backup-level compression will be worse then using qemu-img compression. This is because when restoring the backup, we must have access to the whole backing chain. If we use other types of compression, we will have to decompress the whole chain before restoring. Using qemu-img, the backing files are still valid and do not need to be decompressed, we actually never have to decompress ever. This is the great benefit of using qemu-img.

In any case, here is a brief comparison of using qemu-img with the zstd library and 8 threads and using the pigz implementation of multi-threaded compression, also using 8 threads. The original file is the root volume of a VM that I use.

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding to the processing time of backing up/restoring a backup.

   * As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.

The compression feature is optional, if you are using storage-level compression, you probably will not use backup-level compression. However, many environments do not have storage-level compression, thus having the possibility of backup-level compression is still very interesting.

   * I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

The compression does not add any interaction with the SSVM.

3. Do we need a separate backup offering table and api?
   
   * Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?
   
   * Did you face any issues with checkpoints and bitmaps?

There are two main issues with using bitmaps:

1. They are prone to corruption, while this can be mitigated in some ways, since the incremental volume snapshot feature was added, we have noticed multiple cases of bitmap corruption with different causes. It is possible to detect the corruption and delete corrupt bitmaps, but this would add more complexity to the feature.
2. Using bitmaps is not compatible with the file-based incremental VM snapshot feature added in File-based disk-only VM snapshot with KVM as hypervisor #10632. After some internal discussion and feedback from users, we have come to the conclusion that being able to use both the incremental VM snapshot and backup features at the same time is very interesting.

At the end of the day, this PR adds a new backup provider option for users. They will be free to choose the provider that best fits their needs. This is one of the reasons why it was done as a new backup provider; KNIB and other backup providers do not have to cancel each-other out.

Hi @JoaoJandre
Please find my response inline.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings.
Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

Capacity is just for Admins for Backup Repository also, so that is not the issue.
But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups.
With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider
that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding
to the processing time of backing up/restoring a backup.

It does have its benefits, but do keep in mind that the decompression cost will be paid by Reads. Also if someone is using a restored VM, they might see the physical size increase disproportionately to the actual data written due to decompression. It's still a useful feature and it's good that it is optional.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

If there is possibility these parameters can be added for other providers they should be added to the existing API.
Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

• What defines a provider as Native?
• Do the native offerings differ in functionality or usage to the existing BackupOfferings?
• How will this look like in the UI? Will the user see Backup Offering and another Native Backup Offering?
• Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

Hello, @abh1sar

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings. Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

You can have exactly that using this implementation. Essentially backup repositories and secondary storages are the same, with different names. Using selectors, you can have offerings that are tied to specific secondary storages, or you can have offerings that go into multiple storages, it was made to be flexible.

Capacity is just for Admins for Backup Repository also, so that is not the issue. But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups. With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Again, you can have dedicated secondary storages using selectors.

If there is possibility these parameters can be added for other providers they should be added to the existing API. Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

While I was not the one that introduced the backup framework, looking at its design, it was clearly intended to have as little configuration as possible on the ACS side while leaving these details to the backup provider. If we add these parameters to the import backup offering API, I'm sure a lot of users will be confused when they do nothing for Veeam and Dell Networker. I did not intend to warp the original design of configuring the offerings on the provider side and only importing it to ACS.

This is why I created the concept of native offerings. As with KNIB (and NAS) the provider is ACS, and thus the configurations can still be made on the "backup provider", and the import API will follow the same logic it always had.

* What defines a provider as `Native`?

A provider is native if the backup implementation is made by ACS. Thus, the current native providers would be NAS and KNIB. Veeam and Networker are external providers.

* Do the native offerings differ in functionality or usage to the existing BackupOfferings?

They are used to configure the details that would be configured in the backup provider if it was an external provider, such as Veeam for example.

* How will this look like in the UI? Will the user see `Backup Offering` and another `Native Backup Offering`?

I have yet to add it to the GUI, if you have any suggestions for the GUI design, you are free to give your opinion. The GUI for the native backup offerings will be added in the future.

* Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

Again, you absolutely can do this with KNIB. But you may also choose not to do it, its flexible.

@JoaoJandre
Although I’m not fully convinced that the mixed use of Secondary Storage is a good approach, I appreciate the work you’ve put into this. It would probably have been better to discuss the idea with the community earlier in the process, rather than presenting the design document and PR after most of the development had already been completed.

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

I don't see any issue with "Native", and I'm not claiming that KNIB is the only "Native" provider. As I explained before:

• Native = CloudStack is the backup provider, implementing all of the processes necessary for the backup to be executed.
• External = CloudStack integrates an external solution that is responsible for the backups, such as Veeam and Dell Networker.

This PR does not remove or substitute anything, the provider I am proposing is a new option for users to use.

Also. The Validation was added in the last commits. All the features that were meant to be included in this PR are already here.

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

I don't see any issue with "Native", and I'm not claiming that KNIB is the only "Native" provider. As I explained before:

• Native = CloudStack is the backup provider, implementing all of the processes necessary for the backup to be executed.
• External = CloudStack integrates an external solution that is responsible for the backups, such as Veeam and Dell Networker.

This PR does not remove or substitute anything, the provider I am proposing is a new option for users to use.

We do not use the term "Native" in CloudStack. Plugin names are typically based on the name of the external device or backup repository they integrate with.

Please follow the same convention.

Hello, @weizhouapache, all

We do not use the term "Native" in CloudStack

Yes, we currently do not have any APIs using the term Native. But, this does not imply or require us to avoid using this expression when it is suitable.

Plugin names are typically based on the name of the external device or backup repository they integrate with.

Yes, that is correct. In the case of the new backup plugin, it is fully implemented by ACS, including backup creation, chain management, compression, validation and other related processes. Therefore, its name is fully coherent with its functionality. In other words, KNIB (KVM Native Incremental Backup) stands for:

• KVM: provides support for the KVM hypervisor;
• Native: Fully native to Apache CloudStack, i.e., all processes are implemented by the cloud platform orchestrator itself;
• Incremental Backup: Incremental backups are supported.

Please follow the same convention.

Sorry, but is this a established convention? I am not aware of any guideline stating that the term Native is not supported.

Again, if there is any semantic inconsistency with the plugin name KNIB (KVM Native Incremental Backup), we can address it. However, the plugin supports incremental backups, for KVM, natively in ACS; thus, the name accurately reflects what it delivers. It does not promote any concept or capability that is not actually provided.

@JoaoJandre, btw, thanks for all the effort you put into designing and implementing this backup plugin. It will certainly be a stable and efficient solution, with strong adoption across cloud providers and users.

Hello, @weizhouapache, all

We do not use the term "Native" in CloudStack

Yes, we currently do not have any APIs using the term Native. But, this does not imply or require us to avoid using this expression when it is suitable.

Plugin names are typically based on the name of the external device or backup repository they integrate with.

Yes, that is correct. In the case of the new backup plugin, it is fully implemented by ACS, including backup creation, chain management, compression, validation and other related processes. Therefore, its name is fully coherent with its functionality. In other words, KNIB (KVM Native Incremental Backup) stands for:

• KVM: provides support for the KVM hypervisor;
• Native: Fully native to Apache CloudStack, i.e., all processes are implemented by the cloud platform orchestrator itself;
• Incremental Backup: Incremental backups are supported.

Please follow the same convention.

Sorry, but is this a established convention? I am not aware of any guideline stating that the term Native is not supported.

Again, if there is any semantic inconsistency with the plugin name KNIB (KVM Native Incremental Backup), we can address it. However, the plugin supports incremental backups, for KVM, natively in ACS; thus, the name accurately reflects what it delivers. It does not promote any concept or capability that is not actually provided.

@JoaoJandre, btw, thanks for all the effort you put into designing and implementing this backup plugin. It will certainly be a stable and efficient solution, with strong adoption across cloud providers and users.

I don’t want to continue debating this topic, so I’ll just summarize my view:

• The mixed use of Secondary Storage is not a good approach.
• The term "Native" is unclear and may confuse users. It would be better to follow the existing naming conventions used for other network/storage/backup plugins and choose a more descriptive name.
• Please move API/DB/Java classes into the plugin as much as possible to keep the implementation isolated.

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@vishesh92 I finished adding tests.
@weizhouapache , @abh1sar, @DaanHoogland as far as I am aware, I have addressed all pending reviews.

@blueorangutan package

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17573

@blueorangutan package

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17574

@vishesh92 I finished adding tests. @weizhouapache , @abh1sar, @DaanHoogland as far as I am aware, I have addressed all pending reviews.

Thanks @JoaoJandre
Sorry, I have been a bit busy. I'll do another round of review next week. Hope that's fine.

@blueorangutan package

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17586