Tag Archives: Backup

VM File Server with enabled dedup – High change rate every 28 days and how to avoid this

We see more and more customers enabling Windows deduplication within a VM to save space on the file servers. Even more with Windows 2016 this will become more and more the standard.

With deduplication enabled you will see a ~20x higher change rate every 28 days at the block level backups (e.g. Veeam). The root cause is the garbage collection run of the Windows deduplication engine.

You can find more informations here:

…and can discuss the solutions here:

SAP HANA Backup with Veeam

my colleague and friend Tom Sightler created an toolset to backup SAP HANA with Veeam Backup & Replication. He documented everything in the Veeam Forum:

Basically it follows the same way that storage systems like NetApp use for Backup of HANA. You implement in Veeam Pre and Post Scripts that makes HANA aware of the Veeam Backups. As well Logfile Handling is included (how many backup data do you want to keep on HANA system itself?).

In case of a DB restore, you go to HANA Studio and can access the backup data on HANA system directly. If you need older versions you can restore them with Veeam File Level Recovery Wizard or more comfortable with the Veeam Enterprise Manager File Restore (Self Services) and hit the rescann button at HANA Studio restore wizard. They are detected and you can proceed with the restore.


CU andy

Active Directory and Veeam


in this post you will find more and more information`s of how to protect Active Directory with Veeam.

For the start let me share the following 3 things with you:

Veeams Userguide for Veeam Explorer for ActiveDirectory:

To be able to restore a user account or machine account back to original place, you need a existing thumbstone in you AD for it.
A very good Thumbstone documentation including all kind of version/lifetime settings can be found in the following article.
Yes, it is in German, but you will get the point by just have a look at the lists and reg keys.
https://www.faq-o-matic.net/2006/07/28/das-geheimnis-der-tombstone-lifetime/ (German)
http://www.microsofttranslator.com/bv.aspx?from=de&to=en&a=http://www.faq-o-matic.net/2006/07/28/das-geheimnis-der-tombstone-lifetime/ (Google Translation)


When you install your first Active Directory Server it will as well create the recovery certificate for Windows EFS encryption. It is used for all domain members. It will be automatically placed on the c: drive. In any way protect this certificate with multiple backups.  Without it you can not renew the certificate (default lifetime is 3 year) or restore EFS encrypted data.  You can configure the certificate renew process by RSOP.msc and browse to Computer Configuration\Windows Settings\Security Settings\Public Key Policies\Encrypting File System

Interview with Anton Gostev about “Agentless” Backup

Hi everybody,

as you might know Veeam do not install backup agents on the VMs to process application aware and application- and filesystem consistent backups. Veeam looks into the VM and it´s applications and register plus start an according run time environment that allow application aware backups.

We had lately an internal discussion about this topic and Anton Gostev Vice President of Product Management at Veeam Software allowed me to share his thoughts and ideas behind Veeam’s unique approach.

Andreas Neufert:  “Let´s talk first about the definition of Agents. According to http://en.wikipedia.org/wiki/Software_agent an Agent is defined as an installed software piece that stays on the servers. Veeam´s unique functionality register (install) start and unregister (uninstall) his run time environment just for job processing. Anton why do you think this is better than installed agents? ”

Anton Gostev: “All problems which cause issue known as “agent management hell” are brought by the persistency requirement
…(of that Agents from other solutions)…

– Need to constantly deploy agents to newly appearing VMs
– Need to update agents on all VMs
– Need to babysit agents on all VMs to ensure reliability (make sure it behaves correctly in the long run – memory leaks, conflicts with our software etc.)
Auto-injected temporary process addresses all of these issue, and the server stay clean of 3rd party code 99.9% of time.”

Andreas Neufert: “I think we all were at the point where we need to install a security patch in our application and have to wait till the backup vendor released a compatible backup agent version. Or I can remember that we have to boot all Servers because of a new version of such an agent (before I joined Veeam). But what happens if the Application Server/VM is down?”

Anton Gostev: “… Our architecture address the following two issues …
– Persistent agent (or in-guest process) requires VM from running at the time of backup in order to function. But no VMs are running 100% of time – some can be shutdown! We are equally impacted, however the major difference is that we do not REQUIRE that in-guest process was operating at the time of backup (all item-level recoveries are still possible, they just require a few extra steps). This is NOT the case with legacy agent-based architectures: shutdown VM means no item-level recoveries from the corresponding restore point.
– Legacy agent-based architectures require network connectivity from backup server to guest OS – rarely available, especially in secure or public cloud environments. We are not impacted, because we can failover to network-less interactions for our in-guest process. This is NOT the case with legacy agent-based architectures: for them it means no application-aware backup, and no item-level recoveries from the corresponding restore point.

Andreas Neufert: “Everyone who operate a DMZ knows the problem. You isolated the whole DMZ from your normal internal network, but the VMs need a network connection to the backup server which hold as well data from other systems. So the Veeam approach can bring additional security to the DMZ environment. Thank you Anton!”

Thanks for reading. Please send me comments if you want more interviews on this blog.

Cheers… Andy

vCenter connection limitation and backup in big environments

Hi Team,

my friend and workmate Pascal Di Marco ran into some VMware connection limitation while backing up 4000VMs in a very short backup window.

If you ran a lot of parallel backup jobs that use the VMware VADP backup API you can run into 2 connection limitations… on vCenter SOAP connections and on some limitation on NFC buffer size on ESXi side.

All backup vendors that use VMware VADP implement in their product the VMware VDDK kit which help the backup vendor with some standard API calls and it also helps to read and write data. So all backup vendors have to deal with the VDDK own vCenter and ESXi connection count in addition to their own connections. VDDK connections vary from VDDK version to version.

So if you try to backup thousands of VMs in a very short time frames you can hit these limitations.

In case you hit that limitation, you can increase the vCenter SOAP connection limitation from 500 to 1000 by this VMware KB 2004663 http://kb.vmware.com/kb/2004663
EDIT: In vCenter Server 6.0, vpxd.cfg file is located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx

As well you can optimze the ESXI Network (NBD) performance by  increasing the NFC buffer size from 16384 to 32768 MB and optimize the Cache Flush interval from 30s to 20s by VMware KB 2052302  http://kb.vmware.com/kb/2052302

Link: Pernixdata + Veeam Scripts for Direct SAN processing

Hi everybody…

My friend and workmate Preben created some cool scripts to use the VMware VADP Direct SAN mode together with Pernixdata write caching.

The Problem here is that Pernixdata commits writes out of the cache and not all data is on disk to process VADP based backups in Direct SAN mode.  The provided scripts just disable the caching for the time of backup

You can find the post here:

Lotus Domino Backup with Veeam Backup & Replication

Hi everybody,

on customer request I created a video that shows backup and single mail restore for lotus domino with Veeam Backup & Replication.

A Lotus Domino is non VSS aware (anyway this is the case under Linux).  So you have only 2 options for consistent backups as IBM do not support VSS Filesystem only backups:

  1.  Shutdown the VM => Service offline or at cluster do this only on one side.
  2. Close the connections and write the cache to disk

The question is why should I use a non Domino Backup API based backup?

For Veeam the answer is:

  • Ultra Fast Serivce Restore with Instant VM Recovery (2min + OS boot)
  •  Easy to use Single Mail/Document restore
  • Automated Restore Tests with SureBackup that test if a VM is Restoreable, OS boot, Network Connection is online and Domino Services are up and running on a daily base.
  • And finally a backup on Image Level with Change Block Tracking based Incremental Forever is very efficient even at a Domino Server with high change rate.

Enjoy the video


Tips & Tricks for Backup & Replication not directly related to Veeam (continuously updated)


there are some general tips and tricks for Backup & Replication that are not directly related with Veeam Software. I will update this blog post from time to time to share these tips.


1) Format Backup Target disks with “/l” to avoid  that NTFS blocks access to your very large and frequently updated (fragmented) backup files.
format /FS:NTFS /L
This will take a while and will overwrite the selected folume (data loss be carefull).
If you have Win7/Win2008R2 you need to first install the following patch: http://support.microsoft.com/kb/967351/en-us

2) Fix CPU load VMXnet3 network card bug if you use one virtualized backup server/role or an VM with high disk load:

Windows 8.1 and Veeam Backup & Replication

Hi everybody….

sometime the easiest things do not work and you have a tough time to find the root cause.

A customer of mine uses an virtual Windows 8.1 as a Veeam Backup Server and also as a VMware HotAdd Proxy for Branch Offices.  For security reasons the UAC and Windows Firewall was enabled and username administrator disabled.

We found 3 major challenges in this situation:

– Backup & Replication was not able to run on an other local user than “Computername\Administrator”  (“Can not access admin$ share” error.
– Random disconnect of VMware Tools with stunning Backup Jobs.
– After adding the Branch office Backup & Replication Server itself as a Veeam proxy to Head Quarters Backup & Replication server, local hot add processing was not possible anymore. Manual hotadd of disks was still possible… strange

Solutions for this situation:
– Enable File&Print sharing to use another local admin user than “Administrator.


– The second one was fixed by enabling “high performance”  or “Höchstleistung” at the windows power options.

– Hotadd processing problem was related to different patch levels of B&R in the branch office.  The HQs Backup & Replication Server was on a higher patch level and local branch office server was not able to process hotadd anymore. Running same patch level solved it.

Happy backup…. Andy


Recommended Hyper-V Patches (and Failover Cluster Patches)

Hi everybody,

in the last time more and more customers test Hyper-V and migrate their workloads.
Many of them are not aware, that Hyper-V need some critical updates for a stable operation (and Backup).

Please check the following Links:

Veeam list of all patches that are recommended before doing backups (independent which backup software you use):
And here you can find the links to the Microsoft recommended patches.
If you use CSV,CSVv2 or SMB3 Shared Volumes check also the second link of each OS Version.
Win2012 R2
Win2012 R2 (Hyper-V) Failover Clusterhttp://support.microsoft.com/kb/2920151/de

Win2012 Hyper-V
Win2012 (Hyper-V) Failover Cluster
Win2008R2 SP1 Hyper-V
Win2008R2 SP1 (Hyper-V) Failover Cluster
Win2008R2 Hyper-V
Win2008R2 (Hyper-V) Failover Cluster

Prioritisation of Veeam Backup & Replication Proxy Modes from my field experience.

Update 1: 23.05.2016 => Veeam Backup & Replication v9 + new best practices.
Hi everybody,
just want to share with you a short list of Veeam Backup & Replication Proxy modes, because I got so many questions about it in the past.
VMware Backup from FibreChannel Block Storage.:
Priority 1:
For most common VMs (90%) I would use Veeams Direct Storage (Direst SAN) backup mode at backup and HotAdd (implement virtual Proxies) at restore for best performance.
For the biggest VMs (10%) with high change rates use Veeam Storage Integration (Backup from Storage Snapshot) to optimize VMware SnapShot commit processes. This feature is available for HP 3PAR StoreServe / HP StoreVirtual incl. VSA / NetApp ONTAP systems and EMC VNX(e). Nimble will follow this year. If you do not have this feature, use standard processing from above.
As Direct SAN need FibreChannel Access and FC passthrough is not really supported, you need physical Veeam Proxy Server.
Priority 2:
If you want to use virtual only infrastructure, go with 10GbE Interfaces at VMkernel, 10GbE Veeam Proxy Servers and use the Veeam Network Mode (NBD) mode. This mode is limited for a maximum throughput of 40% of the VMKernel Interface (at multiple parallel streams). You can use HotAdd for faster restore.
Priority 3:
Use Hotadd if you want to go with virtual proxies and there is only a 1GbE network.
What you should not do:
Avoid HotAdd backup processing in big environments. By design of VMware it will bring extra load on vCenter and singnificantly increase the chance that VMware get lost on his own snapshots (orphaned snapshots). As well by design of VMware VM stuns can happen at snapshot commit. If you really want to go with it, consider  ESXi bound Veeam Proxies with special Veeam registry setting. Ask Veeam Support or a SE for design and Reg Key.
VMware Backup iSCSI Block Storage.:
The priority list is the same then FC Block Storage above.
As it is iSCSI you can use virtual Direct Storage (Direct SAN) servers which should be priority 1 if you want to go with virtual Veeam Proxies. However physical Server reduce the load on your VMware Servers significantly.

VMware Backup from NFS (File) Datastores:

Priority 1:
For most common VMs (90%) I would use Veeams Direct Storage (new Veeam Direct NFS) backup mode for backup and restore. Direct NFS is the fastest restore method within Veeam as it is written from scratch by Veeam and do not leverage the VMware VDDK kit.

For the biggest VMs with high change rates use Veeam Storage Integration (Backup from Storage Snapshot) to optimize VMware SnapShot commit processes. This feature is available for  NetApp ONTAP systems and EMC VNX(e) (HP 3PAR and StoreVirtual do not have a NFS options). Nimble will follow this year. If you do not have this feature, use standard processing from above.
You can use virtual or physical Servers for processing. However physical Server offload the backup load from your hosts.
Priority 2 (or better say “No priority”):
As there is no downside of using Direct NFS method I highly recommend to use it. However if you need another backup method, go with 10GbE Interfaces at VMkernel and Veeam Proxy Servers in Network Mode (NBD). This mode is limited for a maximum throughput of 40% of the VMKernel Interface. You can use Direct NFS or HotAdd for faster restore.
What you should not do (in no way!):
Avoid HotAdd backup processing in ANY NFS  environments. By design of VMware it will bring extra load on vCenter and singnificantly increase the chance that VMware get lost on his own snapshots (orphaned snapshots). As well by design of VMware VM stuns WILL happen at snapshot commit, specifically within Linux VMs. If you really want to go with it, consider  ESXi bound Veeam Proxies with special Veeam registry setting. Ask Veeam Support or a SE for design and Reg Key.

Veeam Backup & Replication Proxy Mode Autodetection process works like this:

It will check Direct Storage Mode (Direct NFS/Direct SAN) first, 
then it will try HotAdd (Virtual Appliance Mode) and the it will use
NBD (Network Mode).
So if you want to use 10GbE NBD Mode instead of HotAdd as default, you have to select it manually at the Veeam Backup & Replication – Backup Infrastructure – Proxy settings.

New Video (German) about Exchange Backup and VMware related challanges.

Hi everybody,

I did a new updated Exchange Backup Video for Veeam in German.
There are a lot of tips and tricks for general Exchange Backup at VMware, but also cover Veeam Exchange Backup and Restore.
Specific Exchange DAG settings are also discussed.

I hope to get a lot of feedback from you ;-)


CU Andy

Veeam Backup & Replication Target Sizing Calcualtor (non official)

Hi everybody,

kudos to my colleague Timothy Dewin who wrote an excellent Veeam Backup & Replication Restore Point Simulator.

Keep in mind that this is not official and has not support or garantie that it work correct.
Also deduplication and compression is dependent to the data and is from customer to customer different.

The Default values are OK from my Feeling. If you have Domino DBs this is a bit too low because of Domino own compression.

And don´t forget to select Reverse Incremental if you want max space saving.

Here you can find the tool.

CU Andy

HA? DataProtection? DataRecovery? – IBM San Volume Controller SVC and Veeam Backup & Replication

Many DR scenatios didn´t reflect the need of the companies. Sometimes because of budget problems, sometimes of other things.
Why you want to bring data to another site?
HA? DataProtection? DataRecovery?

If you look at syncron mirrors. This is a typical HA szenario. If you go active-active there it can help to bring the servers back online as fast as of a boot. (VMware HA or othe cluster/failover solutions at legacy systems). Because this is an expensive one for legacy systems, this leeding to a scenario which VMs are ways better protected (HA) than my Tier 1 legacy systems. This and the advantages for maintenance (vmotion), power and cooling savings brings customer to a point that more and more Tier 1 applications are placed on virtualization.
Products that can be helpful here are IBM SVC, Datacore, Netapp, others.

Why this szenario has nothing to do with DP or DR.
You replicate only the disk data, so applications and DBs are not in a consistent state. Also the Applications and DBs are not in a Application Restore aware state. (see below)
Software errors are mirrored as well and if there are bugs in the code of the solution both sides are affected.
If you look at the storage and storage virtualization systems and their bigger and bigger fail domains you need a backup solution that fits your datarecovery needs.
Many customers looking because of the big fail domain at Replication solutions and looking for storage replication that can store more then 1 restore point (replicated snapshots).
Here you need to regard that your Servers/DBs/Applications are in the following states that you have no problems in case of restore/recovery. This is pretty the same demand for BackupSoftware.
a) Consistency (Application/OS/DBs). Basically before you do a replication/snapshot all RAM caches needs to be written to disk and no open write commands of the filesystem are there.
b) Application awareness: You have to set some settings in the OS/Application/DBs that after next boot(after restore/recovery) they jump into a mode that they needed in that secenario to avoid problems. A typical problem that is a good example for this is if you start active directory servers without this from different snapshots/recovery points you end up with an inconsistent and not supported active directory database.

So in most cases you need some software that can do this in addition to the storage snapshot replication base. IBM Flash Copy Manager or Netapp SnapManager for example. Or if you do this on the Software side Veeam (Virtualization) is a good example for this.

If you look at the size of your fail domains and your demand to bring back a lot of servers in a fail szenario. You need tools that can do this with an easy to use solution. Because of budget concerns many go only here with their backup software.

Most Backupsoftware can help to restore a Server in a timewindow you need (SLA) but if more than one server are affected this leeding into the situation where normal SLA are exceeded. There are some Backup Software that can help with this and can start systems directly out of the backup, start the OS and Applications and user can work, while later the data is transfered back to the repaired storage system. Veeam do this for example with VMs since 2010, but there are other solutions that can do this, for example NetBackup has announced that they will have that sort of recovery for VMs. Veeam holds the patents for this Instant VM Recovery.

Because this solutions uses the backup environment ressources to bring up the systems this can only be done with a limited number of systems, depending on the backup environment (20-100 VMs). So the idea was to add software based replication (Veeam for example) that has no direct interaction with storage System (storage fails can not harm this system) to replicate most critical systems to a separated datacenter over IP WAN links. There you can start (application aware) your core systems and you can recover over the next time your not so critical systems.

One cool thing I want to add from Veeam side. You are able to automatically test your Replicas (with v7 or newer) or Backups (v5 or newer) if they are able to restore your workloads. This scheduled restore checks test the  OS boot,  Network Connection and application Response.This include problems that are based on source side server. Simple example is a corrupt windows boot file that you do not detect in production. All backup solutions checks only the data with checksums > if same = backup/replication successfull. And you fail in case of restore. So this can be helpful to detect problems before you run into a failover scenario.

This described scenario reflect most budgets and can dramatically increase your operating safety.

Examples for a small szenario:

2 ESX Hosts with shared storage (redundant controllers)
Third ESX host on separate fire section or hosted datacenter. Which hold allsVMs as replicas.
Backup on the first side for fast restore and application/file restore.
Backup and recovery cheks on the second side of the replicas will help to prevent you from trouble in case you need your DR. 
Implementation of automatic Restore Checks with SureBackup and SureReplica.
 HW: Small Budget (Standard solution)
SW: Hyper-V or VMware Essentials + Veeam Essentials
Powerfull solution I think

Example for midsize – enterprise szenario:
VMware Hosts
Active-Active mirrored Storage (vdisk mirroring) for example with IBM SVC splitted over to datacenter on two fire sections. Third site with SVC Quorum.
Another datacenter many miles away with another Vmware environment which you use with Veeam replication of most critical systems.
Backup to third site (SVC Quorum) with Veeam (Fast Restore of Files/Objects/Servers).
Implementation of automatic Restore Checks with SureBackup and SureReplica.

What do you think? Yes it is very virtualization related but you get more operation safety than in legacy environments. Why not virtualize your biggest DBs on a 1:1 ratio and profit from this DR scenario. If you are concerned about VMware Snapshot commit szenarios or 2TB volume size limit, have a look at Hyper-V 3. The main idea behind this described szenario works there as well.

CU Any

Performance comparison vRDM, pRDM und VMDK

Check out this old ESX3.5 article about performance differences between vRDM, pRDM and VMDK. You can see that even in these good old ESX3.5 days there was no significant performance gap. As actual VMware Volumes do not have the 2TB limitation anymore, there is no real blocker to use VMDK (and vRDM).
If you use vRDM don´t forget to reserve some space next to the vmx file for snapshots (e.g. Backup-Snapshot-helper) :