Category Archives: VMware

Geek Stuff, VMware

How hard can it be to find the correct IP

Leave a comment
I guess for some of you this is somehow obvious, but for me it was a refreshing experience that everything is pretty straight forward in the networking world. I ran into an environment with NSX-t where I had a DMZ like network area which included a HTTPS web service. I had control of a stripped-down firewall configuration wizard with no other access to NSX-t or firewall areas. Simple things first:  “Source ANY” to “Destination <myservicePublicIP>” Port TCP443 worked as expected and I was able to access the service by his public IP. The goal was to only allow access to this service from another NSX-t network area which where somehow only connected with each other over public IP and NSX-t internal routing. OK so I googled whatismyip and to my surprise I was not able to connect to the service when I entered this IP in the firewall rule. There was no way to get any additional access to NSX-t management (blackbox) and no Admin accessible. The IPs shown by tracert and other such tools didn´t worked neither. So something “special” was configured within NSX-t. Dead end? Remembering in my head if you always half the IP addresses it will take only some tries to find out the correct IP address for the firewall rule source entry. So, I started to guess the class A subnet based on other IPs that I had tested above and where shown at the tracert output. Third try with class a Class Subnet 54.0.0.0/8 was a hit and it worked. From there I started to increase the subnet mask by 1 and tried to refresh the webpage. If the webpage was shown I increased again the subnet mask by one. If not, I had to switch to the other remaining subnet and try again. Example: If 54.0.0.0/9 worked I set the firewall rule to 54.0.0.0/10 If 54.0.0.0/10 did not worked anymore I had to switch to the other remaining subnet in that subnet mask scope. So, I continued by testing 54.64.0.0/10. This worked again and I increased the subnet mask again by 1 to 54.64.0.0/11. And so on … and so on… After only ~30 tries I had found the correct IP. Overall it took me just some minutes for the 30 connection tests and 30 firewall rule updates o filter out the single IP out of the 4.294.967.296 “possible” IPs (IPv4) in the world. I was lazy with the subnet calculation and just used a tool for it: http://www.ip-tools.net/ipcalc You can type in any IP address and a subnet mask and it will show you the subnet (“net”), broadcast IP and IP range, so it was easy to find the other remaining subnet if needed. I was just surprised how fast it was to manually track down the IP out of the possible amount of all Internet IPs. It is always refreshing that in the network world everything has it´s place and “just work”      ;-)
Backup, Hyper-v, VMware

SAP HANA Backup with Veeam

1 Comment

Hi,
my colleague and friend Tom Sightler created an toolset to backup SAP HANA with Veeam Backup & Replication. He documented everything in the Veeam Forum:
https://forums.veeam.com/veeam-backup-replication-f2/sap-b1-hana-support-t32514.html

Basically it follows the same way that storage systems like NetApp use for Backup of HANA. You implement in Veeam Pre and Post Scripts that makes HANA aware of the Veeam Backups. As well Logfile Handling is included (how many backup data do you want to keep on HANA system itself?).

In case of a DB restore, you go to HANA Studio and can access the backup data on HANA system directly. If you need older versions you can restore them with Veeam File Level Recovery Wizard or more comfortable with the Veeam Enterprise Manager File Restore (Self Services) and hit the rescann button at HANA Studio restore wizard. They are detected and you can proceed with the restore.

 

CU andy

VMware

ESXi NTP Service not working?! (for Example with Windows NTP Server)

Leave a comment

Hi,
sometimes ESXi NTP Service is a bit tricky. (Configuration see kb.vmware.com/kb/2012069)
When it do not update the time but all outputs show the correct NTP settings when you type in “watch ntpq” on ESXi console,
you can try to add the NTP Version to the /etc/ntp.conf .

Change
“Server <NTP name or IP>”
to
“Server <NTP name or IP> version 3”

Specifically with Windows NTP Server you had to add this option to it.

Yes it is written at http://kb.vmware.com/kb/1005092
but It is at the end of the document hidden and in most cases people do the first steps in the document before they read the whole document and waste time. And… I didn´t found this solution at Google.

Update: There is as well a good KB that describe the Windows NTP + VMware ESXi configuration: http://kb.vmware.com/kb/1035833

Backup, Exchange, Hyper-v, VMware

Interview with Anton Gostev about “Agentless” Backup

Leave a comment

Hi everybody,

as you might know Veeam do not install backup agents on the VMs to process application aware and application- and filesystem consistent backups. Veeam looks into the VM and it´s applications and register plus start an according run time environment that allow application aware backups.

We had lately an internal discussion about this topic and Anton Gostev Vice President of Product Management at Veeam Software allowed me to share his thoughts and ideas behind Veeam’s unique approach.

Andreas Neufert:  “Let´s talk first about the definition of Agents. According to http://en.wikipedia.org/wiki/Software_agent an Agent is defined as an installed software piece that stays on the servers. Veeam´s unique functionality register (install) start and unregister (uninstall) his run time environment just for job processing. Anton why do you think this is better than installed agents? ”

Anton Gostev: “All problems which cause issue known as “agent management hell” are brought by the persistency requirement
…(of that Agents from other solutions)…

– Need to constantly deploy agents to newly appearing VMs
– Need to update agents on all VMs
– Need to babysit agents on all VMs to ensure reliability (make sure it behaves correctly in the long run – memory leaks, conflicts with our software etc.)
Auto-injected temporary process addresses all of these issue, and the server stay clean of 3rd party code 99.9% of time.”

Andreas Neufert: “I think we all were at the point where we need to install a security patch in our application and have to wait till the backup vendor released a compatible backup agent version. Or I can remember that we have to boot all Servers because of a new version of such an agent (before I joined Veeam). But what happens if the Application Server/VM is down?”

Anton Gostev: “… Our architecture address the following two issues …
– Persistent agent (or in-guest process) requires VM from running at the time of backup in order to function. But no VMs are running 100% of time – some can be shutdown! We are equally impacted, however the major difference is that we do not REQUIRE that in-guest process was operating at the time of backup (all item-level recoveries are still possible, they just require a few extra steps). This is NOT the case with legacy agent-based architectures: shutdown VM means no item-level recoveries from the corresponding restore point.
– Legacy agent-based architectures require network connectivity from backup server to guest OS – rarely available, especially in secure or public cloud environments. We are not impacted, because we can failover to network-less interactions for our in-guest process. This is NOT the case with legacy agent-based architectures: for them it means no application-aware backup, and no item-level recoveries from the corresponding restore point.

Andreas Neufert: “Everyone who operate a DMZ knows the problem. You isolated the whole DMZ from your normal internal network, but the VMs need a network connection to the backup server which hold as well data from other systems. So the Veeam approach can bring additional security to the DMZ environment. Thank you Anton!”

Thanks for reading. Please send me comments if you want more interviews on this blog.

Cheers… Andy

Backup, VMware

vCenter connection limitation and backup in big environments

Leave a comment

Hi Team,

Update from 2019-05-20: Since some years the below SOAP modifications within vCenter are not needed anymore as Veeam caches all needed vCenter information in RAM which reduced the vCenter connection count drastically at the backup window. See Broker Service note here: https://helpcenter.veeam.com/docs/backup/vsphere/backup_server.html?ver=95u4

My friend and workmate Pascal Di Marco ran into some VMware connection limitation while backing up 4000VMs in a very short backup window.

If you ran a lot of parallel backup jobs that use the VMware VADP backup API you can run into 2 connection limitations… on vCenter SOAP connections and on some limitation on NFC buffer size on ESXi side.

All backup vendors that use VMware VADP implement in their product the VMware VDDK kit which help the backup vendor with some standard API calls and it also helps to read and write data. So all backup vendors have to deal with the VDDK own vCenter and ESXi connection count in addition to their own connections. VDDK connections vary from VDDK version to version.

So if you try to backup thousands of VMs in a very short time frames you can hit these limitations.

In case you hit that limitation, you can increase the vCenter SOAP connection limitation from 500 to 1000 by this VMware KB 2004663 http://kb.vmware.com/kb/2004663
EDIT: In vCenter Server 6.0, vpxd.cfg file is located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx

As well you can optimze the ESXI Network (NBD) performance by  increasing the NFC buffer size from 16384 to 32768 MB and optimize the Cache Flush interval from 30s to 20s by VMware KB 2052302  http://kb.vmware.com/kb/2052302

Backup, VMware

Link: Pernixdata + Veeam Scripts for Direct SAN processing

Leave a comment

Hi everybody…

My friend and workmate Preben created some cool scripts to use the VMware VADP Direct SAN mode together with Pernixdata write caching.

The Problem here is that Pernixdata commits writes out of the cache and not all data is on disk to process VADP based backups in Direct SAN mode.  The provided scripts just disable the caching for the time of backup

You can find the post here:
http://poulpreben.com/veeam-direct-san-backups-and-pernixdata-fvp/

Backup, Hyper-v, Videos, VMware

Lotus Domino Backup with Veeam Backup & Replication

3 Comments

Hi everybody,

on customer request I created a video that shows backup and single mail restore for lotus domino with Veeam Backup & Replication.

A Lotus Domino is non VSS aware (anyway this is the case under Linux).  So you have only 2 options for consistent backups as IBM do not support VSS Filesystem only backups:

  1.  Shutdown the VM => Service offline or at cluster do this only on one side.
  2. Close the connections and write the cache to disk

The question is why should I use a non Domino Backup API based backup?

For Veeam the answer is:

  • Ultra Fast Serivce Restore with Instant VM Recovery (2min + OS boot)
  •  Easy to use Single Mail/Document restore
  • Automated Restore Tests with SureBackup that test if a VM is Restoreable, OS boot, Network Connection is online and Domino Services are up and running on a daily base.
  • And finally a backup on Image Level with Change Block Tracking based Incremental Forever is very efficient even at a Domino Server with high change rate.

Enjoy the video

https://www.veeam.com/videos/backing-up-non-vss-aware-applications-ibm-lotus-domino-4867.html

Backup, VMware

Tips & Tricks for Backup & Replication not directly related to Veeam (continuously updated)

Leave a comment

Hi,

there are some general tips and tricks for Backup & Replication that are not directly related with Veeam Software. I will update this blog post from time to time to share these tips.

 

1) Format Backup Target disks with “/l” to avoid  that NTFS blocks access to your very large and frequently updated (fragmented) backup files.
format /FS:NTFS /L
This will take a while and will overwrite the selected folume (data loss be carefull).
If you have Win7/Win2008R2 you need to first install the following patch: http://support.microsoft.com/kb/967351/en-us

2) Fix CPU load VMXnet3 network card bug if you use one virtualized backup server/role or an VM with high disk load:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2039495

VMware

No SSO visible at vSphere Web Client 5.5 (Reset vSphere Administrator password because of “!” in the password)

Leave a comment

Hi,

maybe this is an well known thing, but I never used an “!”  at vSphere Password before and experienced some “inconvenience”. I installed vCenter Server and was not able to see the vcenter and also not able to access SSO configuration, because it was not there. (Remeber SSO configuration in 5.5 was rewritten and it is normaly found directly in vSphere Web Client – Administration tab.

“!” are not an allowed character at vSphere SSO Administrator password, but Setup process allow it.
If you used it in you password, you are able to logon to Web Client but you see no vSphere Server nor are you able to see the SSO configuration area. If you do not want to install your Server from scratch, you can use the following command to change the password:

Windows:
cmd
c:Program FilesVMwareInfrastructureVMwareCISvmdirdvdcadmintool.exe
3
cn=Administrator,cn=users,dc=vSphere,dc=local

Appliance:
Open SSH connection
/usr/lib/vmware-vmdir/bin/vdcadmintool
3
cn=Administrator,cn=users,dc=vSphere,dc=local

The funny thing is, that you need to take care as well, that there is not an “!” in the auto generated password ;o)

After that login to Web Client and you can access SSO configuration now. In my case the vSphere Server showed up automatically after this as well.

VMware

Performance comparison vRDM, pRDM und VMDK

Leave a comment

Check out this old ESX3.5 article about performance differences between vRDM, pRDM and VMDK. You can see that even in these good old ESX3.5 days there was no significant performance gap. As actual VMware Volumes do not have the 2TB limitation anymore, there is no real blocker to use VMDK (and vRDM).
If you use vRDM don´t forget to reserve some space next to the vmx file for snapshots (e.g. Backup-Snapshot-helper) :
http://www.vmware.com/files/pdf/vmfs_rdm_perf.pdf

VMware

Automatic VMware vSphere Client login

Leave a comment

Hi everybody,

in my Labs and presentations I find it very unhandy to typein connection and username/passwords at my vcenter client.

You can crete a link with parameter to do so.
Also you can change the UI language to your choice.

Parameter:

-u Username
-p Password
-locale Location  (e.g.  de-DE for GermanUI  and en-US for Englisch UI)
-s für VCenter Server or ESX Host

Example:
“C:Program Files (x86)VMwareInfrastructureVirtual Infrastructure ClientLauncherVpxClient.exe” -u administrator -p XXX -s 192.168.1.1 -locale en-US

Maybe it is not a good idea to save your password for your production environment in a link, but it is very helpful for you lab environment. If you do not add the -p parameter, it will ask you for the password, but the rest of the settings are filled out

Grüße Andy