DNS Made Easy: DNS IP Fail Over really works !

Previously I mentioned I switched to using DNS Made Easy as work around for Centminmod.com’s load balanced Haproxy fail over implementation.

My Haproxy load balanced 5x LEB (Low End Box) VPSes for Centminmod.com are OpenVZ based VPSes with no way of making sysctl.conf edits for non-local binding of virtual IP addresses for load balancer fail over with either Keep-Alived or Heartbeat. Guess it’s the price you pay for cheap LEB OpenVZ based hosting at ~US$36/month total cost for the 5x LEB VPSes – 4x with BuyVM.net and 1x with ChicagoVPS.net.

OpenVZ VPS Haproxy Load balancer fail over solution:

DNS Made Easy was the solution to my problem at US$59.95/yr compared to cost of switching to KVM/Xen based VPSes for sysctl.conf editing privileges at up to US$360/yr. And today I got to see DNS Made Easy’s DNS IP fail over in action as the main Haproxy load balancer located on my BuyVM.net VPS lost connectivity to the provided internal IP address which Haproxy listens to. This seems to happen when BuyVM.net is migrating or upgrading VPS nodes to other VPS node host servers – the internal IP address looses connectivity momentarily.

All BuyVM.net VPSes come with a private internal IP address you can use for load balancing, separate MySQL server for MySQL replication etc etc. When Haproxy looses connectivity to the internal IP address it listens to for all VPSes in the load balancing cluster, it won’t be able to serve any pages even if the VPS itself is still up. I could choose to switch BuyVM.net VPS to listen to public IP address but it would count towards bandwidth quota unlike the free bandwidth via internal IP address.

The ChicagoVPS.net VPS serves as a secondary Haproxy load balancer which listens the public facing IP addresses of all VPSes. So I setup DNS Made Easy’s IP failover to use this secondary Haproxy load balancer ip in case of failure at primary Haproxy load balancer.

Haproxy load balancer fail over in action:

This IP fail over happened today ~2hrs ago got the email alerts and I got to see it in action so I thought I’d document it here. It didn’t last long though, less than 2hrs later the system automatically switched back to the primary Haproxy load balancer IP on BuyVM.net VPS once the system detected it was working.

First up, the ChicagoVPS VPS secondary Haproxy load balancer statistics show server1 and server3 are the ones down – one of which is the main Haproxy load balancer listening to internal IP addresses.

This is what you see in DMS Made Easy’s control panel for Centminmod.com – I blanked out specific details, but you can see in SM/FO (fail over) column the red alert icons indicating fail over IP has been triggered.

Specific account action log details when fail over actually occurred and what IP address was failed over.

The specific fail over configuration I used – with fail over sensitivity set to high and I set it up so the DNS Made Easy system checks for a html file which contains the word ‘up’ in it and set to max number of email alerts to 5.

The email address alert I get when primary Haproxy load balancer was down

Notification: System is DOWN
IP: xxx.xxx.xxx.xxx
System Desc: Centminmod.com DNSMADEEASY ALERT
Port: 80
Protocol: HTTP
2012-01-15 00:53:39.72 UTC
ID: 7550599
Previously failed: 25 times.

The email alert when it was back up

Notification: System is UP
IP: xxx.xxx.xxx.xxx
System Desc: Centminmod.com DNSMADEEASY ALERT
Port: 80
Protocol: HTTP
2012-01-15 01:42:50.419 UTC
Previously failed: 39 times.
ID: 7550599

The fail over sensitivity level of low, medium and high is a new DNS Made Easy feature which was added and didn’t make it’s way into their online documentation yet, so I had to contact their technical support to ask for clarification. By default the system is setup for medium level fail over sensitivity level.

• high sensitivity there are 3 instantaneous checks from 3 remote locations and only 2 remote location detected failures will trigger failover
• medium sensitivity there are 6 instantaneous checks from 6 remote locations and for failover to be triggered, 5 of the remote locations need to detect ip failure
• low sensitivity there are 9 instantaneous checks from 9 remote locations and failover occurs only when 8 remote locations detect ip failure

So higher the sensitivity level, the quicker the IP fail over system is triggered. I had it set to high sensitivity level for quicker Haproxy load balancer IP address fail over combined with low DNS TTL of 120 for my A records. I’m really liking the system and takes alot of head aches out of setting up IP fail over myself. Not bad for US$59.95/yr of which you get 25 domains hosted on a awesome IP Anycast based network  with 3 DNS fail over records included and additional 1 DNS fail over monitoring record cost US$4.95 or a pack of 10 costs US$45.95 :)