Comments on: How to Ruin Your Weekend and Other Hazards of Mis-Configured HA http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha Using technology to make businesses better Sun, 29 Jan 2012 15:26:32 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: John http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-691 John Sun, 29 Jan 2012 15:26:32 +0000 http://www.mooselogic.com/blog/?p=157#comment-691 Excellent explanation, never actually realized that Xenserver would 'fence' to protect it's data. Yeah, it happened to me too when a SAN controller did a failover. Three server setup with XS5.0, using HBA's. The only one that remained online was the pool master, the other 2 restarted kicking off VM's and all users at 8.17 in the morning. Ouch! The heartbeat link was gone for about 50 seconds due to the failover. Pretty short time on HA timeout too, 30 seconds is not much. Very interesting article, kudos! Excellent explanation, never actually realized that Xenserver would ‘fence’ to protect it’s data. Yeah, it happened to me too when a SAN controller did a failover. Three server setup with XS5.0, using HBA’s. The only one that remained online was the pool master, the other 2 restarted kicking off VM’s and all users at 8.17 in the morning. Ouch! The heartbeat link was gone for about 50 seconds due to the failover. Pretty short time on HA timeout too, 30 seconds is not much. Very interesting article, kudos!

]]> By: Ben Piper http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-685 Ben Piper Tue, 27 Dec 2011 15:29:41 +0000 http://www.mooselogic.com/blog/?p=157#comment-685 This is a fantastic explanation of host self-fencing! Thanks! This is a fantastic explanation of host self-fencing! Thanks!

]]> By: Meyer http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-668 Meyer Tue, 13 Sep 2011 13:06:15 +0000 http://www.mooselogic.com/blog/?p=157#comment-668 1rst, sorry 4 my poor english. Our citrix server farm (3) went into emergency mode after the ups starts (!!!) after a brown out in our district. After returning to the office, i checked the server. All running (including our heartbeat/nas), allthough not reachable via xencenter. Ping, ok, putty a mess, lousy responses, xsconsole shows no nics. you may ask for the vms, well they were running. I was really surprised. Ok, sherlock mode on and i found the reason why. All three server were plugged on the same switch. the switch was temporarily out of order (a few seconds, imho) until it returned to normal operation. These few seconds lasts in the result, that none of the server could find his heartbeat to the iSCSI/nas and fences (emergency mode). I seperate now the three server on different switches an add a single ups for the switch and the NAS (keeping up the heartbeat). Btw. the server are operated in a server room which is protected by a huge ups. Scaring (...) Last but not least: If one of the server without ha enabled (--forced) should be back to operation in the Pool, don't search 4 an xe command, just disable HA for the pool and activate it again, the missing slave joins the pool, thats all. 1rst, sorry 4 my poor english. Our citrix server farm (3) went into emergency mode after the ups starts (!!!) after a brown out in our district. After returning to the office, i checked the server. All running (including our heartbeat/nas), allthough not reachable via xencenter. Ping, ok, putty a mess, lousy responses, xsconsole shows no nics. you may ask for the vms, well they were running. I was really surprised. Ok, sherlock mode on and i found the reason why. All three server were plugged on the same switch. the switch was temporarily out of order (a few seconds, imho) until it returned to normal operation. These few seconds lasts in the result, that none of the server could find his heartbeat to the iSCSI/nas and fences (emergency mode). I seperate now the three server on different switches an add a single ups for the switch and the NAS (keeping up the heartbeat). Btw. the server are operated in a server room which is protected by a huge ups. Scaring (…)
Last but not least: If one of the server without ha enabled (–forced) should be back to operation in the Pool, don’t search 4 an xe command, just disable HA for the pool and activate it again, the missing slave joins the pool, thats all.

]]> By: chris http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-543 chris Fri, 18 Feb 2011 20:07:37 +0000 http://www.mooselogic.com/blog/?p=157#comment-543 This fixed me http://support.citrix.com/article/CTX126382 This fixed me http://support.citrix.com/article/CTX126382

]]> By: TheG http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-431 TheG Fri, 24 Sep 2010 21:14:26 +0000 http://www.mooselogic.com/blog/?p=157#comment-431 Nice Write Up (though could be misleading in some areas until you get to the comments)... It may however be a good idea to indicate that the afore mentioned is no longer an issue within the writeup, especially since you have confirmed same in your lab. What version of XenServer are you testing with? Nice Write Up (though could be misleading in some areas until you get to the comments)… It may however be a good idea to indicate that the afore mentioned is no longer an issue within the writeup, especially since you have confirmed same in your lab.

What version of XenServer are you testing with?

]]> By: Sid Herron http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-408 Sid Herron Wed, 08 Sep 2010 02:49:07 +0000 http://www.mooselogic.com/blog/?p=157#comment-408 @Oliver - Thanks for your input. Your comment prompted us to go back and re-test with the latest version of XenServer. Although our testing is not yet complete, it does appear that some of the conditions we described in this blog post no longer cause both XenServers to fence. Once our testing is complete, we'll be posting an update on our findings. @Oliver – Thanks for your input. Your comment prompted us to go back and re-test with the latest version of XenServer. Although our testing is not yet complete, it does appear that some of the conditions we described in this blog post no longer cause both XenServers to fence. Once our testing is complete, we’ll be posting an update on our findings.

]]> By: Oliver http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-403 Oliver Mon, 30 Aug 2010 11:11:55 +0000 http://www.mooselogic.com/blog/?p=157#comment-403 I don't understand this. I have two Xen Servers in a pool and I have no problems. I test it. First I remove the first server from network, the protectet machine starts on the other server, than I test it with the other server, also no problem. I have no fencing...... I don’t understand this. I have two Xen Servers in a pool and I have no problems. I test it. First I remove the first server from network, the protectet machine starts on the other server, than I test it with the other server, also no problem. I have no fencing……

]]> By: XenServer Host Is In Emergency Mode | Moose Logic Blog http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-173 XenServer Host Is In Emergency Mode | Moose Logic Blog Thu, 15 Apr 2010 22:50:10 +0000 http://www.mooselogic.com/blog/?p=157#comment-173 [...] stranded and goes into emergency mode to protect its own VMs. There are other ways this can happen (an incorrectly configured pool with HA turned on for example), but this is the most common reason that I have personally [...] [...] stranded and goes into emergency mode to protect its own VMs. There are other ways this can happen (an incorrectly configured pool with HA turned on for example), but this is the most common reason that I have personally [...]

]]> By: Charles Stevenson http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-57 Charles Stevenson Sat, 06 Feb 2010 06:31:09 +0000 http://www.mooselogic.com/blog/?p=157#comment-57 Thanks for this post! I have a three XenServer pool and had enabled HA tonight around midnight whilst doing other sorts of network maint... recabling a few things... I must've bumped a cable and suddenly both servers at the building I was in rebooted. I thought perhaps there was a power issue in the rack or that I'd been a klutz... then I looked in XenCenter and all three servers had 'fenced'. Ugh. At least I found this out late at night and read your post. I'm going to look into getting enough redundancy with switches and NIC bonding so that I can have HA enabled. Thanks for this post! I have a three XenServer pool and had enabled HA tonight around midnight whilst doing other sorts of network maint… recabling a few things… I must’ve bumped a cable and suddenly both servers at the building I was in rebooted. I thought perhaps there was a power issue in the rack or that I’d been a klutz… then I looked in XenCenter and all three servers had ‘fenced’. Ugh. At least I found this out late at night and read your post. I’m going to look into getting enough redundancy with switches and NIC bonding so that I can have HA enabled.

]]> By: Frost Hon http://www.mooselogic.com/blog/how-to-ruin-your-weekend-and-other-hazards-of-mis-configured-ha#comment-48 Frost Hon Tue, 26 Jan 2010 10:18:10 +0000 http://www.mooselogic.com/blog/?p=157#comment-48 Excellent Post, really love how your explain everything in detail. we are testing our HA as well and came across your article. Excellent Post, really love how your explain everything in detail. we are testing our HA as well and came across your article.

]]>