> You’re trying to say that those affected by yesterday’s
> downtime hadn’t built in any reliability to their setup.
> The device that failed was a SAN, which is a measure
> of reliability and redundancy on its own. It’s not a
> single server or a single hard drive. Brightbox provide
> this by default, so everybody complaining had taken
> steps to build in reliability by choosing Brightbox in
> the first place.

I absolutely am not saying that you didn’t build reliability in, what I am saying is that you made a choice as to what level of reliability was appropriate for the service/content/whatever that you are hosting on that box.

Brightbox use redundant disks inside a SAN and a redundant MySQL cluster, this is a level of redundancy and you chose it over a cheaper option which did not do this. Good for you… and you already went further than many do. But, consciously or subconsciously, you chose that it was not worth the money to increase that reliability with load balancing just as I have chosen not to replicate my entire hosting platform somewhere else.

I have invested what I believe to be an appropriate amount of money to get the best reliability I could for the service I am trying to make available. So did you and we just made a different decision. What I am saying is that was your decision, just like it was mine to not invest in a backup hosting provider if the Brightbox data centre slides into the sea one day.

> So ultimately I’m still not saying its Brightbox’s fault
> (these things happen) but your point is misguided
> and looks from a pretty poor perspective about the
> people complaining.

I did not mean to judge the people complaining, I even said that they were naturally upset. I would have been upset too if I were in that position. I just said they knew the world we live in is not perfect and that “shit happens” and they made a decision about what was an appropriate level of redundancy for the data/service they were protecting.

By load balancing you’ve doubled up the existing reliability level, which is obviously a good thing but I am assuming you didn’t need to share storage between the web servers, which introduces an additional cost that isn’t in any way an easy decision.

The same concept could be taken to the MySQL Cluster, its reliable and has redundancy by default, but if that fails you’ll be down like everyone else, no matter how many load balanced web servers you have, and the cost to remedy that isn’t an easy decision either.

These things happen to every hosting provider (trust me I know :(), and as a new business you are exposed to them until you can fork up the cash to make sure it doesn’t happen, in the mean time you just have to hope you’re lucky. No hosting provider can provide you with 100% reliability until you can provide them with the ££££.

So ultimately I’m still not saying its Brightbox’s fault (these things happen) but your point is misguided and looks from a pretty poor perspective about the people complaining, in fact you’re still in the same boat as them. If you want to really say you’ve got 100% reliability, you need a completely duplicated setup in another data centre ;)

I use exactly the same set up for the McSweeney’s app (http://iphone.mcsweeneys.net) and one of my boxes *was* affected by the mega-outage yesterday. The load balancer kept things running though :)

Russell.