This is directly from our hosting service, Westhost:
As a result of the fire suppression system error which was triggered at the Data Center during a routine system check by one of the fire system vendors, critical hardware components were damaged across dozens of our servers. Our first course of action was to replace those damaged components. We were able to repair or replace almost every failed hardware component during the first 36 hours through a combination of our on-hand hardware supplies and an expedited hardware shipment from our primary supplier.
Unfortunately, many of these hardware problems impacted hard drives, causing hard drive failures. When this happens, data loss becomes an immediate focus. Retrieving lost data and restoring from backups can take up to 24 hours per server. This is where we are currently with a majority of our downed servers. Our backup process restores three servers simultaneously. We will get to each server as quickly as we can, but given the number of servers needing to be restored from backups, this places our ETA anywhere from 24 to 96 hours. In other words, it could very well be Friday of this week before all servers and accounts are back on line. If ANYTHING changes with regard to this timing (good or bad) we will immediately inform you. We profusely apologize for the "stops and starts", however, once one issue was resolved another problem surfaced which resulted in adjustments to the published ETA's.
In addition to the failed hard drives, we have also found other dedicated and shared servers that are experiencing file system problems. While accounts can generally function on these servers, file system problems can negatively impact the performance of the server. We are aware of these servers and are tackling these issues as they become evident.
What does this mean to you?Due to the length of this restoration process, we are currently preparing new servers in our WestHost 4.0 environment for you to get your site back up sooner if needed. These new servers will be ready for clients to start using within the next few hours. As soon as they are ready, we will send you further details. This free solution will temporarily enable you to get your email and a basic site up and running so you can communicate with your customers. We recognize the impact this has had on your business and we plan to offer compensation via free month(s) of service for your loss. Once again, we are 100% committed to doing what we can to restore your business and confidence in our company going forward.
For those who might have a question about the Data Center facility, it is important for you to note that we are co-locating at this location which is a Tier 4 data center managed by some of the best professionals in the industry. It has been fully operational throughout this entire process. Most of our systems are running perfectly fine, without any residual effects. Power, bandwidth, cooling, and other critical systems are working properly. This issue has purely been a result of a fire suppression error, and not as a result of any mistake by WestHost personnel or systems.
We will continue to provide email updates every few hours. Again, thank-you for your patience as we restore your services.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment