Tuesday, February 23, 2010
The latest update
This is directly from our hosting service, Westhost:
As a result of the fire suppression system error which was triggered at the Data Center during a routine system check by one of the fire system vendors, critical hardware components were damaged across dozens of our servers. Our first course of action was to replace those damaged components. We were able to repair or replace almost every failed hardware component during the first 36 hours through a combination of our on-hand hardware supplies and an expedited hardware shipment from our primary supplier.
Unfortunately, many of these hardware problems impacted hard drives, causing hard drive failures. When this happens, data loss becomes an immediate focus. Retrieving lost data and restoring from backups can take up to 24 hours per server. This is where we are currently with a majority of our downed servers. Our backup process restores three servers simultaneously. We will get to each server as quickly as we can, but given the number of servers needing to be restored from backups, this places our ETA anywhere from 24 to 96 hours. In other words, it could very well be Friday of this week before all servers and accounts are back on line. If ANYTHING changes with regard to this timing (good or bad) we will immediately inform you. We profusely apologize for the "stops and starts", however, once one issue was resolved another problem surfaced which resulted in adjustments to the published ETA's.
In addition to the failed hard drives, we have also found other dedicated and shared servers that are experiencing file system problems. While accounts can generally function on these servers, file system problems can negatively impact the performance of the server. We are aware of these servers and are tackling these issues as they become evident.
What does this mean to you?Due to the length of this restoration process, we are currently preparing new servers in our WestHost 4.0 environment for you to get your site back up sooner if needed. These new servers will be ready for clients to start using within the next few hours. As soon as they are ready, we will send you further details. This free solution will temporarily enable you to get your email and a basic site up and running so you can communicate with your customers. We recognize the impact this has had on your business and we plan to offer compensation via free month(s) of service for your loss. Once again, we are 100% committed to doing what we can to restore your business and confidence in our company going forward.
For those who might have a question about the Data Center facility, it is important for you to note that we are co-locating at this location which is a Tier 4 data center managed by some of the best professionals in the industry. It has been fully operational throughout this entire process. Most of our systems are running perfectly fine, without any residual effects. Power, bandwidth, cooling, and other critical systems are working properly. This issue has purely been a result of a fire suppression error, and not as a result of any mistake by WestHost personnel or systems.
We will continue to provide email updates every few hours. Again, thank-you for your patience as we restore your services.
As a result of the fire suppression system error which was triggered at the Data Center during a routine system check by one of the fire system vendors, critical hardware components were damaged across dozens of our servers. Our first course of action was to replace those damaged components. We were able to repair or replace almost every failed hardware component during the first 36 hours through a combination of our on-hand hardware supplies and an expedited hardware shipment from our primary supplier.
Unfortunately, many of these hardware problems impacted hard drives, causing hard drive failures. When this happens, data loss becomes an immediate focus. Retrieving lost data and restoring from backups can take up to 24 hours per server. This is where we are currently with a majority of our downed servers. Our backup process restores three servers simultaneously. We will get to each server as quickly as we can, but given the number of servers needing to be restored from backups, this places our ETA anywhere from 24 to 96 hours. In other words, it could very well be Friday of this week before all servers and accounts are back on line. If ANYTHING changes with regard to this timing (good or bad) we will immediately inform you. We profusely apologize for the "stops and starts", however, once one issue was resolved another problem surfaced which resulted in adjustments to the published ETA's.
In addition to the failed hard drives, we have also found other dedicated and shared servers that are experiencing file system problems. While accounts can generally function on these servers, file system problems can negatively impact the performance of the server. We are aware of these servers and are tackling these issues as they become evident.
What does this mean to you?Due to the length of this restoration process, we are currently preparing new servers in our WestHost 4.0 environment for you to get your site back up sooner if needed. These new servers will be ready for clients to start using within the next few hours. As soon as they are ready, we will send you further details. This free solution will temporarily enable you to get your email and a basic site up and running so you can communicate with your customers. We recognize the impact this has had on your business and we plan to offer compensation via free month(s) of service for your loss. Once again, we are 100% committed to doing what we can to restore your business and confidence in our company going forward.
For those who might have a question about the Data Center facility, it is important for you to note that we are co-locating at this location which is a Tier 4 data center managed by some of the best professionals in the industry. It has been fully operational throughout this entire process. Most of our systems are running perfectly fine, without any residual effects. Power, bandwidth, cooling, and other critical systems are working properly. This issue has purely been a result of a fire suppression error, and not as a result of any mistake by WestHost personnel or systems.
We will continue to provide email updates every few hours. Again, thank-you for your patience as we restore your services.
Monday, February 22, 2010
good thing it's the offseason
New fix ETA of 10pm EST. Could be up before then... but at this point I'm not counting on anything. Check here or on Facebook for updates.
Site still down : (
Here's the latest word from our hosting service:
On Saturday, February 20 2010, at approximately 2:20 p.m. MST during an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers. Not all servers have been affected by the discharge. We are working to restore servers and services as quickly as possible. Due to the nature of this interruption we will be experiencing extended outages.
We might be back up by 10am EST this morning. Stay tuned.
On Saturday, February 20 2010, at approximately 2:20 p.m. MST during an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers. Not all servers have been affected by the discharge. We are working to restore servers and services as quickly as possible. Due to the nature of this interruption we will be experiencing extended outages.
We might be back up by 10am EST this morning. Stay tuned.
Sunday, February 21, 2010
Server still down...
Looks like the server issues that started yesterday have extended in to today... this is the official word from our host:
"We are experiencing sporadic issues on several servers and are working to resolve them as quickly as possible."
ETA for the fix 10pm EST.
Wow.
"We are experiencing sporadic issues on several servers and are working to resolve them as quickly as possible."
ETA for the fix 10pm EST.
Wow.
Monday, December 29, 2008
All is well
We experienced some down time from yesterday into today due to issues with the domain registrar. We're now back up and it's business as usual.
Subscribe to:
Posts (Atom)