While we continue to monitor closely, our storage vendor confirmed that the work we performed overnight has put us on the right path. We have some follow up work to do to ensure this never happens again. Thanks for your patience and understanding this week.
Posted Apr 28, 2023 - 09:46 EDT
Update
We are bringing services back online now.
Posted Apr 28, 2023 - 06:36 EDT
Update
Our maintenance continues. We should have things back online near 7AM EST.
Posted Apr 28, 2023 - 05:09 EDT
Update
Just a quick update. Storage maintenance is ongoing. We have confirmed through log analysis and general performance that imposing an outage was absolutely necessary to maintain the safety and integrity of data.
Posted Apr 28, 2023 - 00:27 EDT
Update
Our outage has begun. We will keep you posted here.
Posted Apr 27, 2023 - 20:04 EDT
Update
We are planning the outage starting at 8PM Eastern. We apologize for the inconvenience. We feel that this is going to be the only way to perform the storage maintenance required to get everything back online before tomorrow.
Posted Apr 27, 2023 - 17:35 EDT
Update
We are planning an emergency downtime tonight in order to get things back into working order for tomorrow. Based on what we have learned about the issue impacting us, this is the best, safest and quickest course of action available to us.
Posted Apr 27, 2023 - 14:08 EDT
Update
Performance is still very poor, impacting inbound email, webmail access and POP/IMAP access. Our team understands the root cause and how to fix the issue, but the storage needs to be responsive in order for us to fix the issue which involves moving users off of the storage. There will likely be no noticeable improvements during the day today. So sorry and appreciate your patience with us. This is our #1 priority.
Posted Apr 27, 2023 - 12:35 EDT
Update
Storage congestion is causing slowness with all aspects of the application today. We understand the root cause of the issue and are doing everything possible to keep things operational. We need to complete the maintenance work as described by our storage vendor and that was underway overnight and will continue after the work day today. It generates a lot of IO and we do not want that to further the issues you are experiencing.
Posted Apr 27, 2023 - 09:51 EDT
Identified
We are experiencing performance degradation again today. We understand the root cause and have been actively executing the plan from the datacenter and storage teams. We will keep you posted.