We are closing this incident after running at full performance for the day. We believe we are on track to fully resolve and prevent this from occurring going forward. We will send an email update with more information about the issue and its resolution and prevention measures we are putting into place.
Posted Apr 26, 2023 - 14:27 EDT
Update
The first wave of corrective actions have taken place and all servers and services are back online. All inbound mail queues have processed and our general performance stats are back to normal. Now that the root cause is understood, we will work closely with our datacenter and storage teams to put monitoring in place that should alert us before it occurs in the future.
Posted Apr 26, 2023 - 08:30 EDT
Monitoring
As per the recommendation of our storage vendor and datacenter teams, we executed a plan which seems to have had positive impacts on performance overall. We still have some work to do, but we remain cautiously optimistic.
Posted Apr 26, 2023 - 04:25 EDT
Update
The storage subsystem has been confirmed to be the issue with performance. We are executing a plan based on the storage vendor's recommendations. Providing your service and keeping everything running is our number one priority. Please know that we are working to get this completed as soon as possible in order to get everything back to normal operation. Thanks for your continued understanding.
Posted Apr 25, 2023 - 23:45 EDT
Update
Our performance continues to be degraded. Our datacenter has identified that the storage subsystem is doing a low-level filesystem check. During that filesystem check, all other operations operate degraded. They are analyzing with the storage vendor to understand why the check is impacting performance so greatly. This is a normal operation that runs ever other week and isn't new to our system. Mail is delivering very slowly to your inboxes and will continue to queue up during this degradation. At present, we do not have an ETA, but hope to have things operational through the night.
Posted Apr 25, 2023 - 20:10 EDT
Update
We have updated the status to critical since performance remains severely degraded and we have two mailstores that are performing filesystem repairs. We will continue to keep you updated here.
Posted Apr 25, 2023 - 18:14 EDT
Update
Hello Everyone. Our datacenter and storage teams continue to analyze the situation. Given the current situation, we may need to do an emergency maintenance tonight on the storage subsystem. Please note that email is being queued up on our servers and is tricking through. The queues will be delivered once the issue resolves.
Posted Apr 25, 2023 - 16:18 EDT
Update
Our performance degradation continues. Our storage vendor continues to analyze and we are monitoring the situation. Everything is up and running, but running degraded. Thanks for your patience while we work through this.
Posted Apr 25, 2023 - 14:18 EDT
Update
Thanks for your patience. Our storage engineers are analyzing. All services are up, albeit running degraded.
Posted Apr 25, 2023 - 11:48 EDT
Update
The degradation continues. We are prepping for a brief outage to reboot our storage. We believe that the short outage will be far better than an extended degradation.
Posted Apr 25, 2023 - 10:18 EDT
Identified
BlueTie's email service is experiencing a performance degradation due to the storage subsystem. We are actively monitoring and may need to engage in our storage vendor to resolve. All services are operational, but performance degraded.