General performance degradation
We are closing this incident after running at full performance for the day. We believe we are on track to fully resolve and prevent this from occurring going forward. We will send an email update with more information about the issue and its resolution and prevention measures we are putting into place.
The first wave of corrective actions have taken place and all servers and services are back online. All inbound mail queues have processed and our general performance stats are back to normal. Now that the root cause is understood, we will work closely with our datacenter and storage teams to put monitoring in place that should alert us before it occurs in the future.
As per the recommendation of our storage vendor and datacenter teams, we executed a plan which seems to have had positive impacts on performance overall. We still have some work to do, but we remain cautiously optimistic.
The storage subsystem has been confirmed to be the issue with performance. We are executing a plan based on the storage vendor's recommendations. Providing your service and keeping everything running is our number one priority. Please know that we are working to get this completed as soon as possible in order to get everything back to normal operation. Thanks for your continued understanding.
Our performance continues to be degraded. Our datacenter has identified that the storage subsystem is doing a low-level filesystem check. During that filesystem check, all other operations operate degraded. They are analyzing with the storage vendor to understand why the check is impacting performance so greatly. This is a normal operation that runs ever other week and isn't new to our system. Mail is delivering very slowly to your inboxes and will continue to queue up during this degradation. At present, we do not have an ETA, but hope to have things operational through the night.
We have updated the status to critical since performance remains severely degraded and we have two mailstores that are performing filesystem repairs. We will continue to keep you updated here.
Hello Everyone. Our datacenter and storage teams continue to analyze the situation. Given the current situation, we may need to do an emergency maintenance tonight on the storage subsystem. Please note that email is being queued up on our servers and is tricking through.
The queues will be delivered once the issue resolves.
Our performance degradation continues. Our storage vendor continues to analyze and we are monitoring the situation. Everything is up and running, but running degraded. Thanks for your patience while we work through this.
Thanks for your patience. Our storage engineers are analyzing. All services are up, albeit running degraded.
The degradation continues. We are prepping for a brief outage to reboot our storage. We believe that the short outage will be far better than an extended degradation.
BlueTie's email service is experiencing a performance degradation due to the storage subsystem. We are actively monitoring and may need to engage in our storage vendor to resolve. All services are operational, but performance degraded.
This incident affected: BlueTie Proprietary Solutions (Webmail, POP-IMAP-SMTP, Outbound Email Deliverability, Inbound Email Delivery).