Final status: We've completed the restarts to address the residual impact, and after an extended period of monitoring, we've confirmed that the issue is resolved. We continue to investigate why the update caused impact and further details will be provided within the Post-Incident Report.
Scope of impact: Impact was primarily to users hosted within infrastructure that received the update prior to the deployment halt. However, any user may have experienced some degree of impact as described in the "More Info" section above.
Start time: Monday, June 5, 2023, 10:10 AM EDT
End time: Monday, June 5, 2023, 1:13 PM EDT
Preliminary root cause: A recent update, intended to add additional resilliencies against downstream request remaining in a hanging state, inadvertently caused impact.
Next steps: - We're reviewing dump files gathered from impacted components during the incident to better understand why impact occurred. - We're investigating to content of the update to identify the underlying root cause.
We’ll publish a Post-Incident report within five business days.
Posted Jun 05, 2023 - 15:06 EDT
Update
We are continuing to investigate this issue.
Posted Jun 05, 2023 - 12:03 EDT
Investigating
What we know from Microsoft: Title: Some users are experiencing impact to multiple Microsoft 365 services and features
User impact: Users are experiencing impact to multiple Microsoft 365 services and features.
More info: Impact services and features include, but are not limited to: Exchange Online - Users are unable to access Outlook on the web (as communicated under EX571516)
Microsoft Teams - Users may have difficulties scheduling meetings and/or live events - Users may have trouble loading people profile cards - Users may have issues loading file lists - Users may not be able to create new teams & channels - Users may not be able to install apps - Users may have trouble issuing searches - Users may see delays in admin policy changes taking effect - Users might see errors when using messaging extensions. - Teams Graph APIs may be impacted
SharePoint Online and OneDrive for Business - Users may be unable to use Search functionality
Current status: We're reviewing monitoring health systems targeted at networking, Client Access Front End (CAFE) routing, and database storage to determine the underlying cause of the issue.
Posted Jun 05, 2023 - 12:00 EDT
This incident affected: 3rd Party Solutions (Office365 Services).