We are continuing to monitor for any further issues.
Posted Apr 30, 2024 - 20:07 UTC
Update
We continue to see M365 Exchange backup success rates improve across the fleet and continue to monitor the backlog to ensure re-ingest of M365 Exchange services complete.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
M365 Exchange backup success rates continue to improve since Microsoft completed the process to pause the reindexing of M365 Exchange mailboxes on Saturday, April 20th.
Our team continues to monitor M365 Exchange backup success rates as we continue to re-ingest this backlog over the coming days.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
Please note : Limit increases are temporary and will need to be applied again as needed.
Posted Apr 24, 2024 - 18:23 UTC
Update
We continue to see an improvement in the M365 Exchange backup success rates since Microsoft completed the process to pause reindexing Exchange mailboxes on Saturday, April 20th.
As this backlog is re-ingested over the coming days, the backup success rates will continue to improve.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
Please note : Limit increases are temporary and will need to be applied again as needed.
Posted Apr 23, 2024 - 19:25 UTC
Update
On Saturday, April 20th, Microsoft completed the process to pause reindexing Exchange mailboxes – the root cause of this issue - for all Datto SaaS Protection and Backupify customers.
Since the reindexing was paused, we have seen an improvement in the Exchange backup success rate, as the backlog of re-indexed data is re-ingested. As this backlog is re-ingested over the coming days, the backup success rates will continue to improve.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
Please note : Limit increases are temporary and will need to be applied again as needed.
Posted Apr 22, 2024 - 21:47 UTC
Update
Microsoft has agreed to pause the re-indexing process for all impacted Datto SaaS Protection and Backupify domains. As the SaaS Protection and Backupify products work through the backlog of data that has already been re-indexed, the backup success rates will improve.
The Datto R&D team is continuing to work on mitigations to process through this backlog at a faster rate.
Some domains may still be throttled while the system works through the re-index backlog. The mitigation below can be applied to any domains that are still seeing Exchange backup failures as those domains work through the backlog.
Please note : Limit increases are temporary and will need to be applied again as needed.
Posted Apr 19, 2024 - 19:18 UTC
Update
Microsoft has confirmed that “significant changes” to their Exchange APIs involved re-indexing certain folders, causing all items in those folders to be re-synchronized and re-ingested by Datto SaaS Protection and Backupify.
Microsoft has stated that this issue is impacting multiple backup vendors.
This issue has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests, which caused a corresponding reduction in backup success rates, particularly for Microsoft Exchange mailboxes.
The queue of data that must be re-ingested is decreasing at a linear rate and customers will see their Exchange backup success rate improve as their re-ingest completes.
We are working directly with Microsoft’s Messaging Product team on a 24x7 basis to reduce the impact of this issue and it has been escalated to a ‘Sev A’ – Microsoft’s top level.
We are also rolling out additional infrastructure to assist in processing the additional data being sent from Microsoft.
Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
Please note : Limit increases are temporary and will need to be applied again as needed.
This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Thursday, April 18 and is expected to complete by April 24th.
Posted Apr 18, 2024 - 15:44 UTC
Update
Microsoft has confirmed that a change to the Microsoft API has caused data to re-ingest for a sub-set of folders within certain users’ Exchange mailboxes and is impacting multiple backup vendors.
This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests, particularly for Microsoft Exchange mailboxes.
As of Tuesday, April 16th, this issue was escalated within Microsoft to a SEV A and the Datto R&D team is working directly with Microsoft engineers on a 24x7 basis to bring the issue to resolution.
In parallel, additional API throttling mitigation changes were deployed on April 16th and backup success rates for both Microsoft Teams and SharePoint backups have now returned to normal levels.
Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.
Please note : Limit increases are temporary and will need to be applied again as needed.
This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Thursday, April 18 and is expected to complete by April 24th.
Posted Apr 17, 2024 - 16:53 UTC
Update
An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.
This issue has been escalated within Microsoft to a SEV A and the Datto R&D team is working directly with Microsoft engineers on a 24x7 basis to address the issue.
The platform continues to work through the queue of re-ingested data. We anticipate that the API throttling will decrease as the queue of re-ingested data is processed.
In parallel, additional API throttling mitigation changes were deployed on April 16th to improve backup success rates. This change is showing significant improvement in both Microsoft Teams and SharePoint backups, which are expected to return to normal by end of day April 16th.
Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.
Please note : Limit increases are temporary and will need to be applied again as needed.
This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Wednesday, April 17 and is expected to complete by April 24th.
Posted Apr 16, 2024 - 17:26 UTC
Update
An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.
Over the weekend, the platform continued to work through the queue of re-ingested data. We anticipate that the API throttling will decrease as the queue of re-ingested data is processed.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
Please note : Limit increases are temporary and will need to be applied again as needed.
This issue is also impacting some V2 users where backups will start and not complete. A fix for the incomplete backups issue on V2 has been identified and is being rolled out beginning Monday, April 15 and the rollout is expected to complete by mid-week.
Posted Apr 15, 2024 - 14:21 UTC
Identified
An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.
The impact is a reduction in backup success rates for the users experiencing this issue.
We are actively working with Microsoft to resolve this problem.
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
We are currently aware of a problem where a subset of Datto SaaS Protection users can experience a M365 Exchange failure due to error "ExchangeServerBusyError".
Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :