M365 backup fails with ‘ETHROTTLE’ or Throttling error
Problem description
M365 backup fails showing error ‘ETHROTTLE ’ or ‘Throttling error’ on Druva management console.
Cause
Throttling limits the number of concurrent calls to a service to prevent overuse of resources. Microsoft Graph is designed to handle a high volume of requests. If an overwhelming number of requests occurs, throttling helps maintain optimal performance and reliability of the Microsoft Graph service.
When a throttling threshold is exceeded, Microsoft Graph limits any further requests from that client for a period of time.
When throttling occurs, Microsoft Graph returns HTTP status code 429 (Too many requests), and the requests fail.
A suggested wait time is returned in the response header of the failed request. Throttling behaviour can depend on the type and number of requests.
The most common causes of throttling of clients include:
-
A large number of requests across all applications in a tenant.
-
A large number of requests from a particular application across all tenants
This commonly occurs if you have configured M365 newly and are running backups for the first time.
Traceback
The failures will be intermittent.
+snip+
2022-11-12 19:22:47,133] [INFO] Microsoft 365 group backup finished successfully. For group: JXXX
[2022-11-13 19:22:48,336] [DEBUG]HttpError 429 when requesting https://graph.microsoft.com/v1.0/gro...644/sites/root for user returned 'Response :{'error': {'code': 'activityLimitReached', 'message': 'The request has been throttled', 'innerError': {'code': 'throttledRequest', 'innerError': {'code': 'quota'}, 'date': '2022-11-13T19:22:48', 'request-id': '1a0257b9-02b5-454c-9b2e-23d260fd84c2', 'client-request-id': '1a0257b9-02b5-454c-9b2e-23d260fd84c2'}}}'Retrying 2 after sleeping 20 secs.
+snip+
+snip+
[2022-11-15 09:52:59,785] [DEBUG] API HttpError 500 when requesting https://graph.microsoft.com/v1.0/gro...bscribedByMail for user returned 'Response :{'error': {'code': 'UnknownError', 'message': '', 'innerError':
[2022-11-16 07:23:16,685] [INFO] Microsoft 365 group backup finished successfully. For group: XXXX
[2022-11-10 09:53:59,618] [DEBUG] API HttpError 429 when requesting https://graph.microsoft.com/v1.0/gro...e59/sites/root for user returned 'Response :{'error': {'code': 'activityLimitReached', 'message': 'The request has been throttled', 'innerError':
+snip+
Resolution
The following are best practices for handling throttling
-
Increase the frequency of backups. For example if the frequency is set to once every day then change it to “Twice a day”.
If throttling is still observed during the backups, sub- app can be configured depending on throttled M365 data source.
To configure subapps:
-
Click SaaS Apps > Microsoft 365.
-
On the Overview page, click Re-Configure.
-
You will see all configured and non configured sub-apps are listed.
-
On the Re-Configure for Backup page, click on “Configure” button in front of sub-application.
-
On the Microsoft 365 login page, enter the Microsoft 365 global administrator's username and password, and then click Sign in.
-
Click Accept to grant the inSync app the required permissions to access Microsoft 365 data.
-
The permissions displayed are a combined list for all Microsoft 365 apps. If you want to know more about specific permissions applicable for each app, see Microsoft 365 Permissions for Druva App.
inSync gets connected to Microsoft 365 data of all users in your organisation.
Verification
After increasing the backup frequency, Number of requests would be reduced as changes would be backed up frequently. Backups will complete successfully.
Monitor the backups for a couple of days, If you still observe throttling please contact Druva support for deploying Auxiliary app.