Compaction fails showing key error with old IP address
This article applies to:
OS: Windows and Linux
Product edition: inSync On-Premise
Compaction fails due to following error where it shows Key error with old IP address
Internal Document (Only for Druva Support Engineers)
Backup fails with traceback (inSyncCloudServer.log) as follows:-
[2016-10-07 00:30:57,808] [INFO] Storage compact of storage=9 to node=ABC.xxx.com
[2016-10-07 00:30:57,885] [INFO] Compacting device(19 out of 57) did=54, uid=12 of storage=9
[2016-10-07 00:30:57,901] [INFO] Done compaction [Daily] for storage(id:9 name:iSCSI Storage 2016-1) customer_id:0
[2016-10-07 00:30:57,901] [INFO] Done compaction for storage iSCSI Storage 2016-1.
[2016-10-07 00:30:57,901] [ERROR] Compaction: doStorageCompact has been failed for storage:9. Error:'192.168.232.2'.
[2016-10-07 00:30:57,901] [ERROR] Error <type 'exceptions.KeyError'>:'192.x.x.x'. Traceback -Traceback (most recent call last):
File "inSyncSrv\inSyncCompact.pyc", line 234, in __storageCompact
File "inSyncSrv\inSyncCompact.pyc", line 358, in __doStorageCompact
File "inSyncSrv\inSyncCompact.pyc", line 481, in __compactDevices
File "inSyncSrv\inSyncCompact.pyc", line 889, in process
File "inSyncSrv\inSyncCompact.pyc", line 473, in __compactDevice
File "Lib\CloudMasterTasks.pyc", line 571, in device_compact
File "Lib\CloudMasterTasks.pyc", line 2494, in _device_compact
File "Lib\CloudMasterTasks.pyc", line 2221, in _get_nodeconn
Note: IP shown in the above traceback would defer as per your environment
References of old IP address are present in the inSync Configuration.
- Stop the inSync Services (All 3 Services).
- Make sure no processes are running in Task Manager.
- Check whether the old IP address is present in inSyncCloudServer.cfg. If yes, take a backup of the old .cfg file and then replace the old IP address with new IP (See Embedded mySQL server with inSyncServer for reference).
- Take a Backup of MySQL DB.
- Open command prompt with Administrator rights.
- Connect to MySQL using the command line and verify if there are old IP addresses in Zone, edge_ip and description. If yes, replace the old IP with new IP address.
- Check the task table for old IP address entries and replace them with new IP address. Use the following steps to replace the entries:
- Connect to MySQL and use the following command.
select id, type, sid, uid, status, worker from task;
If you know the old IP address, you can also write the query in the following manner.
select id, type, sid, status from task where worker="IP Address";
- To delete these stale tasks, you can run the following command.
delete from task where worker="IP Address";
- Start the inSync Services and wait for the storage to come up healthy.
- Run the Manual Compaction and verify whether the compaction completes successfully.