Both PRIMARY and BACKUP DS are in a synchronizing standby state

Overview

Both PRIMARY and BACKUP are in a synchronizing standby state.
Management container shutdown because of the following severe error:

(severe) Both PRIMARY and BACKUP are in a synchronizing standby state

This signifies a Synchronization issue between Primary and Backup Directory Service (DS).
Both Primary and Backup management containers show the following message and wait forever:

(info) Using Deep Synchronization to replicate the Directory Service 
Both Primary and Backup DS are in Synchronizing Standby state (SYNCHRONIZING_STANDBY)

Solution

This problem occurs when both Primary and Backup domain DS folders were renamed/removed.

If both Primary and Backup domain Directory Service (DS) folders do not exist (because they have been deleted/removed/renamed) then the Sonic management framework will be unable to access any domain configuration (as expected). The DS host location is the value set to "HOST_DIRECTORY" attribute in the DS's bootfile file (ds.xml by default).

When the primary management container (hosting the Primary DS) starts up and cannot find its local DS (as pointed out by its ds.xml), or if its local DS is uninitialized (happens when Sonic framework creates a default empty and uninitialized domain when it can't find one), it will assume that the backup management container must be active then; and so it will start transitioning into Standby state and attempt rebuilding the domain from the backup domain by using "Deep Synchronization" as it needs to reconstruct the domain by replicating (copying data) the whole DS configuration from its peer Backup.

The same scenario occurs when the backup management container (hosting the Backup DS) starts up and cannot find its local DS, or if its local DS is uninitialized, it will assume that the primary management container must be active then; and so it will start transitioning into Standby state and attempt rebuilding the domain from the primary domain by using "Deep Synchronization" as it needs to reconstruct the domain by replicating the whole DS configuration from its peer Primary.

Thus both Primary and Backup DSs end up in a 'synchronizing standby' state (see note below) waiting to rebuild and synchronize their domain from each other (this is why they both log "Using Deep Synchronization to replicate the Directory Service" message). Which of course will never happen as neither the primary nor the backup has an updated active DS to start with. And this is when the following error is logged and consequently the management container (where this error is thrown) is immediately shutdown:

(severe) Both PRIMARY and BACKUP are in a synchronizing standby state

In an FT setup of the DS, at least one DS domain needs to be accessible and active (either on the primary or the backup side). Normally removing a DS (domain name folder) is not needed except if it needs rebuilding from its peer (in this case the peer DS needs to exist). When both primary and backup DS connect to each other they would know how to synchronize with each other using either basic or deep synchronization.