I found a solution - restart :). It seems that the advskew wasn't picked up correctly. I set it to 0 but it reverted to 100 on both nodes which went unnoticed. After making sure it is at 0 for the primary and removing and readding peer IP it recovered.