Bump and some additional info:
With that line removed, we initially thought we had gotten rid of the problem, but it looks like we only added delay. After about 2 weeks, the IPSec tunnels entered the same weird state where they are up and traffic going out can be seen, but dpinger (or a manual ping) seems to never reach the remote end. Hard to tell if the exact timing is reproducible, but it will happen again eventually after a few days/weeks. (we've seen it happen twice since i initially posted here).
Rebooting seems to be the only way to fix this. Restarting ipsec service didn't do anything, neither did disconnecting the tunnels via webinterface and reconnecting.
Our resort for now was to move the AWS Site-to-Site tunnels off to an EdgeRouter, as we only started to encounter the issues once they were added. Worker fine before with only pfsense<->pfsense tunnels. I'll continue to monitor the situation, and am waiting for the 2.4.5 release to become available. Hopefully the various bugfixes will in the end resolve the problem.
I'll also soon have the opportunity to make another, almost identical setup using two MB-4220T pfsense appliances (https://store.netgate.com/MBT-4220-system.aspx) instead of SG-3100s.