High traffic on pfsync interface

  • Hi,
    we have a working PFSense 2.1.5 Setup based on two machines with AMD64 architecture. Carp failover was tested and is running ok.
    Hardware is 2 NICs em0/1 and 2NICs Intel Quad Port (= 10 Ports) per box.
    Synchronization is done via a crosslink over igb7 (last port of the 2nd Intel Quad Port).
    We have setup state transfer and XMLSync, which works correctly.
    The role of the firewall is an internal gateway which routes between different network zones (production/DMZ, admin, database).
    Internet access in front of the DMZ is done by another pair of PFSense, same release.
    We need to separate the databases services from the DMZ (where they were located before) into a separate segment, but we have high traffic
    from the DMZ to the database machines. In the DMZ, our application Servers are located, setting up database connections to the
    DB servers using DB connection pools.
    We have about 2000 (!) constantly existing connections from the app servers to the DB systems which are relatively static.
    Now, we have seen that the traffic on the DB net is about 80Mbps, which is OK for a GBit link.
    But - on the CROSS Interface at the same time we encountered 180 Mbps traffic, just from the pfsync protocol !
    That leads to my question: why is the state transfer (I assume this is mostly state transfer messages) taking more load
    than the actual data transfer ?
    I can only assume that the pfsync always transfer ALL states and not the modified states, cause when looking
    at the state table, the sockets listed there are not changed very often.
    This would lead to an enormous mount of states which are transferred, because of the >2000 connections, which are mostly unchanged.
    The other option would be that the state includes the received/written package counters, which would trigger a state change for each packet forwarded.
    This obviously would trigger a pfsync packet for each IP packet received/forwarded.
    We have to move other servers into the DB zone and we expect overload to the cross interface, if the load increases.

    On the Internet firewall, we have about 30Mbps Traffic, while the corresponding cross interface has about 5-6Mbps traffic. (here are
    steadily changing states for connections with short lifetime, http/s protocol mostly, but a huge number).

    Any thoughts on this ? I cannot see any misconfiguration and the load was not that high when the setup was initially done.

    Best regards, Marcus Haarmann