Strange Packet loss - PFsense 2.0

  • Hi all, experiencing some weird issues when trying to setup a new set of routers.

    We have single wan coming into two pfsense boxes, with CARP between them on its own interface, and shared Public IP along with a public IP assigned to each box, and on the lan side a shared LAN IP, and then a lan IP assigned to each box, nothing too "complicated" as far as I'm aware.

    If I login to the router directly, I can ping any site with no loss -, slashdot, my own home servers. However if I connect from a system on the lan side I'm getting anything between 15 - 60% packet loss. However pinging the router directly with no issues.

    tim.bowers@TimPad:~$ ping
    PING ( 56(84) bytes of data.
    64 bytes from icmp_req=1 ttl=49 time=22.3 ms
    64 bytes from icmp_req=2 ttl=49 time=17.3 ms
    64 bytes from icmp_req=3 ttl=49 time=17.7 ms
    64 bytes from icmp_req=5 ttl=49 time=17.2 ms
    64 bytes from icmp_req=7 ttl=49 time=17.4 ms
    64 bytes from icmp_req=8 ttl=49 time=23.9 ms
    64 bytes from icmp_req=9 ttl=49 time=17.3 ms
    64 bytes from icmp_req=10 ttl=49 time=17.3 ms
    ^C64 bytes from icmp_req=11 ttl=49 time=20.9 ms
    --- ping statistics ---
    11 packets transmitted, 9 received, 18% packet loss, time 42255ms
    rtt min/avg/max/mdev = 17.291/19.076/23.943/2.454 ms
    tim.bowers@TimPad:~$ ping
    PING ( 56(84) bytes of data.
    64 bytes from icmp_req=1 ttl=64 time=0.435 ms
    64 bytes from icmp_req=2 ttl=64 time=0.422 ms
    64 bytes from icmp_req=3 ttl=64 time=0.279 ms
    64 bytes from icmp_req=4 ttl=64 time=0.256 ms
    64 bytes from icmp_req=5 ttl=64 time=0.250 ms
    64 bytes from icmp_req=6 ttl=64 time=0.321 ms
    64 bytes from icmp_req=7 ttl=64 time=0.277 ms
    64 bytes from icmp_req=8 ttl=64 time=0.233 ms
    64 bytes from icmp_req=9 ttl=64 time=0.285 ms
    64 bytes from icmp_req=10 ttl=64 time=0.287 ms
    64 bytes from icmp_req=11 ttl=64 time=0.286 ms
    64 bytes from icmp_req=12 ttl=64 time=0.260 ms
    64 bytes from icmp_req=13 ttl=64 time=0.378 ms
    --- ping statistics ---
    13 packets transmitted, 13 received, 0% packet loss, time 11998ms
    rtt min/avg/max/mdev = 0.233/0.305/0.435/0.064 ms
    tim.bowers@TimPad:~$ route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface   U     1      0        0 eth0     U     1000   0        0 eth0         UG    0      0        0 eth0

    I've disabled all services other than NTPsync and the DNS forwarder.

    I'm kind of at a complete loss. The network its self is a bit more complicated - As this is a new set of routers we are doing, there is also an existing pfsense box, which everyone is using currently as their gateway (so I can't turn this off/remove it). However I don't see why this would have any effect on the new systems.

    Your help is very much appreciated :)

  • Kind of disappointed theres been no responses…. is that normal?

    Anyway, the show goes on. I rebuilt the boxes due to finding one had broken its heatsink off the raid card.

    I have have fully CARP setup which falls over beatifully when using Automatic NAT - however then setting to Manual nat and I'm once again seeing the loss I was seeing before.

    I have the rules which match other posts - translating on the wan to update the IP with the gateway IP yet this doesn't seem to make any difference. I am yet to check what the IP that is seen from outside is, but I believe it'll be the correct IP (i.e. the one of the virtual CARP ip)

  • Do you have any traffic shapers enabled on either the LAN or WAN? If so, try putting ICMP packets in your ACK queue (which should have the highest priority), and try again.

    Another thing that can cause it (which was in our case) We had the pfsense sitting between the LAN and a Cisco router which connected to our ISP. The cisco router actually caused us to have an average 15% packet loss. Removed the Cisco router from the equation, and now we average around .1% during heavy traffic.

  • Here is a comparison during the actual transition from using the Cisco router on the edge, to using only the pfsense box now.

    Gateway that was used for quality was the internal IP of the Cisco router (low latency, but horrible packet loss)

    Notice that Monday through Friday the packet loss was bad, but the latency was very low.

    Gateway now used for quality is the ISP gateway.

    Sunday you can see a small break, that was the transition time from removing the cisco router. The spike in latency on Sunday was from downloading 4 1080p HD movies from youtube at the same time. Notice however there was no to minimal packet loss during those downloads.

  • Pardon the question but,

    A single WAN –> two different pfsense boxes interconnected with CARP -->  2 LAN's ?
    Where is the redundancy in this configuration?

    Why not use:

    WAN --> pfsense --> larger switch -->  single LAN

    As a home user, I never had packet loss because of pfsense not working properly, only experienced packet loss because of faulty ISP hardware.

    Last thursday , my ISP updated the software my cable modem, but forgot to reset it afterwards.

  • Rebel Alliance Developer Netgate

    That kind of loss could also be explained by CARP flapping up/down back and forth between the master and slave boxes. I would especially suspect a CARP issue (probably at layer 2, meaning your switch).

    Check the system log, and the carp status on both.

  • Was a network card issue as far as I can tell.

    With new boxes - no loss, however I haven't tried with the onboard broadcom cards (which was what I was partly using before).

    The reason for the carp is if either of the routers falls over - plus it means we can upgrade one and have the other running happily.

    We do have a 2nd ISP, but no IP range with them…. it was never setup correctly in the past and I'm doubtful of it happening now - too much chance of knocking everything offline by accident.

Log in to reply