pfSense cluster and 2 switches



  • Hi,

    i have a 2 node pf sense cluster and 2 HP Procurve 2824 Switches.

    The switches are not fully stacked and configured with RSTP. (works greate so far).
    The pfSense Node are a apu4c4 board, the interfaces assigned as follows:
    igb0 -> wan
    igb1-2 - lagg (failover)
    igb3 -> ha sync port

    the pfsense igb1 is connected to switch1 port 1 and igb2 is connected to switch2 port 1,
    so the second node is connected so switch1 port2 and switch2 port2.

    There are several carps configured and sometimes i can't access the web interface of the backup node or very laggy.

    Is that a problem with my pfsense lagg config and the connection to the switches?

    Regards David


  • LAYER 8 Netgate

    Never used those switches but generally to configure a lagg using member ports on two different switches they either need to be stacked or need to implement something usually called Multi-Chassis Trunking or similar.

    Ah - failover mode not LACP... In failover mode it might or might not work. Hmm. You might need to dig a little deeper into what exactly is happening when the connectivity issues are occurring.



  • thanks for your answer,

    hm, i dont know how i can dig deeper.
    Have pfsense a STP functionality for a lagg interface?

    Regards


  • LAYER 8 Netgate

    No. It does not need STP because it will not forward traffic received on one member out another member so it cannot create a loop.

    The switches should never block one of the ports going to pfSense since it should never receive a BPDU from them.

    Does it work fine with one of the failover links disconnected?



  • @godav said in pfSense cluster and 2 switches:

    Procurve 2824 Switches

    Any specific reason you're not stacking them?



  • @NogBadTheBad this model only does a configuration stack, not a full logical stack.



  • @Derelict if i disable one failover port (at switch site), the behavior is a little bit better but still laggy or sometimes there happens nothing in de web ui.



  • the problem is strange, if i open a single browser windows to my backup node, all is working great.
    if i open a second browser window to my master node the backup node is laggy.

    i have two carps on that lagg.
    One carp on the lagg0, 192.168.12.254 <-- web ui access
    and another carp on the lagg0.100, 192.168.11.254 <-- vlan 100



  • i have two new cisco sg500 in a logical stack and connected the two firewalls with a lacp lagg. But my problem still be there. I cant figure it out where the problem is. Sometimes the gui appears and sometime the gui is loading and loading and nothing happens.


  • LAYER 8 Netgate

    What is a "logical stack" in this case? Can you LACP to both switches on one LAGG there? You can usually only do that with a physical stack or something like Multi-Chassis Trunking (MCT).

    How does it perform if you disconnect one of the LACP member links?



  • It's a physical stack, i have the pfsense-master lagg0 connected to switch port 1/1/1 and 1/1/2, the pfsense-slave is connected to 2/1/1 and 2/1/2. So yes one LAGG per firewall to both switches.

    If i disconnect a lacp member link the issue is still the same.

    Regards David


  • LAYER 8 Netgate

    You should be connecting to 1/1/1 and 2/1/1 to the primary and 1/1/2 and 2/1/2 to the secondary so a switching failure does not blow up the routing cluster.

    Really hard to say what you are seeing. You might have to pcap to see who is not responding to whom.



  • @Derelict Is cabled as you say, mentiont it false. :)

    i've done a pcap when i can't connect to the gui:

    07:25:33.069437 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 13064, offset 0, flags [DF], proto TCP (6), length 82)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0x943f (correct), seq 1918627193:1918627235, ack 3166548767, win 1026, length 42
    07:25:33.075572 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 127, id 13065, offset 0, flags [DF], proto TCP (6), length 110)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0xdfba (correct), seq 42:112, ack 1, win 1026, length 70
    07:25:36.764479 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 13066, offset 0, flags [DF], proto TCP (6), length 82)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0xa55a (correct), seq 112:154, ack 1, win 1026, length 42
    07:25:36.773192 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 127, id 13067, offset 0, flags [DF], proto TCP (6), length 110)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0x7e1c (correct), seq 154:224, ack 1, win 1026, length 70
    07:25:43.113812 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 13068, offset 0, flags [DF], proto TCP (6), length 82)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0xcfe5 (correct), seq 224:266, ack 1, win 1026, length 42
    07:25:43.124429 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 127, id 13069, offset 0, flags [DF], proto TCP (6), length 110)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0xfd1b (correct), seq 266:336, ack 1, win 1026, length 70
    07:25:46.490591 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 13070, offset 0, flags [DF], proto TCP (6), length 82)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0xc0f6 (correct), seq 336:378, ack 1, win 1026, length 42
    07:25:46.502714 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 127, id 13071, offset 0, flags [DF], proto TCP (6), length 110)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0x19c6 (correct), seq 378:448, ack 1, win 1026, length 70
    07:25:46.889703 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 13072, offset 0, flags [DF], proto TCP (6), length 82)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0x2177 (correct), seq 448:490, ack 1, win 1026, length 42
    07:25:46.901803 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 124: (tos 0x0, ttl 127, id 13073, offset 0, flags [DF], proto TCP (6), length 110)
        192.168.11.149.60257 > 192.168.12.2.10443: Flags [P.], cksum 0x071f (correct), seq 490:560, ack 1, win 1026, length 70
    07:25:50.268866 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 127, id 13074, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [S], cksum 0x69f0 (correct), seq 3722673935, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    07:25:50.270682 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13075, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x155d (correct), seq 3722673936, ack 695034584, win 1026, length 0
    07:25:50.271608 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 571: (tos 0x0, ttl 127, id 13076, offset 0, flags [DF], proto TCP (6), length 557)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x9f78 (correct), seq 0:517, ack 1, win 1026, length 517
    07:25:50.282487 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13077, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x0d09 (correct), seq 517, ack 1616, win 1026, length 0
    07:25:50.284896 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 180: (tos 0x0, ttl 127, id 13078, offset 0, flags [DF], proto TCP (6), length 166)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x6be7 (correct), seq 517:643, ack 1616, win 1026, length 126
    07:25:50.285757 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 231: (tos 0x0, ttl 127, id 13079, offset 0, flags [DF], proto TCP (6), length 217)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x9029 (correct), seq 643:820, ack 1616, win 1026, length 177
    07:25:50.285830 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 317: (tos 0x0, ttl 127, id 13080, offset 0, flags [DF], proto TCP (6), length 303)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x5555 (correct), seq 820:1083, ack 1616, win 1026, length 263
    07:25:50.288860 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13081, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x0a52 (correct), seq 1083, ack 1745, win 1026, length 0
    07:25:50.288896 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 92: (tos 0x0, ttl 127, id 13082, offset 0, flags [DF], proto TCP (6), length 78)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x6f7a (correct), seq 1083:1121, ack 1745, win 1026, length 38
    07:25:50.324940 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13083, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xfaf8 (correct), seq 1121, ack 5636, win 1026, length 0
    07:25:50.351593 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 219: (tos 0x0, ttl 127, id 13084, offset 0, flags [DF], proto TCP (6), length 205)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xa79c (correct), seq 1121:1286, ack 5636, win 1026, length 165
    07:25:50.351846 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 143: (tos 0x0, ttl 127, id 13085, offset 0, flags [DF], proto TCP (6), length 129)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x9be4 (correct), seq 1286:1375, ack 5636, win 1026, length 89
    07:25:50.351890 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 127, id 13086, offset 0, flags [DF], proto TCP (6), length 128)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xf9d3 (correct), seq 1375:1463, ack 5636, win 1026, length 88
    07:25:50.352155 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 159: (tos 0x0, ttl 127, id 13087, offset 0, flags [DF], proto TCP (6), length 145)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x5eaa (correct), seq 1463:1568, ack 5636, win 1026, length 105
    07:25:50.352757 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 159: (tos 0x0, ttl 127, id 13088, offset 0, flags [DF], proto TCP (6), length 145)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xc624 (correct), seq 1568:1673, ack 5636, win 1026, length 105
    07:25:50.352828 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 127, id 13089, offset 0, flags [DF], proto TCP (6), length 130)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xb4fc (correct), seq 1673:1763, ack 5636, win 1026, length 90
    07:25:50.358834 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13090, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xd63e (correct), seq 1763, ack 14396, win 1026, length 0
    07:25:50.359477 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13091, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xbf6e (correct), seq 1763, ack 20236, win 1026, length 0
    07:25:50.361409 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13092, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x9d36 (correct), seq 1763, ack 28996, win 1026, length 0
    07:25:50.361925 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13093, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x8666 (correct), seq 1763, ack 34836, win 1026, length 0
    07:25:50.364071 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13094, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x642e (correct), seq 1763, ack 43596, win 1026, length 0
    07:25:50.364418 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13095, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x47aa (correct), seq 1763, ack 50896, win 1026, length 0
    07:25:50.366374 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13096, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x368e (correct), seq 1763, ack 55276, win 1026, length 0
    07:25:50.366848 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13097, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x08ee (correct), seq 1763, ack 66956, win 1026, length 0
    07:25:50.367715 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 172: (tos 0x0, ttl 127, id 13098, offset 0, flags [DF], proto TCP (6), length 158)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xb149 (correct), seq 1763:1881, ack 66956, win 1026, length 118
    07:25:50.368467 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13099, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xe63f (correct), seq 1881, ack 75716, win 1026, length 0
    07:25:50.370736 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13100, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xc9bb (correct), seq 1881, ack 83016, win 1026, length 0
    07:25:50.370924 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13101, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0xa1cf (correct), seq 1881, ack 93236, win 1026, length 0
    07:25:50.373197 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13102, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x5d5f (correct), seq 1881, ack 110756, win 1026, length 0
    07:25:50.376322 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13103, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x18ef (correct), seq 1881, ack 128276, win 1026, length 0
    07:25:50.418359 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 127, id 13104, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [.], cksum 0x133b (correct), seq 1881, ack 129736, win 1026, length 0
    07:25:53.417041 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 127, id 13105, offset 0, flags [DF], proto TCP (6), length 83)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x5085 (correct), seq 1881:1924, ack 129736, win 1026, length 43
    07:25:53.417277 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 127, id 13106, offset 0, flags [DF], proto TCP (6), length 83)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xcf05 (correct), seq 1924:1967, ack 129736, win 1026, length 43
    07:25:53.417315 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 127, id 13107, offset 0, flags [DF], proto TCP (6), length 83)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0x12b3 (correct), seq 1967:2010, ack 129736, win 1026, length 43
    07:25:53.417344 00:0d:b9:50:61:e5 > 00:0d:b9:50:61:c9, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 127, id 13108, offset 0, flags [DF], proto TCP (6), length 83)
        192.168.11.149.60296 > 192.168.12.2.10443: Flags [P.], cksum 0xdfb8 (correct), seq 2010:2053, ack 129736, win 1026, length 43
    
    


  • @Derelict i think i found a solution. Disabled hardware checksum offload and all is running smooth now!


  • LAYER 8 Netgate

    That is an odd thing to have to do using physical nodes but glad you found it.


Log in to reply