Pfsense strange issue Flooding the network when Multi-Wan

Electrone

i faced strange issue when i install pfsense with 2 GW and 1 LAN , to able to give LB between the 2 GW .

sometimes when i start to edit the FW rules for any of the GWs interfaces i face huge traffic going to the SW from the Pfsense server and it flood the network . and i cant ping or do any analyze with any tool over the network , and in seconds the SW get fully flooded . i can see that later with prtg ( i see that all interfaces on the SW spike to the max and never get down till i remove pfsesne server )

I installed latest version of pfsense on vmware esxi 5.5 u1

sometimes this issue happen when installing squid with the same above scenario. and i cant fix it unless i reinstall pfsense .

i dont know what really happen , i will remove that server ( esxi ) from the network then reboot it . then fast logging to it and then disable the cards on the pfsense before it boot up . then access it using the esxi console and access pfsense shell .

but what commands can i use to debug that issue?

Thanks

cmb

Sounds like you're somehow creating a neverending loop of traffic. What type of WANs, static, DHCP, PPPoE, …? What is your network like in general? How is your gateway group configured?

It's difficult, but theoretically possible, to create a never ending routing loop with route-to (firewall rules specifying a gateway or gateway group) because it doesn't decrement the TTL. I'm guessing you're somehow doing something similar to that.

@Electrone:

but what commands can i use to debug that issue?

Things I would check, via console when it's broken:

pfctl -si

primarily interested in State Table, current entries.

State Table                          Total             Rate
  current entries                      375

To see how many active sessions there are. Knowing whether it's a small or large number of connections helps troubleshoot from there.

pftop shows some potentially useful stats, and may be enough to find the specific traffic in question.

pfctl -ss | more

to dump the state table, page by page. Glancing through a few pages may help.

And most useful, tcpdump. Grab some traffic from each interface to see what it's seeing. Pay attention to the source and destination MAC addresses as well.

tcpdump -nei em0 -c 1000 
tcpdump -nei em1 -c 1000
tcpdump -nei em2 -c 1000

Replace emX with vmxX or whatever your NICs are. Those will capture 1000 packets (good idea to put a count when capturing a flood), disable name resolution so it's easier to read and doesn't have delays, and show layer 2 info (including src and dst MACs).

Electrone

Thanks for your reply

well both GW are configured with static IPs

Group configured simply as tier 1 both and packet loss or high latency , then lan configured with that new grouped GW

this is a test network so it is very simple no vlans . just Normal Manged Switches without any network configurations.

both GWs are aDSL 20 MB

all SWs are Gb speed per each port

today i reinstalled pfsense and just configured the group and nothing more ( didnot install any packages ) and didnot create any FW rules , and it works fine .

btw if i didnot add any packages or didnot change any FW rules , it will work without any issue .
once i start creating FW rules or install packages , it will loop

i will try to recreate the same scenario and try to get more results.

i will update here later