WAN acces droping packets
-
Hi,
i have strange problem with pfsense and my FTTH WAN connection. I already lost many days/weeks trying to solve the problem, but i cannot, that is why i am asking here.
Problem is, that when i use pfsense with my internet ISP provider using optical cable (and through switch) pfsense suddenly start droping packets.
If i use any other router (Zyxel, Endian, Clearos …) it works OK.What is even more strange, is, that if i use router before pfsense, than it works OK.
Please see my attached schema.png file, where i have drawn both scenarious:
-Scenario 1: Where i use router between pfsense and my WAN provider. I use router because i also have 20 public fixed IP addresses, and for this i need mediation router (Zywall USG). In this scenario pfsense works with no packets droping. All works OK. If i use pfsense for my mediation router, i have dropped packets.
-Scenario 2: Here i have pfsense (WAN) connected directly to ISP switch. I have this, because i have 2 public fixed IP addresses from before (and i use them). Here, i sudenly start to have dropped packets. It looks like that: suddenly internet stops working, and for 30 second all packets are droped (for all workstations), and than sudenly it start working again. Than this occurs again after some time (maybe 20 minutes, or few hours). If i use ADSL for WAN it works OK. Also if i use other router it works OK. Only combination of FTTH from this ISP and pfsense have problems.
I ran packet capture in pfsense during this problem (pinging some host) . And on LAN interface you can see in LAN.PNG, that workstation sends packets, and no reply is received.
BUT on WAN interface (see attachement WAN.PNG) , there is no ICMP REQUEST messages visible. As if pfsense is not sending packages to WAN interface. ICMP requests are sent every second, but (as you can see on picture WAN.PNG) no packages are sent after 12:08:47 until 12:09:17.Pictures from LAN and WAN wireshark traces are not taken on the same time. I will try to take them manualy over ssh for "any" interface.
What can be the problem with my pfsense ?
PS: I have tried running pfsens on virtual and physical machine and same problem. So there is no HW problem.
I have tried to check : Clear invalid DF bits instead of dropping the packets
I have set Firewall Optimization Options to conservative
I have enabled net.inet.ip.redirect to 1Nothing helped.
In logs there is nothing visible in that time (when packets are droping) except in Gateways i see that GW is lost.
thanks for any help,
iI would realy love to use pfsense, but this problems are eating my nerves :)br,
Andrej
-
I added 2 more pictures of the problem. All for same time.
-
Let's start at the beginning.
There should be no reason why you need a firewall before your pfsense box. If the box is connected to the switch, which is also terminating the fiber connection, and you have a proper layer 2 vlan (unrouted) configured, you should be able to access the WAN with no issue.
Can you provide a list of your firewall rules? It sounds like the intermediary router is NAT'ing… which may be causing some of your issues in that scenario.
-
Dear Mike,
For my problem, i do not have any firewall before pfsense.
I just show that for example, that if i have FW/router before pfsense, than all is ok.Also my WAN works, problem is that suddenly i have lost packets. And that problem is only with my fiber connetion. If i use ADSL on same pfsense, all is ok.
Right now pfsense is setup as installed. No NAT rules (except aoutomatic outbound, which is default)
Also i am sure my configuration and setup is correct (i am not network novice). There is one reason why pfsense do not work OK with my fiber ISP and that is hard to figure it out.
thanks and br,
Andrej -
Is the fiber converter/modem hardcoded to a specific link speed/duplex? It's not uncommon to find those hardcoded to 100/full and if you do not match their hardcoded speed in the interface settings, you will see lost packets as a result of the speed/duplex mismatch.
-
jimp,
regarding fiber switch i have interesting story:
Few months ago, i installed pfsense for the first time. I had this problems (droping packets).
So i decided to connect fiber directly to my main switch with SFP, to eliminate switch from my ISP (it is MILAN switch with 100Mb/s interfaces).And after that, all was working ok with pfsense (no packets was dropped anymore). And i said: problem solved.
And that was until we had power failure (2 week ago) and my main switch got restarted. After that, i have droped packets also when fiber is connected to my main switch.
Also connection to my Vmware is done over 10G copper connection. But i have tried direct LAN to Esxi host, but no luck.
PS: Now (until i solve problem with pfsense) i am running Endian FW for my main router (insted of pfsense) and it works ok (also on vmware with same interfaces).I will try m0n0, since it is also BSD based.
br,
Andrej -
The switch may have had its port set to a specific speed/duplex but then lost that on reboot and went back to automatic. Or, by luck, it may have been at the right speed but then autonegotiate failed on the next switch boot.
-
jimp: so i should try to FIX speed on my SFP connection ?
I will try it thanks for suggestion. -
It depends on how it's set, really. You need to find out from the fiber provider how their port/handoff is set and then set yours to match.
-
Update:
Speed change did not do anything.So than i connected SFP tp other switch.
So now i have FTTH –> switch --> switch --> Vmware
And now it works OK with no packets lost.Port speef is same as on prevoius swtich (1000G full).
And why it works ? Go figure.
