Several problems with two PFSense 2.7.0
-
Hello PFSense comunity.
I have had problems with PFSense 2.7.0 for a week that I cannot solve.
Some information.
There are two Supermicro X10SSL servers with PFSense software.
Both are connected together and work in HA and CARP.
Both are connected to 1Gbit/s internet.
I am attaching a connection diagram.
As I mentioned, I have had problems for a week that occurred suddenly and affect both FWs.
I have noticed that when pinging VIP, which is the GW of the network segment, it loses packets. Packet loss around 10%.
What is worse, the Internet throughput from 1Gbit/s suddenly dropped to about 5Mbit/s Download and about 15Mbit/s Upload.
I tried to uninstall the Demons. Both firewalls do not have Traffic Shapers set, I checked the Offload Options of network cards. I tried to restore the configuration from before the failure - no solution to the problem.I connected a different router (Cisco) to each of the WAN sources. On both there is a throughput of about 1Gbit/s measured when downloading files.
I don't know where to look, can someone suggest what to check?
Thanks for any help.To the moderators, this is my first post, if it is not in this section as it should be, please move the post.
-
@Tadmin said in Several problems with two PFSense 2.7.0:
occurred suddenly
So, find out what happened at that moment, and you'll be very close tho the solution.
Btw : from our point of view : could be ... anything. Only you can find the details.Btw : be ware that 'no one' is using the very deprecation 2.7.0 these days.
2.7.2 came out a long time ago.@Tadmin said in Several problems with two PFSense 2.7.0:
I tried to uninstall the Demons
?
@Tadmin said in Several problems with two PFSense 2.7.0:
I tried to restore the configuration from before the failure - no solution to the problem.
If the settings are 'as before' and the file system is still ok (binaries etc) did not change, then 'the bits' or software is still identical.
Then only one thing is left : hardware.Btw : what can severally break you system : as you are using an ancient pfSense, you can't upgrade or install pfSense packages anymore.
-
Thank you for your answer.
@Gertjan said in Several problems with two PFSense 2.7.0:
So, find out what happened at that moment, and you'll be very close tho the solution.
Btw : from our point of view : could be ... anything. Only you can find the details.Btw : be ware that 'no one' is using the very deprecation 2.7.0 these days.
2.7.2 came out a long time ago.I know what happened on the day of the failure, but I still can't find a logical explanation for the cause.
On the day of the failure, the ISP was splicing the optical fibers going to both firewalls again. Because this work was done at night, I detected the failure the next day.
Later, I checked both optical fibers separately by connecting them to a simple Small Business Cisco Router. Everything seemed to be fine then. The throughput was correct at about 1 Gbit/s.
Yesterday, I upgraded the PFSense version from 2.7.0 to 2.7.2, unfortunately, it didn't solve the problem.
Do you think that NIC could have stopped working in both devices at once?
Is it possible, does it happen?@Gertjan said in Several problems with two PFSense 2.7.0:
I tried to uninstall the Demons
?
I meant add-ons like Suricata, Ovpn, Suricata. I uninstalled all additional "plugins" to make sure that no add-on was causing the problem. After all, these are add-ons that can be easily reinstalled, so it was worth a try. I'm running out of ideas on what else I can do :/@Gertjan said in Several problems with two PFSense 2.7.0:
If the settings are 'as before' and the file system is still ok (binaries etc) did not change, then 'the bits' or software is still identical.
Then only one thing is left : hardware.Btw : what can severally break you system : as you are using an ancient pfSense, you can't upgrade or install pfSense packages anymore.
It seems that everything is fine with the file system. Both Supermicros have 2 SSD drives installed (mirror operation)
both drives do not show sector errors using "S.M.A.R.T.Damn, a failure of both cards at once seems so unlikely that it is impossible to occur.
And do you know what could be responsible for the loss of packets to GW?
If there was a problem with communication between two firewalls on SFP+ ports (see the diagram from the first post)
Would it manifest itself in packet loss and connection to the WAN?
Maybe we should take a closer look at this SFP+ connection going to the Core Switches?Is there a way to test the internet speed directly from PFsense? Something like "speedtest" directly on FW.
-
@Tadmin said in Several problems with two PFSense 2.7.0:
Is there a way to test the internet speed directly from PFsense? Something like "speedtest" directly on FW.
Like this .... speedtest on pfSense ?
nooo, never heard about that one ^^
@Tadmin said in Several problems with two PFSense 2.7.0:
responsible for the loss of packets to GW?
Yep, easy ...
Could be the NIC on pfSense side : it sends and pfSense thinks it has send packets but nothing left the interface ... so nothing comes back. Because you were not asking for it.
Or its the cable.
Or the NIC on the other side - ISP devices ?
Or the ISP device.
Or the connection between the ISP device and the ISP equipment.
And so on.@Tadmin said in Several problems with two PFSense 2.7.0:
two firewalls on SFP+ ports
Ah, forgot about that one.
Just out of my head : these SFP+ run burning hot.
When burned, things start to operate less well ?! -
I have some new information about this problem. Unfortunately, I still haven't solved it.
Today I discovered that by changing the NAT settings I am able to restore full internet speed.
If I select the "WAN interface" option in "Translation Address", the full speed appears. However, if I select "CARP WAN VIP", the speed drops to a few Mbit/s.I recorded packets with wireshark from the WAN port.
Both IP addresses (IP Address from WAN interface and CARP WAN VIP)
come out with the same MAC Address.Maybe this is the problem?
I tried to connect a switch + two routers to the cable from the ISP in which I set the same MAC Address but different IP addresses.
After starting the routers, they interfered with each other and worked interchangeably, but neither cut the other's speed. Maybe if they came out of the same physical device, there would be a different symptom.
Maybe it is somehow related to the ISP. But I can't confirm or deny it.
Does anyone have any ideas what it could be?
