Ports some days opened, other days all closed
-
Hi
I have a VDSL modem (BBOX 3 - Proximus - Belgacom). The PPPoE on the VDSL modem is disabled and the PPPoE connection is made on the pfSense. So I have NAT also enabled and some ports are forwarded (SSH, DVBLink, Grafana, OpenHab, etc..). IPv6 is also enabled and working.
Every day at 23h30 there is a shutdown (a hard reset I think, no nice shutdown - not sure) and at 6h30 he comes back.
Some days I can access all of the open ports and it stays working until the end of the day. Some days I can't access none of the ports until the end of the day. It recovers itself the day after or two days after. Internet from in to out always keeps working, it's only out to in which is like a random change of working by 40%.
Strange thing is when I check the connection log at the ISP I see connections of 24h and they are stopped at 2h30 at night.
Could it be the hard shutdown so the PPPoE keeps living as a ghost connection? But that's strange as it stops at 2h30 at night and pfSense boots later on. My ISP won't help me as it is behind there hardware.I have a dynamic IP with a script sending the IP every 30 minutes. I checked every time I had the problem and then the IP was correct.
I noticed I can also solve it by powering off the VDSL modem and connecting it again. So without touching anything in pfSense. But of course I guess pfSense just reconnect that time.So IN > OUT always working.
OUT > IN sometimes working.Could it be a timing issue somewhere?
-
I'm having a similar problem.
I have a pppoe WAN (vr0), a ppp backup WAN and two lan re0 and re1 (last one with captive portal).After every reboot almost one of the network connection is not working. For example:
- internet is ok but no NAT rules are working from outside
- pfsense is unreachable from LAN (no ping) but internet is ok and I get DHCP address (OPT1 is OK)
- pfsense is unreachable from OPT2 so no captive portal (but DHCP is working) (LAN is OK)
after some reboot everything start working again, but it seems to me that the only troubleless reboot is the one from the console.
I have the last updates plus squid enabled on opt1.Since dhcp is always reachable it seems a firewall related issue and not an hardware/nic problem.
After many years of ipcop pfsense seems too unreliable even for home use.
-
Good to hear I'm now alone!
Sounds like a firewall issue indeed.
I'm also up to date and not running any additional plugins, so no squid.
I boot and shutdown using IPMI (Super Micro)Question is how to debug this?
-
I tried digging around SystemAdvancedNetworking
If I set "disable Disable hardware checksum offload" I'm cut off the firewall: only the 443 openvpn rule is working but even if i can successfully connect to the vpn my lan subnet is unreachable. I can only see the webgui.
-
I would check the pfSense bug-tracker to see if this is a known bug, but I would focus on user error as well since both of you seem to be new to pfSense.
I know that I always have strange things happen when I first begin using something that is unknown or new to me.
-
Don't mess with advanced settings trying to troubleshoot something that's hit and miss as described, at best it won't do anything, at worst it'll make things worse and harder to troubleshoot.
When it's not working, packet capture on WAN filtered on the port in question, see if it's getting to you at all. If not, problem is upstream. If so, see if a state is created, see if the internal host is replying.
-
The problem is difficult to reproduce and is not always the same. Every time I reboot I keep my finger crossed because I'm not sure that all networks interfaces will be working fine.
I got the followings cases:
A) Firewall is up but unreachable from lan. DHCP is working and I get internet connection, but nmap from host to firewall shows host down.
B) From LAN everything is ok, but all forwarded ports are closed from wan. Tested with Shields Up - GRC.com and everything is stealth.
C) From LAN is ok and wan forwarded ports are ok, but from secondary LAN the firewall is unreachable. The clients get IP address like case A but the captive portal is unreachable so no internet connection. I tried to add MAC address to authorized hosts but still no internet.
Every time after several reboot the firewall start to work fine again.
Once I had to restore a previous backup to get things work again, but I had not changed the configuration between.My dmesg
http://pastebin.com/N4sisRx8 -
The problem is difficult to reproduce and is not always the same. Every time I reboot I keep my finger crossed because I'm not sure that all networks interfaces will be working fine.
I got the followings cases:
A) Firewall is up but unreachable from lan. DHCP is working and I get internet connection, but nmap from host to firewall shows host down.
B) From LAN everything is ok, but all forwarded ports are closed from wan. Tested with Shields Up - GRC.com and everything is stealth.
C) From LAN is ok and wan forwarded ports are ok, but from secondary LAN the firewall is unreachable. The clients get IP address like case A but the captive portal is unreachable so no internet connection. I tried to add MAC address to authorized hosts but still no internet.
Every time after several reboot the firewall start to work fine again.
Once I had to restore a previous backup to get things work again, but I had not changed the configuration between.My dmesg
http://pastebin.com/N4sisRx8Why does nmap fail? No ping response?
Can you share your network topology?
-
Yes, no ping response. In this case I cannot ping any lan/opt2 host from the pfsense console too. And DHCP and nat are working because I have internet on lan.
WAN (wan) -> pppoe0 -> v4/PPPoE: 88.149.xxx.xxx/32
PRIMARY BROADBAND CONNECTIONLAN (lan) -> re0 -> v4: 192.168.1.250/24
LAN CONNECTION. SQUID ENABLED BUT NOT TRANSPARENTOPT1 (opt1) -> ppp1 -> v4: 10.90.209.120/32
BACKUP MOBILE CONNECTION (/dev/cuaU0.0)OPT2 (opt2) -> re1 -> v4: 192.168.2.250/24
GUEST CONNECTION WITH CAPTIVE PORTAL AND TRANSPARENT SQUIDOPENVPN 172.16.0.0/24 on port 443 TCP, shared with a port forward
configured through wizardGateways
WAN_PPPOE 81.174.0.21 8.8.8.8 17.315ms 13.176ms 0.0% Online Interface WAN_PPPOE Gateway
OPT1_PPP 10.64.64.1 8.8.4.4 54.214ms 4.702ms 0.0% Online Interface OPT1_PPP GatewayFirewall (no traffic shaper, no schedules, Automatic outbound NAT, no floating rules)
Please ignore the duplicated ssh entry: I was using a mobile browser and my tap was not precise. Now there's again only one ssh forward.
-
I upgraded this morning and did the capture (see attachment). Ports did arrived in the capture. Strange this is this time I DID see the traffic in the firewall logs.
Now the strange thing: it was blocked because the destination wasn't 10.0.0.107. Now that's a rule PFsense created himself by the NAT rule.
After adding the same rules but with a wildcard destination it all seems to work for now. It was blocked because the destination wasn't the local address, but the public IP.
Now I can understand it was blocked, only it's strange that:- The NAT rule used the local destination and not a wildcard? And you can't change it also because it's greyed out.
- Most strange things: it used to work 50% of the days and 50% of the days it didn't. So why did it worked sometimes? Before the wildcard destinations didn't exist.
- Why wasn't this logged in the firewall logs in previous version?
Firewall now:
-
Seems that it isn't solve. Problem got worse in that case that instead of closed ports I now end up with opening my firewall to the internet: https://forum.pfsense.org/index.php?topic=114731.0
-
You for sure do not want or need wild card dest for your nat stuff. Your nat rules go first, then that traffic would hit your firewall and its dest would be where the nat is sending it.
So I am curious how many connections do you have open due to the tarpitting? Maybe your running into a connections limit issue? Or issue with memory due to connections staying open?
-
Regardless of what's been said here I'm 100% sure that pfSense is not the problem unless there's some serious misconfiguration involved. Start looking at the equipment that is in front of pfSense on the WAN side, contact your ISP and ask if they can assist with the problem in case there's something wonky on their side.
-
Well first thing I would fix is a HARD boot everyday?? For what possible reason would anyone do that, does power go off? That for sure needs to be fixed!
2nd thing would be anything enolved with dynamic IP - is your IP changing? So if some fqdn points to 1.2.3.4, your IP changes to 5.6.7.8 no shit forwards wouldn't be working if still going to 1.2.3.4 ;)
-
Hi
The box goes off for about 7 hours a day because it's running off grid on batteries with solar cells. I think that shouldn't be a problem?
The IP is dynamic, but changes at night when it's powered off. I can see this in the statistics of my ISP.
My ISP is a big one and doesn't want to help me: what happens behind there lines is my problem they say (don't have much choose of ISP in Belgium).
The reason I added the wildcards is because it doesn't work if it isn't. In my previous post you can see a ZIP with traffic captures.
As you open that in Wireshark, you can see it was blocked because the destination isn't 10.0.0.107 but the public dynamic IP. Hence the wildcard.
The capture was done on pfsense itself and then exported and opened locally with Wireshark.
Or is there another way to solve this?
-
What I can tell you for FACT is that when you do a nat, you do not need a rule with dest Any.. You need a firewall rule that is to the IP to where the nat is sending the traffic. That is how it works, that is how the nat creates the rule when you create the nat. That is how mine are setup and they work just fine.
I don't know what you saw in your traces. Most likely you did not have rule or nat active? See example below, I created a nat to 22, it created rule on my wan that says the private IP. As you can see from the sniff on my wan the packet is set to my public IP.. but still gets through to my box behind on the private.
So if your power goes off because its on battery you need to look into doing a graceful shutdown.. Hard off anysort of device is rarely a good idea. Especially when the media the device is running off of it writable.. Is this running off some ro cf card or something?
If the battery runs out of juice, I would setup so your devices gracefully shutdown before the juice runs out.. And then comes back online when there is sufficient juice to run.. Or get a bigger battery to last while the sun is down ;)
-
Hi
I removed the wildcards and I changed the shutdown script to a shutdown -p now and now all ports are blocked again some days (not all days).
The days they are blocked, it shows in the Firewall log? So they are blocked because the destination is my WAN address?
What extra rule do I need?
-
None of those rules show any hits on them at all.. Why is floating and Wan lite up.. Do you have any rules in floating?
What rule is blocking those. Turn on showing the rule in the log..
Status / System / LogsSettings
Are you seeing any errors in system log about the loading of the rules?
-
What do you mean by "Why is floating and Wan lite up"?
When I click "floating", the list is empty:No floating rules are currently defined. Click the button to add a new rule.
.
Which option in Logs > Settings do you mean?
There are errors, but they seem to be IPv6 related, so not linked I guess:
-
Nothing with "rule" in it after booting:
