New setup, first time user
-
Hi
I've been managing a small WISP network for familly and neighbors since we could get no better than bad satellite internet.
My knowledge base is limited. I mainly used ubiquiti products and I mostly copy/paste configurations.I was worried my edgerouter4 was no longer cutting it, I disabled some functions to ease its load but now I'm ready to try something else.
I reassembled my old pc, got an overkill intel 10gb NIC and installed pfsense.
I tried to copy the subnets of my edgerouter4 and added firewall rules for my server.I have good internet access when only my laptop is connected to the pfsense router but as soon as I plug the rest of the network to it, internet gets “sort of mostly blocked” and, it seems, by the firewall log, default deny rule is blocking port 443 when it shouldn't. Any to any rules are in place on the lans so that confuses me. Then randomly, a page loads.
Most of the wisp hardware is on subnets 192.168.88.0/23 192.168.11.0/24 10.0.0.0/24. They are on the same interface via ip aliases.
DHCP is running on a different interface on subnet 192.168.6.0/23 and I want the clients to access internet, the router(for my convenience) and my server.WAN is set to DHCP and properly gets an IP
I tried resetting states and rebooting the brocade switch it is connected to.
PFsense is version 2.6.0
Here's a few snaps of my config:
As can be shown, I tried adding some redundant any to any rules.
Can someone tell what is wrong with my setup from this?
PS..
I am having problems with my new fiber ISP, pages are slow to load and they blamed my router as everything is fine when only my laptop is connected. MTR results are bad. Speedtests are good. My step-brother has his own connection with them and got a tech to admit that they are having problems but wont publicize it.
I wanted an overkill router to show them they have a problem but it seems they already know. -
@sphex where is this 192.168.11 network - not seeing it on your interfaces?
Are you using a larger than /24 mask - I see a 192.168.6.1 on your landhcp interface..
Are these blocks your seeing to 11.4:443 Syn or SA, or something other than SYN?
If your mask on your 192.168.6 interface also includes 192.168.11 then those rules are pretty pointless on your lan dhcp interface allowing traffic allowing access to 6.4 and 11.4 - traffic on the same network would never talk to pfsense. Pfsense is a router to get off a network.. But since I see hits on the 11.4 rule - that would tell me atleast your client doesn't think 11.4 is on the same network?
-
@johnpoz
192.168.88.0/23 192.168.11.0/24 10.0.0.0/24. They are on the same interface via ip aliases. Or should be anyway. I configured them in the virtual ip menu.
192.168.6.0/23 is on landhcp interface
Likely, most of the ubiquiti wisp hardware is pinging and reporting to the server on 11.4:443 or x.x.x.4, that is around 300 devices. Now I think most are on x.x.88.x/23 subnets. I had been moving them to eventually delete the 192.168.11.0/24 subnet. I don't know all the protocols they use but I think I want them to work. -
@sphex said in New setup, first time user:
They are on the same interface via ip aliases
well that is not a good setup at all.. Different networks should be setup with vlans, or multiple physical interfaces - running multiple layer 3 on the same layer 2 is not good idea..
But those blocks your seeing to 11.4:443 are they syn blocks or some other flags, like SA, A or FA, R etc..
-
@johnpoz said in New setup, first time user:
But those blocks your seeing to 11.4:443 are they syn blocks or some other flags, like SA, A or FA, R etc.
I don't know what that is or how to tell what it is.
-
@sphex look at the full firewall log vs just the widget
-
@sphex I'm not that familiar with WISP but by "WISP hardware" is that how the connection is made to the Internet? I'm not really clear why that would not be on WAN and why it would need multiple subnets but I may just be not understanding. I'm asking because the clients on LAN1DHCP would not go through the LANINTERNETJCB network to get out to the Internet, which might cause asymmetric routing.
Your four "allow server JCB" rules will never trigger because of the two LANINTERNETJCB files allowing all traffic, which will also not trigger because of the "any to any rule" which I'm guessing is temporary.
pfSense will block stray/extra/resent packets if the state is closed. See https://docs.netgate.com/pfsense/en/latest/troubleshooting/log-filter-blocked.html There's a picture on that page, see the Protocol column in the block log for the type of block.
Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
Upvote 👍 helpful posts! -
^ exactly - if the block is not a SYN (S) then it points to out of state for the reason for the block, vs any sort of firewall rule not allowing it.
-
@johnpoz
They are marked (A) -
If they are all ACK packets that looks like an asymmetric route.
I also notice pfSense shows it;s unable to check for updates.
With a DHCP WAN and only one gateway it should always have a correct default route so I would check for DNS issues.Steve
-
@steveits
The router is connected to fiber.
Clients, far from the fiber, are connected to it via wifi devices in bridge or router mode. NATed, I believe, is the word. There is only one public IP.
I don't realy need multiple subnets, I just have to get around to clean that up and it never seemed to matter.
LAN1DHCP clients need to talk to my server on a different subnet only because I didn't take the time to tell them they can do it on their own subnet. ie 6.4 is also 11.4 and 88.4
Yes my firewall rules are redundant. I tried something... -
@sphex if they were all SA then that would scream asymmetrical as mentioned by @stephenw10 but if they are all just A.. that could just be loss of a state..
But yeah running vips and multiple L3 networks on the same L2 could lead to asymmetrical problems for sure. Or no state for traffic, or loss of state on timeout as once the devices start talking they would not need to bounce off the router/firewall.
You really need to clean up the use of multiple L3 on the same L2.. Be it you use different physical networks, or vlans - running multiple L3 on the same L2 can be very problematic, and should always be avoided..
Or just put them all in the same network - not like there is any actual security between these networks since they are on the same L2 network.
-
@stephenw10
I probably took the snapshot after I moved things back to the edgerouter hence no internet on the pfsense.Could be the same with the TCP:A blocked packets, meaning they don't reflect the main problem if I didn't take the snapshot while everything was plugged.
I'll have to check the firewall log again next time I try to use PFsense. I'm not doing that in the day.
-
@johnpoz said in New setup, first time user:
You really need to clean up the use of multiple L3 on the same L2
Ok thanks
I'll start with that. -
Yup that will probably solve all the issues. But even if it doesn't it makes diagnosing it many time easier!
-
@johnpoz
I cleaned up my network, devices properly communicate with the server without going through the router. The main subnet is alone on its interface.I still lose internet when I plug the 300+ devices network(192.168.88.0/23) to the pfsense router.
I am convinced it is because of my ISP. Gateway status is marked as down (or something, I didn't take a snapshot) with latency above 3 seconds.
My current edgerouter4 is configured with load-balance with another (slower) provider and I had to set the route-test failure count to 60 so that it wouldn't drop the connection during rush hour.(for the problematic ISP only)
That old setup works*, MTR test results are ugly, smart TVs say they are disconnected every 5 min but VoIP works fine, gaming and streaming is fine.
Because the pfsense router isn't setup in load-balance, I didn't think it would drop the WAN connection if latency was too bad. How can I tell if that is the case and can I adjust/disable the fail-route test of the pfsense router?
I was hoping those 10gb interfaces might help but seems it was a waste. That NIC has tx rx pause option, could it be worsening the problem?
Otherwise, could a purpose-built 10w router be that much better that my 13 years old pc? -
@sphex said in New setup, first time user:
Gateway status is marked as down
if pfsense believes its gateway is down - then yeah not going to work.. if its not really down you can set it to be considered always up..
-
If your ISP has massive buffer-bloat you're going to see large latency increases when traffic increases however powerful your router is. To actually address that you need to use some traffic shaping on the firewall.
If you only have one gateway defined it will always be the default route and pfSense will always try to use it. However it will still trigger a bunch of scripts that aren't required if you only have one. So I'd recommend editing the gateway and setting 'Disable Gateway Monitoring Action' to prevent that. However if you move the load-balancing over to it you will need to re-enable it.
Where do you lose internet access from when you connect the 192.168.88.0/23 devices? What are you actually doing to connect them?
Steve
