Default deny overriding pass rule on one interface
-
Greetings! I am running 2.4.5-RELEASE-p1. Below is a diagram of my network
Apologies in advance for using mspaint. I would like to add that ALL ipv6 is disabled/unused.
I will post the output of my full firewall ruleset below. But apart from some NAT stuff for gaming on the WAN, I have all 3 of the local interfaces set to pass all/any ipv4 traffic.
-The LAN10 and WLAN1 subnets send and receive from each other
-The LAN10 and WLAN1 subnets send/receive WAN traffic
-The LAN30 subnet cannot talk to any other subnet, or the WAN, nor can devices on LAN30 ping the LAN30 gateway.
-The LAN30 DHCP server assigns a DHCP address to the laptop_ on the VLAN30 switchport on the managed switch.
-When the laptop attempts to access anything, it times out or fails.
-When I check the firewall logs, the traffic is absolutely getting to pfsense, but it's being blocked by the default deny ipv4 rule.Why is this happening?? I have no asymmetric WAN setup, no gateways specified in any of the firewall rules, DHCP traffic is working correctly on VLAN30, and all the other firewall rules for the other 2 local interfaces are configured identically-- in fact, I copied the any/all ipv4 rule from LAN10 and changed the interface to LAN30 in the rule before saving/applying.
Here's the DHCP lease for the laptop in the leases tab, so you know I'm not crazy (about that part at least =):
Things I have tried:
-Resetting the states
-Dis-/re-enabling LAN30
-Changing the IP of LAN30 and modifying the DHCP range (changed it back when it didn't change the behavior)
-Power cycling every device in my network, including the modem, the firewall the switch the wifi AP and my laptop
-Completely deleting the LAN30 interface, deleting the re1.30 VLAN, and re-creating the interface from scratch
-Using VLAN50 instead of VLAN30, then changing it back to VLAN30 when it didn't workHere's what I get in the firewall log when I try to navigate just to 192.168.30.1:
This is the output of pfctl -sr at the userrules anchor (redacted WAN IP data):
anchor "userrules/*" all pass in log quick on openvpn inet all flags S/SA keep state label "USER_RULE: OpenVPN wizard" pass in quick on re0 reply-to (re0 ###.###.###.###) inet proto udp from any to ###.##.###.### port = openvpn keep state label "USER_RULE: OpenVPN wizard" pass in quick on re0 reply-to (re0 ###.###.###.###) inet proto tcp from any to 192.168.10.128 port = 8124 flags S/SA keep state label "USER_RULE: NAT gaming" pass in quick on re0 reply-to (re0 ###.###.###.###) inet proto tcp from any to 192.168.10.70 port = 5321 flags S/SA keep state label "USER_RULE: NAT gaming2" pass in quick on re1 inet all flags S/SA keep state label "USER_RULE" pass in log quick on re1.10 inet all flags S/SA keep state label "USER_RULE" pass in log quick on re1.30 inet all flags S/SA keep state label "USER_RULE" pass in log quick on ath0_wlan0 inet all flags S/SA keep state label "USER_RULE" anchor "tftp-proxy/*" allWhat am I doing wrong???
(edited to tag code and redact wan IP data =)
-
@amorimpermissus can you post screenshots of your floating and vlan30 firewall rules?
-
@griffo Hi thanks, I added the output of the pfctl -sr command for all the user rules above, but here are screenshots too:
-
@amorimpermissus Weird. The logs seem to show it's coming in the right interface, which you'd think rules out switch tagging issues. But its simply not hitting that rule (which shows 0 active states)
What's the output of ifconfig for the re1.30 interface?Can you ping the 192.168.30.100 address from the firewall while doing a packet trace on the LAN30 interface?
What's the ARP table show for LAN30? Does it show the mac of the PC on .100 ?
-
@griffo Yeah I'll do the ping test in a bit. But it's definitely in arp
Here's ifconfig
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80003<RXCSUM,TXCSUM,LINKSTATE> ether 40:62:31:03:e5:f9 inet6 fe80::4262:31ff:fe03:e5f9%re1.30 prefixlen 64 scopeid 0x9 inet 192.168.30.1 netmask 0xffffff00 broadcast 192.168.30.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 30 vlanpcp: 0 parent interface: re1 groups: vlan -
@griffo here's the pcap for ping test. I just used the built in ping from diagnostics. FYI I disabled windows firewall before I ran the ping with netsh advfirewall set allprofiles state off
PING 192.168.30.100 (192.168.30.100): 56 data bytes --- 192.168.30.100 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss03:59:28.743718 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 135, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->bc6c)!) 192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 0, length 64 03:59:29.745060 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 30776, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->44bb)!) 192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 1, length 64 03:59:30.746068 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27366, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->520d)!) 192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 2, length 64 03:59:31.314871 54:ee:75:3b:ab:f9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.30.1 tell 192.168.30.100, length 46 03:59:31.314885 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.30.1 is-at 40:62:31:03:e5:f9, length 28 03:59:34.840063 c8:d1:2a:c3:b0:93 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.10.5, length 42 03:59:37.644328 54:ee:75:3b:ab:f9 > 40:62:31:03:e5:f9, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.30.1 (40:62:31:03:e5:f9) tell 192.168.30.100, length 46 03:59:37.644341 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.30.1 is-at 40:62:31:03:e5:f9, length 28 -
This is showing an outbound rule
That means the traffic was blocked on the outbound direction.
You can only place outbound rules in floating.
-
@johnpoz Hi and thanks so much for your response. I will add a floating rule and test, but I am also confused. As shown, I don't have any floating rules in existence. I created LAN30 (re1.30) the exact same way as LAN10 (re1.10). The only differences between the two are their VLAN and subnet. What did I misconfigure / forget that makes the LAN30 traffic behave differently than the LAN10 traffic (which has no problems getting to 192.168.10.1)?
-
I have no idea what you could of have done. I am just saying that the log is showing that is blocked on the outbound
And your traffic would be outbound as well that is your SA back to your client 30.100.. This would be outbound traffic from pfsense..
-
@johnpoz OK, just for the record, I created this floating rule:
I then waited for filters to reload and reset states. This is (still) the result when attempting to access 192.168.30.1 from 192.168.30.100:
(Apologies for not being able to zoom out more, doing the log viewing from my cell phone)
-
You got something really odd going on that is for sure.
Something is clearly messed up in you rules.. Since there would not be a default rule on the outbound side of any interface.
I would start over ;)
-
@johnpoz I've already completely blown away the interface config and recreated from scratch.
This is the full output of pfctl -sr (specific WAN address and explicit port rules redacted for security):
https://pastebin.com/VByW2jetIf you (or anyone!) is willing to look through it, I would be immensely grateful. I would like to avoid re-configuring the entire system, especially since I might just go through everything only to have the problem continue.
Thanks everyone so much for your time, even if you aren't able or willing!
My main concern with reinstall and reconfig is I have configured (but completely disabled) @marcelloc 's (you rock!) amazing E2Guardian package. I can't find any clear details about what files / folders to make an external backup of if I want to export and re-import the settings. By default, it bundles its backups into the auto-config backup settings, so I wouldn't be able to restore the E2Guardian settings without also restoring any (presumably-corrupted) rules, too. Not asking for anyone to solve that problem here, because I know it's not an official package. I'm just explaining why I'm so reluctant to blow the entire install away and start over =)
-
There is a default block out rule, but that should only come into play where there is no state.
Off the top, guess its possible that if traffic is being seen inbound on different interface than the vlan30 interface. Trying to send out different interface would be stopped by the default deny..
When you ping pfsense 30.1 from 30.100
What interface are you seeing the state create on. Or when you try and hit the gui, which is that 443 SA traffic your seeing..
-
@johnpoz Thanks for looking, I will check the states, shortly when trying to access the web gui on .30.1. I don't think it creates a state for ICMP though does it?
-
yeah you will see a "state" created for icmp
I started a constant ping to pfsense from my pc, then looking in the state table I see it.
edit: Off the top of my head that only thing that is coming to mind where the default block out deny rule would come into play. Is if there is no state for answer, which I would think could only happen when your trying to answer via a different interface than the traffic came in on.
So you have IP set on the native interface your re1, and only have vlans 10 for your lan and 30 for your other vlan sitting on this physical interface.
If traffic came into re1 without a tag, and source IP was 30.x then pfsense would try and answer via its vlan30 interface. But would be denied because no state on that interface - that is a GUESS to possible scenario that could maybe cause what your seeing..
-
@johnpoz Gotcha. Regardless, I started a pcap on the firewall, listening on the LAN30 interface for 192.168.30.100, along with opening up the states page in another tab and filtering on all interfaces to 192.168.30.100. Then I opened https://192.168.30.1 on my laptop. I saw it creating a TCP connection on 443 in Windows perfmon. On the pcap, both the pings and the https traffic showed up, but in the states page, it stayed at "No states were found that match the current filter," no matter how many times I refreshed.
If you don't see anything misconfigured in the pastebin of my firewall rules, I'm leaning toward thinking some system files (or disk sectors) are corrupted. It's just the only logical explanation at this point. Plus, I used UFS when I built this box, so it doesn't have the same protection from bitrot that ZFS does, even when you aren't using ECC memory.
I'm going to see if I can figure out what files/folders the E2Guardian package lands in, and copy those off. Then I'll wipe and reinstall. I hope it's not something inherently wrong with my hardware, but I'll update here once I reconfigure everything. That way you all will know whether it worked or not.
Thanks again, everyone, for your help!
-
Well a state can not be created unless allowed.
I didn't go through all your rules.. But if your saying your seeing traffic hit the firewall
But nothing logged on the inbound?
Your sure the traffic was tagged with vlan30? You can do a tcpdump on pfsense with the -e to view vlan info..
example
I did a tcpdump -i igb2 -e
Where igb2 is parent interface for multiple vlans - you can see here traffic from vlan4
11:48:12.539018 00:08:a2:0c:e6:20 (oui Unknown) > 68:57:2d:98:36:83 (oui Unknown), ethertype 802.1Q (0x8100), length 58: vlan 4, p 0, ethertype IPv4, ec2-54-187-112-38.us-west-2.compute.amazonaws.com.https > 192.168.4.202.53966: Flags [F.], seq 717, ack 851, win 3752, length 0 11:48:12.541636 68:57:2d:98:36:83 (oui Unknown) > 00:08:a2:0c:e6:20 (oui Unknown), ethertype 802.1Q (0x8100), length 60: vlan 4, p 0, ethertype IPv4, 192.168.4.202.53966 > ec2-54-187-112-38.us-west-2.compute.amazonaws.com.https: Flags [.], ack 718, win 3663, length 0 -
@johnpoz Yeah, the log data I posted the screenshots of are, consistently, the only thing appearing in the firewall logs, period, for traffic on that interface/subnet. Even when I do a continual ping from .30.100 to .30.1, the ICMP traffic never shows up in the firewall logs at all, but it's definitely showing up in tcpdumps:
https://pastebin.com/rR36L1QJ
And a tcpdump on re1 shows up as tagged for that MAC:
[2.4.5-RELEASE]/: tcpdump -i re1 -e | grep 54:ee:75:3b:ab:f9 listening on re1, link-type EN10MB (Ethernet), capture size 262144 bytes 12:58:40.912971 54:ee:75:3b:ab:f9 (oui Unknown) > 40:62:31:03:e5:f9 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 30, p 0, ethertype IPv4, SUPPORT3.demasci.intra > 192.168.30.1: ICMP echo request, id 1, seq 192, length 40 12:58:45.914341 54:ee:75:3b:ab:f9 (oui Unknown) > 40:62:31:03:e5:f9 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 30, p 0, ethertype IPv4, SUPPORT3.demasci.intra > 192.168.30.1: ICMP echo request, id 1, seq 193, length 40This is also confirmed by the fact that my laptop always receives an IP address from the DHCP server listening on LAN30. It's so. Freaking. Weird.
-
What rules do you have on port forwards, those are evaluated before.. You don't see them with your -sr
-
@johnpoz Eh, don't remember exactly-- blew the whole install away and reinstalled from scratch. After setup, same problem as before.
Even tried a different vlan again-- 50 instead of 30. Made no difference.
Moved the port configuration to a different switchport on my managed switch... it worked on VLAN 50.
Deleted the re1.50, recreated it as re1.30. Created the allow all firewall rule.
Traffic now working without any issues. I think I may have just put myself through hell for nothing, reinstalling and everything when it was something strange on my switch. But now my firewall is running ZFS instead of UFS, so that's good at least.
THE MORAL OF THE STORY IS THAT, EVEN IF:
-There are no errors in ethtool
-There is every indication that the packets are uncorrupted based on packet captures
-You have switched out all of your cablesIT COULD STILL BE SOMETHING DUMB ON YOUR SWITCH.
For anyone else who might be experiencing this, I am using a Comtrend GS-7408. Maybe all of this information contained herein could help someone else in the future >_<
