Default deny overriding pass rule on one interface

amorimpermissus

@griffo Yeah I'll do the ping test in a bit. But it's definitely in arp

Here's ifconfig

flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80003<RXCSUM,TXCSUM,LINKSTATE>
	ether 40:62:31:03:e5:f9
	inet6 fe80::4262:31ff:fe03:e5f9%re1.30 prefixlen 64 scopeid 0x9
	inet 192.168.30.1 netmask 0xffffff00 broadcast 192.168.30.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 vlanpcp: 0 parent interface: re1
	groups: vlan

amorimpermissus

@griffo here's the pcap for ping test. I just used the built in ping from diagnostics. FYI I disabled windows firewall before I ran the ping with netsh advfirewall set allprofiles state off

PING 192.168.30.100 (192.168.30.100): 56 data bytes

--- 192.168.30.100 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

03:59:28.743718 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 135, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->bc6c)!)
    192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 0, length 64
03:59:29.745060 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 30776, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->44bb)!)
    192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 1, length 64
03:59:30.746068 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27366, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->520d)!)
    192.168.30.1 > 192.168.30.100: ICMP echo request, id 62039, seq 2, length 64
03:59:31.314871 54:ee:75:3b:ab:f9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.30.1 tell 192.168.30.100, length 46
03:59:31.314885 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.30.1 is-at 40:62:31:03:e5:f9, length 28
03:59:34.840063 c8:d1:2a:c3:b0:93 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.10.5, length 42
03:59:37.644328 54:ee:75:3b:ab:f9 > 40:62:31:03:e5:f9, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.30.1 (40:62:31:03:e5:f9) tell 192.168.30.100, length 46
03:59:37.644341 40:62:31:03:e5:f9 > 54:ee:75:3b:ab:f9, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.30.1 is-at 40:62:31:03:e5:f9, length 28

johnpoz

This is showing an outbound rule

That means the traffic was blocked on the outbound direction.

You can only place outbound rules in floating.

amorimpermissus

@johnpoz Hi and thanks so much for your response. I will add a floating rule and test, but I am also confused. As shown, I don't have any floating rules in existence. I created LAN30 (re1.30) the exact same way as LAN10 (re1.10). The only differences between the two are their VLAN and subnet. What did I misconfigure / forget that makes the LAN30 traffic behave differently than the LAN10 traffic (which has no problems getting to 192.168.10.1)?

johnpoz

I have no idea what you could of have done. I am just saying that the log is showing that is blocked on the outbound

And your traffic would be outbound as well that is your SA back to your client 30.100.. This would be outbound traffic from pfsense..

amorimpermissus

@johnpoz OK, just for the record, I created this floating rule:

I then waited for filters to reload and reset states. This is (still) the result when attempting to access 192.168.30.1 from 192.168.30.100:

(Apologies for not being able to zoom out more, doing the log viewing from my cell phone)

johnpoz

You got something really odd going on that is for sure.

Something is clearly messed up in you rules.. Since there would not be a default rule on the outbound side of any interface.

I would start over ;)

amorimpermissus

@johnpoz I've already completely blown away the interface config and recreated from scratch.

This is the full output of pfctl -sr (specific WAN address and explicit port rules redacted for security):
https://pastebin.com/VByW2jet

If you (or anyone!) is willing to look through it, I would be immensely grateful. I would like to avoid re-configuring the entire system, especially since I might just go through everything only to have the problem continue.

Thanks everyone so much for your time, even if you aren't able or willing!

My main concern with reinstall and reconfig is I have configured (but completely disabled) @marcelloc 's (you rock!) amazing E2Guardian package. I can't find any clear details about what files / folders to make an external backup of if I want to export and re-import the settings. By default, it bundles its backups into the auto-config backup settings, so I wouldn't be able to restore the E2Guardian settings without also restoring any (presumably-corrupted) rules, too. Not asking for anyone to solve that problem here, because I know it's not an official package. I'm just explaining why I'm so reluctant to blow the entire install away and start over =)

johnpoz

There is a default block out rule, but that should only come into play where there is no state.

Off the top, guess its possible that if traffic is being seen inbound on different interface than the vlan30 interface. Trying to send out different interface would be stopped by the default deny..

When you ping pfsense 30.1 from 30.100

What interface are you seeing the state create on. Or when you try and hit the gui, which is that 443 SA traffic your seeing..

amorimpermissus

@johnpoz Thanks for looking, I will check the states, shortly when trying to access the web gui on .30.1. I don't think it creates a state for ICMP though does it?

johnpoz

yeah you will see a "state" created for icmp

I started a constant ping to pfsense from my pc, then looking in the state table I see it.

edit: Off the top of my head that only thing that is coming to mind where the default block out deny rule would come into play. Is if there is no state for answer, which I would think could only happen when your trying to answer via a different interface than the traffic came in on.

So you have IP set on the native interface your re1, and only have vlans 10 for your lan and 30 for your other vlan sitting on this physical interface.

If traffic came into re1 without a tag, and source IP was 30.x then pfsense would try and answer via its vlan30 interface. But would be denied because no state on that interface - that is a GUESS to possible scenario that could maybe cause what your seeing..

amorimpermissus

@johnpoz Gotcha. Regardless, I started a pcap on the firewall, listening on the LAN30 interface for 192.168.30.100, along with opening up the states page in another tab and filtering on all interfaces to 192.168.30.100. Then I opened https://192.168.30.1 on my laptop. I saw it creating a TCP connection on 443 in Windows perfmon. On the pcap, both the pings and the https traffic showed up, but in the states page, it stayed at "No states were found that match the current filter," no matter how many times I refreshed.

If you don't see anything misconfigured in the pastebin of my firewall rules, I'm leaning toward thinking some system files (or disk sectors) are corrupted. It's just the only logical explanation at this point. Plus, I used UFS when I built this box, so it doesn't have the same protection from bitrot that ZFS does, even when you aren't using ECC memory.

I'm going to see if I can figure out what files/folders the E2Guardian package lands in, and copy those off. Then I'll wipe and reinstall. I hope it's not something inherently wrong with my hardware, but I'll update here once I reconfigure everything. That way you all will know whether it worked or not.

Thanks again, everyone, for your help!

johnpoz

Well a state can not be created unless allowed.

I didn't go through all your rules.. But if your saying your seeing traffic hit the firewall

But nothing logged on the inbound?

Your sure the traffic was tagged with vlan30? You can do a tcpdump on pfsense with the -e to view vlan info..

example

I did a tcpdump -i igb2 -e

Where igb2 is parent interface for multiple vlans - you can see here traffic from vlan4

11:48:12.539018 00:08:a2:0c:e6:20 (oui Unknown) > 68:57:2d:98:36:83 (oui Unknown), ethertype 802.1Q (0x8100), length 58: vlan 4, p 0, ethertype IPv4, ec2-54-187-112-38.us-west-2.compute.amazonaws.com.https > 192.168.4.202.53966: Flags [F.], seq 717, ack 851, win 3752, length 0
11:48:12.541636 68:57:2d:98:36:83 (oui Unknown) > 00:08:a2:0c:e6:20 (oui Unknown), ethertype 802.1Q (0x8100), length 60: vlan 4, p 0, ethertype IPv4, 192.168.4.202.53966 > ec2-54-187-112-38.us-west-2.compute.amazonaws.com.https: Flags [.], ack 718, win 3663, length 0

amorimpermissus

@johnpoz Yeah, the log data I posted the screenshots of are, consistently, the only thing appearing in the firewall logs, period, for traffic on that interface/subnet. Even when I do a continual ping from .30.100 to .30.1, the ICMP traffic never shows up in the firewall logs at all, but it's definitely showing up in tcpdumps:

https://pastebin.com/rR36L1QJ

And a tcpdump on re1 shows up as tagged for that MAC:

[2.4.5-RELEASE]/:  tcpdump -i re1 -e | grep 54:ee:75:3b:ab:f9
listening on re1, link-type EN10MB (Ethernet), capture size 262144 bytes
12:58:40.912971 54:ee:75:3b:ab:f9 (oui Unknown) > 40:62:31:03:e5:f9 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 30, p 0, ethertype IPv4, SUPPORT3.demasci.intra > 192.168.30.1: ICMP echo request, id 1, seq 192, length 40
12:58:45.914341 54:ee:75:3b:ab:f9 (oui Unknown) > 40:62:31:03:e5:f9 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 30, p 0, ethertype IPv4, SUPPORT3.demasci.intra > 192.168.30.1: ICMP echo request, id 1, seq 193, length 40

This is also confirmed by the fact that my laptop always receives an IP address from the DHCP server listening on LAN30. It's so. Freaking. Weird.

johnpoz

@amorimpermissus

What rules do you have on port forwards, those are evaluated before.. You don't see them with your -sr

amorimpermissus

@johnpoz Eh, don't remember exactly-- blew the whole install away and reinstalled from scratch. After setup, same problem as before.

Even tried a different vlan again-- 50 instead of 30. Made no difference.

Moved the port configuration to a different switchport on my managed switch... it worked on VLAN 50.

Deleted the re1.50, recreated it as re1.30. Created the allow all firewall rule.

Traffic now working without any issues. I think I may have just put myself through hell for nothing, reinstalling and everything when it was something strange on my switch. But now my firewall is running ZFS instead of UFS, so that's good at least.

THE MORAL OF THE STORY IS THAT, EVEN IF:
-There are no errors in ethtool
-There is every indication that the packets are uncorrupted based on packet captures
-You have switched out all of your cables

IT COULD STILL BE SOMETHING DUMB ON YOUR SWITCH.

For anyone else who might be experiencing this, I am using a Comtrend GS-7408. Maybe all of this information contained herein could help someone else in the future >_<

johnpoz

What doesn't make any sense with the switch being the problem is that you were seeing stuff tagged with vlan 30, but it was being blocked outbound..

Some piece of the puzzle is missing to what was actually going on.. But at least you got it working. And zfs is plus coming out of it. I am waiting for 2.5 to drop and then will do a clean install myself to go to zfs on my sg4860.

Griffo

In one of your earlier posts, the dump shows some "Bad Checksum" packets at the beginning. That indicates to me a hardware or switchport config issue, like the vlan tagging was not right leading to framing issues. I tried to post that last night but was still in draft on my phone this morning :-( Glad you fixed the issue.

amorimpermissus

@griffo Doh! Oh well, better late than never lol