Network firewall rules not doing what I expect

incertia

I feel like this is one of those questions that end up getting asked a lot, but let's do it yet again.

I have an interface set up on a VLAN, connected to a Unifi AP broadcasting an SSID on the VLAN. My laptop is able to connect to the SSID and obtain a DHCP address on the correct network (192.168.30.0/24). However, all other traffic is blocked. With no firewall rules (sans pfBlockerNG) on the interface, there is absolutely no connectivity, even within the same subnet (additionally with WAN as well). I am even unable to ping the pfSense box at 192.168.30.1. Of course this may also be due to no firewall rules. So I add one which should pass traffic through the subnet but to no avail.
rules
For all my other interfaces I have skirted around the issue by setting source and destination to any but I would like to change this ASAP.
The interface does indeed see the traffic and pinging from pfSense to the device works as expected.

18:44:47.122214 IP 192.168.30.100.63438 > 192.168.30.1.domain: 19580+ A? ap.spotify.com. (32)
18:44:49.356631 IP 192.168.30.100.64416 > 192.168.30.1.domain: 40732+ A? 48-courier.push.apple.com. (43)
18:44:49.384666 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 0, length 64
18:44:50.391544 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 1, length 64
18:44:50.555514 IP 192.168.30.100.51241 > 192.168.30.1.domain: 25649+ A? bag.itunes.apple.com. (38)
18:44:51.399414 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 2, length 64
18:44:52.116178 IP 192.168.30.100.63870 > 192.168.30.1.domain: 42418+ A? api.apple-cloudkit.com. (40)
18:44:52.116184 IP 192.168.30.100.50602 > 192.168.30.1.domain: 14258+ A? cl2.apple.com. (31)
18:44:52.389959 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 3, length 64
18:44:52.966850 IP 192.168.30.100.50602 > 192.168.30.1.domain: 14258+ A? cl2.apple.com. (31)
18:44:53.450264 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 4, length 64
18:44:54.494938 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 5, length 64

Meanwhile a tcpdump with everything set to all on a different network looks like:

18:48:51.263909 IP 192.168.20.103 > 192.168.20.1: ICMP echo request, id 1, seq 23, length 40
18:48:51.263946 IP 192.168.20.1 > 192.168.20.103: ICMP echo reply, id 1, seq 23, length 40
18:48:52.264663 IP 192.168.20.103 > 192.168.20.1: ICMP echo request, id 1, seq 24, length 40
18:48:52.264684 IP 192.168.20.1 > 192.168.20.103: ICMP echo reply, id 1, seq 24, length 40
18:48:53.265910 IP 192.168.20.103 > 192.168.20.1: ICMP echo request, id 1, seq 25, length 40
18:48:53.265931 IP 192.168.20.1 > 192.168.20.103: ICMP echo reply, id 1, seq 25, length 40
18:48:54.267224 IP 192.168.20.103 > 192.168.20.1: ICMP echo request, id 1, seq 26, length 40
18:48:54.267245 IP 192.168.20.1 > 192.168.20.103: ICMP echo reply, id 1, seq 26, length 40

Is there anything I'm doing horribly wrong? I can manually enter a rule for 192.168.30.0/24 and have it work but why does GUEST net not work?

If it helps at all here are my interfaces:

BRIDGE0 is a bridge of MLX0 and MLX1, which is my connectx-4 card, but nothing is connected to it at the moment and I doubt this would affect anything on VLAN 20 or 30.

JohnKap

Do you have the Unifi AP configured in guest mode / isolation?

incertia

No everything is configured as a standard network. I experienced the exact same issue with wired clients plugged directly into the NIC on the pfSense machine.

Unifi settings here:

Gertjan

@incertia said in Network firewall rules not doing what I expect:

even within the same subnet

@incertia said in Network firewall rules not doing what I expect:

I experienced the exact same issue with wired clients plugged directly into the NIC on the pfSense machine.

Redo the tests.

When devices on a subnet all have an IP in the same /24 (your 92.168.30.0/24, so pfSense's DHCP did it's job) then you could even disconnect pfSense and still have "subnet" communication.

Btw :

You said :

but carefully hide the actual interface LAN name. I presume it's "GUEST"
GUEST is set up as 192.168.30.0/24 - as you told.

Consider again your second firewall rule.
and (example) :

18:44:47.122214 IP 192.168.30.100.63438 > 192.168.30.1.domain: 19580+ A? ap.spotify.com. (32)

The counters in front of the second rule don't count == do not match - so, no PASS.
This means that "192.168.30.100.63438" is not part of "GUEST net".
Or "192.168.30.100.63438" already blocked up front.

Btw : these :

18:44:47.122214 IP 192.168.30.100.63438 > 192.168.30.1.domain: 19580+ A? ap.spotify.com. (32)
18:44:49.356631 IP 192.168.30.100.64416 > 192.168.30.1.domain: 40732+ A? 48-courier.push.apple.com. (43)
18:44:49.384666 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 0, length 64
18:44:50.391544 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 1, length 64
18:44:50.555514 IP 192.168.30.100.51241 > 192.168.30.1.domain: 25649+ A? bag.itunes.apple.com. (38)
18:44:51.399414 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 2, length 64
18:44:52.116178 IP 192.168.30.100.63870 > 192.168.30.1.domain: 42418+ A? api.apple-cloudkit.com. (40)
18:44:52.116184 IP 192.168.30.100.50602 > 192.168.30.1.domain: 14258+ A? cl2.apple.com. (31)
18:44:52.389959 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 3, length 64
18:44:52.966850 IP 192.168.30.100.50602 > 192.168.30.1.domain: 14258+ A? cl2.apple.com. (31)
18:44:53.450264 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 4, length 64
18:44:54.494938 IP 192.168.30.100 > 192.168.30.1: ICMP echo request, id 37140, seq 5, length 64

are all packets coming from "192.168.30.100" == a device (?) to pfSense (== 192.168.30.100).
NOT

@incertia said in Network firewall rules not doing what I expect:

the traffic and pinging from pfSense to the device works as expected.

which is the other way around.
Can you check for incoming and outgoing traffic ?

I advise toy to :
Go basic first : undo VLAN's - undo bridging.
Plain old physical LAN's.

Then add one 'thing' at the time : A VLAN, and test.
undo the VLAN, a make a bridge, and test.
Still there ?
Add a bridged VLAN ** - and test.

A soon as things go broke, you know what not to do - and you're up and running.

** why would people actually do such things ?

incertia

@Gertjan said in Network firewall rules not doing what I expect:

but carefully hide the actual interface LAN name. I presume it's "GUEST"

It is indeed GUEST. Let us, for example, look at MGMT instead. Here is the interface:
MGMT
It has the following DHCP server:
DHCP
I have completely disabled any firewall rules for that interface:

@Gertjan said in Network firewall rules not doing what I expect:

Redo the tests.

Here, we see the tcpdump.

[2.4.4-RELEASE][root@pfsense.localdomain]/root: tcpdump -i igb1.1 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1.1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:16:15.869867 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1315, length 64
11:16:16.883375 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1316, length 64
11:16:17.896794 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1317, length 64
11:16:18.910131 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1318, length 64
11:16:19.923128 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1319, length 64
11:16:20.936858 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1320, length 64
11:16:21.950238 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1321, length 64
11:16:22.963285 IP 192.168.1.104 > 192.168.1.1: ICMP echo request, id 183, seq 1322, length 64
^C
8 packets captured
14 packets received by filter
0 packets dropped by kernel

As you can see there are no replies being sent. The same is true if we enable MGMT net to MGMT net. I have checked the firewall logs, it seems to be falling under the default IPv4 deny rule.
log
I have checked the pf rules, the second rule does not even appear.

[2.4.4-RELEASE][root@pfsense.localdomain]/root: cat /tmp/rules.debug | grep MGMT
MGMT = "{ igb1.1 }"
scrub on $MGMT all    fragment reassemble
antispoof log for $MGMT tracker 1000004720
# allow access to DHCP server on MGMT
pass in  quick on $MGMT proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000004741 label "allow access to DHCP server"
pass in  quick on $MGMT proto udp from any port = 68 to 192.168.1.1 port = 67 tracker 1000004742 label "allow access to DHCP server"
pass out  quick on $MGMT proto udp from 192.168.1.1 port = 67 to any port = 68 tracker 1000004743 label "allow access to DHCP server"
block return  in log  quick  on $MGMT inet from any to $pfB_PRI1_v4 tracker 1770009590  label "USER_RULE: pfB_PRI1_v4"
[2.4.4-RELEASE][root@pfsense.localdomain]/root: pfctl -sa | grep 'igb1\.1'
rdr pass on igb1.1 inet proto tcp from any to 10.10.10.1 port = http -> 127.0.0.1 port 8081
rdr pass on igb1.1 inet proto tcp from any to 10.10.10.1 port = https -> 127.0.0.1 port 9443
scrub on igb1.1 all fragment reassemble
block drop in log on ! igb1.1 inet from 192.168.1.0/24 to any
block drop in log on ! igb1.1 inet from 10.10.10.1 to any
block drop in log on igb1.1 inet6 from fe80::6eb3:11ff:fe1c:b299 to any
pass in quick on igb1.1 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server"
pass in quick on igb1.1 inet proto udp from any port = bootpc to 192.168.1.1 port = bootps keep state label "allow access to DHCP server"
pass out quick on igb1.1 inet proto udp from 192.168.1.1 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
block return in log quick on igb1.1 inet from any to <pfB_PRI1_v4> label "USER_RULE: pfB_PRI1_v4"

Even with pfctl -vvsa, grepping for the tracking ID yields no results.
Rules enabled, ping from pfSense to MGMT device:

[2.4.4-RELEASE][root@pfsense.localdomain]/root: tcpdump -i igb1.1 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1.1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:36:58.104312 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 15, length 64
11:36:58.105255 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 15, length 64
11:36:59.110783 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 16, length 64
11:36:59.111181 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 16, length 64
11:37:00.130726 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 17, length 64
11:37:00.131092 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 17, length 64
11:37:01.152722 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 18, length 64
11:37:01.153531 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 18, length 64
11:37:02.166115 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 19, length 64
11:37:02.166366 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 19, length 64
11:37:03.183370 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 20, length 64
11:37:03.183620 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 20, length 64
11:37:04.197545 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 21, length 64
11:37:04.198122 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 21, length 64
11:37:05.202766 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 22, length 64
11:37:05.203309 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 22, length 64
11:37:06.213229 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 23, length 64
11:37:06.214589 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 23, length 64
11:37:07.216295 IP 192.168.1.1 > 192.168.1.104: ICMP echo request, id 34357, seq 24, length 64
11:37:07.216718 IP 192.168.1.104 > 192.168.1.1: ICMP echo reply, id 34357, seq 24, length 64
^C
20 packets captured
21 packets received by filter
0 packets dropped by kernel

@Gertjan said in Network firewall rules not doing what I expect:

Plain old physical LAN's.

I enabled a regular LAN on igb1 with address 192.168.88.1/24 with DHCP enabled. I also connected my laptop directly via ethernet to the interface. After confirming a DHCP address, ping 192.168.88.1 -I <laptop interface> times out as well. It is the same issue as before.

Gertjan

@incertia said in Network firewall rules not doing what I expect:

onnected my laptop directly via ethernet to the interface

Why the -I parameter ?

ipconfig /all

or

ifconfig

tells what ? IP ok ? Gateway ok ? DNS ok ?

@incertia said in Network firewall rules not doing what I expect:

[2.4.4-RELEASE][root@pfsense.localdomain]/root: tcpdump -i igb1.1 -n icmp

Strange.
My turn.
I started this - em1 is my LAN interface :

[2.4.5-RC][admin@pfsense.brit-hotel-fumel.net]/root: tcpdump -i em1 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
.......

and on a Windows PC I started

ping 192.168.1.1.

This is what I saw on the pfSense console :

19:11:34.892523 IP 192.168.1.6 > 192.168.1.1: ICMP echo request, id 1, seq 43, length 40
19:11:34.892573 IP 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 1, seq 43, length 40
19:11:35.893070 IP 192.168.1.6 > 192.168.1.1: ICMP echo request, id 1, seq 44, length 40
19:11:35.893099 IP 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 1, seq 44, length 40
19:11:36.893243 IP 192.168.1.6 > 192.168.1.1: ICMP echo request, id 1, seq 45, length 40
19:11:36.893277 IP 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 1, seq 45, length 40
19:11:37.893290 IP 192.168.1.6 > 192.168.1.1: ICMP echo request, id 1, seq 46, length 40
19:11:37.893316 IP 192.168.1.1 > 192.168.1.6: ICMP echo reply, id 1, seq 46, length 40

I have a request (from the Windows device) and the reply (from pfSense).

Btw : "igb1.1" is a VLAN interface.

Is your LAN a VLAN ?
Test with 'Plain old physical LAN's' - no VLAN.

Another check :
Save your config - this permits you to get back into this "non workable" situation right away.
Reset pfSEnse to default.
Set up a WAN - if needed.
You wind up having one LAN - no VLAN's - nothing.
Do nothing else.

Do the ping test.
If no ok, ditch your system.
If ok, you know it's your setup .... (config).

incertia

Aha, I figured out why the MGMT net rules were not working. I had edited my config.xml and renamed my opt interfaces. After switching the naming convention back to opt1, opt2, ..., we achieve normal operation.