Some traffic between VLANs blocked after upgrade to 2.8.0/1

Fredish

Hi,
I have an issue with blocked traffic after update to 2.8.0. Before the update everything worked fine. I tried to restore from the backup that I did before updating. Still problem. Added allow all rule on all three vlan:s, still blocked traffic.

Setup
Storage server, vlan 20
Virtual server, vlan 1720
Hypervisor with the virtual server, vlan 101

Traffic
Source: virtual server
Destination: storage server
Port: 2049

Allow rule vlan 1720
Allow IPv4, TCP with source virtual server. Destination storage server, port 2049

Blocked traffic information:
Action: block
Reason: match
Tracker ID: 1000003570
Matched Rule:
@39 block drop in log on ! em0.20 inet from 10.1.20.0/24 to any ridentifier 1000003570
Associated Rules:
@40 block drop in log inet from 10.1.20.1 to any ridentifier 1000003570
@41 block drop in log on em0.20 inet6 from fe80::6a05:dead:dead:dead to any ridentifier 1000003570
(I edited the ipv6 address before posting here)

Any ideas what might happened and how to fix it?

Thank you in advance.

Br,
Fred

johnpoz

@Fredish looks like your seeing traffic on the wrong interface.

that rule 39 says its blocking traffic in on 20 because source is not (!) from the 20 network 10.1.20..

If you allow 1720 to talk to 20, then the state would allow the return from 20 to 1720.

Even with no states, how would a source of non vlan 20 traffic be hitting your vlan 20 interface. The only way that could happen is if vlan 20 was a transit network and you had downstream networks using that transit network. If that was the case, a rule of not (!) vlan 20 shouldn't be on the 20 interface.

Are these rules on the actual interfaces - or in floating?

An actual screenshot of your rules would be easier to read.

Added allow all rule on all three vlan:s, still blocked traffic.

Pointless - when traffic is a allowed on the source of the incoming interface, the return traffic is allowed by the state. The rules are not even evaluated if a state matches.

Fredish

Hi John,
Thank you for your answer. As far as I know, I don't have that rule 39, that's one thing that is confusing me.

This is the rule in 1720 (vm)

Any other rule I should post?

I added this as a test, but still the same block.

Regarding the transit network, I'm not sure. I have the pfsense computer connected to a switch and using different vlans to separate the traffic.

I have no rules in Floating, they are all in the interfaces.

I find it interesting that pfsense seems to be confused about what interfaces it use.

Storage server, vlan 20 = 10.1.20.0/24
Virtual server, vlan 1720 = 172.16.0.0/24
Hypervisor with the virtual server, vlan 101 = 10.100.101.0/24

Best regards,
Fred

johnpoz

@Fredish that SA is a syn,ack - which screams asymmetrical traffic. IE the return traffic was seen on different interface than the state was created on when traffic was allowed.

As to screen, I mean of the rules on the interface.. Like this for example.

SteveITS

@Fredish the above, and you might look into the state policy changes.

https://docs.netgate.com/pfsense/en/latest/releases/2-8-0.html#general

Fredish

Hi @SteveITS,
Thank you for directing me to the solution. I changed it to "Floating States" and it started to work. Now I need to learn more about that and how I should manage it in the future.

Best regards,
Fredrik

PS: If you have any pointers on a good read for that, it would be appreciated.

johnpoz

@Fredish I don't use floating states and have no issues.

I have multiple vlans setup.. If I were to guess you have some sort of asymmetrical flow going on that floating is allowing.. If me I would look to why that is and fix the asymmetrical flow. Unless your doing something that would require the relaxed setting. Like the HA example given in the note.

Fredish

Hi @johnpoz,
I agree 100% with you. I need to learn more about this so I can prevent further issues with it. Thank you for your assistance!

I have 11 different VLANs for different use cases so I need to be on top of this :)

Best regards,
Fredrik

PS: If you have any good reading recommendations for this, please feel free to point me in their direction.

johnpoz

@Fredish well where I would look is that rule @39 you didn't create.. So if I look in my rules. I don't show that specific number but same rid I see this rule. I am not using floating states.

[24.11-RELEASE][admin@sg4860.home.arpa]/var/etc: cat /tmp/rules.debug | grep 1000003570
antispoof  for $WLAN ridentifier 1000003570

If I then track that down with pfctl -vvsr I find rules like this.. here is some for vlans

@98 block drop in on ! igb2.4 inet from 192.168.4.0/24 to any ridentifier 1000007770
@104 block drop in on ! igb2.6 inet from 192.168.6.0/24 to any ridentifier 1000008820
@110 block drop in on ! igb4.1011 inet from 10.1.1.0/24 to any ridentifier 1000009870

Which are exactly like the rule you showed your traffic matching on with the ! (not) statement and the network on that vlan.

Here is the one for the rule that machines the same rid as yours

@81 block drop in on ! igb2 inet from 192.168.2.0/24 to any ridentifier 1000003570

So from what you posted.. Like I said before you have source traffic hitting your vlan 20 interface that is not from the 10.1.20 network - in correctly isolated vlans this should never be possible. Could be issue with multihomed device sending its return traffic out different interface. Could be tagging issue in your switching infrastructure be that physical switching or virtual switches/port groups, etc.

But normally it should not be possible for some different source IP to be inbound into a pfsense interface on a different network. With floating states pfsense is allowing this, when to be honest it shouldn't really - unless for some reason as they mention you have a asymmetrical because you have to for some reason??

The only time you should see different source IPs on an interface would be on a transit/connector network that you are routing downstream networks through.

So I would look there - why is that traffic hitting your em0.20 interface when the source IP is not in the 10.1.20 network?

if I were to guess I would assume something wrong in your virtual switching setup where tags are not being handled correctly.