RST package info - firewall

mislav

This test machine has no rules set, so I allow all outgoing traffic from it - for testing purposes. This connectivity is indeed done from WAN to WAN and tcdump is empty here.

This RST packages I mentioned that are "getting" through are from LAN traffic in pfsense environment. Here is the package flow:

on the pfsense I have multiple LAN interfaces with different networks - for this example I will try to provide you as much details as possible

there is network 10.0.20.0/24 - interface A on the pfsense (gateway 10.0.20.2)
there is network 10.0.30.0/24 - interface B on the pfsense (gateway 10.0.30.2)
static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A

My LAN machine in 10.0.30.0/24 network has IP 10.0.30.150 with default gateway set to pfsense - interface B - 10.0.30.2. So, when I request something on 10.0.40.0/24, it goes through that static route/interface A. From my point of view, this goes through different interface and rules shouldn't be bypassed, correct?

No. Time Source Destination Protocol Length Info
1 2017-09-29 11:59:48.906297 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 1: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
2 2017-09-29 11:59:48.906311 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 2: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
3 2017-09-29 11:59:48.909711 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 3: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
4 2017-09-29 11:59:48.909770 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 4: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

There is no rule in pfsense to allow this, but on the firewall 10.0.40.1 we are able to see this RST packages. I also tried to capture this by setting firewall rule to match my source IP to destination with flags set to any TCP flag and nothing. Also tcpdump is empty there.

johnpoz

"- static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A"

What?

Please draw up this network.. You said wan to wan over public.. But then your not showing anything like that..

So your 10.0.40 is downstream of 10.0.20.. And you have some client sending packets to 10.0.20 to be looped back to 10.0.40?

mislav

@johnpoz:

"- static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A"

What?

Please draw up this network.. You said wan to wan over public.. But then your not showing anything like that..

So your 10.0.40 is downstream of 10.0.20.. And you have some client sending packets to 10.0.20 to be looped back to 10.0.40?

Network is very complex there to draw and you won't understand what is what. The problem is that RST is not logged on the pfsense and although it should be if you set such firewall rule on the pfsense - to log all traffic with all TCP flags. Correct?

There is a additional router between this 10.0.20.0/24 and 10.0.40.0/24 which routes this traffic and enable us to even connect there. pfsense doesn't see this traffic by default, that's why static route is there. Without that route, my test machine wouldn't see this network.

I might have confused you a bit with this, I went too much into details with our environment. Imagine any combination you like with local LANs and local interfaces - the main point is that traffic in every scenario will go through pfsense - because pfsense is always a default gateway for all machines. There is no direct access between machines, no direct connectivity from WAN or to the internet - all internal LAN traffic, incoming/outgoing will always go through pfsense (pfsense has multiple interfaces for every LAN network).

Derelict

Network is very complex there to draw and you won't understand what is what. The problem is that RST is not logged on the pfsense and although it should be if you set such firewall rule on the pfsense - to log all traffic with all TCP flags. Correct?

There is a additional router between this 10.0.20.0/24 and 10.0.40.0/24 which routes this traffic and enable us to even connect there. pfsense doesn't see this traffic by default, that's why static route is there. Without that route, my test machine wouldn't see this network.

I might have confused you a bit with this, I went too much into details with our environment. Imagine any combination you like with local LANs and local interfaces - the main point is that traffic in every scenario will go through pfsense - because pfsense is always a default gateway for all machines. There is no direct access between machines, no direct connectivity from WAN or to the internet - all internal LAN traffic, incoming/outgoing will always go through pfsense (pfsense has multiple interfaces for every LAN network).

So don't test principles like this on your "very complex" network that "confuses" us.

Test it with a simple network in a simple environment (maybe a virtual environment) that you can easily diagram so our simple brains can understand.

johnpoz

hehehehehe - oh fantastic Derelict, you been hanging with dok? ;)

Yeah downstream router and static routes - those can be confusing ;) hehehe

Here is a simple test for you.. Take a box on 10.0.20 and send your RST traffic to another box on 10.0.30, both hanging off pfsense. Sniffing on this 10.0.30 box do you see these RST packets? That you sent from your 20.x box? If so what rule do you have on the 20 interface that allows traffic it is a default any any? Did you set something funny in the flags on the rule? Do you have some rule in floating that would allow it that has something set for flags?

Is 20 and 30 on the same physical interface vis a hairpin to from some downstream router, and you set the do not evaluate rules on traffic that enters and leaves the same interface via a transit network..

Use some crayons to draw this up with so we can understand it ;)

mislav

Thank you all for having patience with me on this one, I really appreciate it.

Latest changes did something and now I've no idea what is going on anymore there. I went to "System-Advanced-Firewall & NAT" and I disabled option under - Static route filtering - Bypass firewall rules for traffic on the same interface.

Now, after disabling this, I was able to see this RST traffic with both tcdump and directly inside pfsense (Status ->System Logs ->Firewall -> Normal View) - and now it makes no difference whether I disable or enable this rule under advanced setting - I notice now this traffic, like this change triggered something that was not there before (as per your suggestion - traffic was done directly from 30 to 20, no routers or anything between, direct communication between LAN machine and pfsense). I even notice now this RST packages as blocked traffic even in my complicated setup with gateways/routes I was trying to explain.

So, I'm confused now! One thing that I noticed is messages in general log "promiscuous mode enabled/disabled" - I don't think was seeing this before, could this effect somehow?

johnpoz

"Bypass firewall rules for traffic on the same interface."

So 20 and 30 are vlans on the same physical interface I take it.. Or your coming in an interface just to hairpin out of it to get to a downstream router? Why wold you have set that checkbox? If you don't understand what it does? That is only there for BS setups that don't use transits, fix asymmetrical messes, etc.

mislav

Could it be that it's set by default? If not, then I probably set it for unknown reason few years back when I was configuring this. Meanwhile new interfaces were added and everybody forgot about this option. The server is in "cloud" and I have around 6-7 NICs added, so every interface is like em0, em1, em2….etc (or eth0, however you prefer) - they're not added like 1 interface and then eth0, eth0:0, eth0:1, eth0:2 - this is not the case. But anyway, I'm start seeing this RST packages inside pfsense now, so I guess this did the trick anyway.
Thank you very much for the patience and guidance!

Derelict

No. It is not set by default. Ever.

mislav

Thank you!