RST package info - firewall

jimp

RST is a response packet. Log entries are only created for the first packet of a connection (e.g. TCP SYN), so if you get multiple RST back for a valid SYN, it can't be logged.

That said, if someone were to spontaneously send you an RST without an existing state, that would be logged. Assuming you didn't have a rule to pass it (e.g. one set to pass with TCP flags set to 'any')

Harvy66

I find it interesting that he claiming pfSense is passing these RST packets. To me it sounds like some important information about what is happening is being left out. pfSense normally blocks out-of-state packets unless something was changed.

johnpoz

Yeah.. Any stateful firewall would normally block out of state packets. Unless something was, once pfsense saw a RST for a state, the state would be closed. Any other RST to that some would be block, and out of the box logged.

So unless something was altered to pass like jimp mentioned already.

Like to see actual sniff of this traffic so you could tell from which mac this traffic was coming from on the L2 network.

mislav

Hi. Thanks for the answers.

@jimp - I did send RST without existing state and this was not logged, even though there exists firewall rule that is matching my IP and TCP flags is set to any. (I've tried with protocol set to ANY and protocol set only to TCP - of course, with TCP logs set to any).

Is this what you were wondering:

from my local PC I do e.g. this (public IP is just pfsense public IP)

hping3 –rst -p 40001 PUBLIC_IP -c 10

HPING PUBLIC_IP (eth0 PUBLIC_IP): R set, 40 headers + 0 data bytes

--- PUBLIC_IP hping statistic ---
10 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

This is tcpdump from my local PC:

tcpdump -vvv -nn -s 0 'dst host PUBLIC_IP and dst port 40001'

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:36:09.645081 IP (tos 0x0, ttl 64, id 4003, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1592 > PUBLIC_IP.40001: Flags [R], cksum 0x34ed (correct), seq 224770321, win 512, length 0
11:36:10.645209 IP (tos 0x0, ttl 64, id 19166, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1593 > PUBLIC_IP.40001: Flags [R], cksum 0x99e6 (correct), seq 1423098205, win 512, length 0
11:36:11.645306 IP (tos 0x0, ttl 64, id 35851, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1594 > PUBLIC_IP.40001: Flags [R], cksum 0x69cd (correct), seq 670192973, win 512, length 0
11:36:12.645384 IP (tos 0x0, ttl 64, id 3872, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1595 > PUBLIC_IP.40001: Flags [R], cksum 0x3eed (correct), seq 1701932750, win 512, length 0
11:36:13.645457 IP (tos 0x0, ttl 64, id 55838, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1596 > PUBLIC_IP.40001: Flags [R], cksum 0x1310 (correct), seq 94386705, win 512, length 0
11:36:14.645533 IP (tos 0x0, ttl 64, id 52380, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1597 > PUBLIC_IP.40001: Flags [R], cksum 0xc76e (correct), seq 1068820496, win 512, length 0
11:36:15.645655 IP (tos 0x0, ttl 64, id 9012, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1598 > PUBLIC_IP.40001: Flags [R], cksum 0xa4fc (correct), seq 1610255996, win 512, length 0
11:36:16.645747 IP (tos 0x0, ttl 64, id 5257, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1599 > PUBLIC_IP.40001: Flags [R], cksum 0x5f20 (correct), seq 1938617573, win 512, length 0
11:36:17.645829 IP (tos 0x0, ttl 64, id 33973, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1600 > PUBLIC_IP.40001: Flags [R], cksum 0x6e46 (correct), seq 498511497, win 512, length 0
11:36:18.645912 IP (tos 0x0, ttl 64, id 19401, offset 0, flags [none], proto TCP (6), length 40)
192.168.0.197.1601 > PUBLIC_IP.40001: Flags [R], cksum 0x60cf (correct), seq 1437471741, win 512, length 0

I did same thing on pfsense but pfsense doesn't log anything on port 40001, nor do I see something in the firewall logs.

When I do SYN, this gets logged of course. on both ends.

johnpoz

So you saw the traffic on pfsense via tcpdump?

Harvy66

The default for the LAN is to let the packets through. Just because you're sending the packets to the WAN IP does not mean they're hitting the WAN interface. Try doing the same thing from the WAN side.

johnpoz

Try your test from WAN side to wan..

I was looking for the sniff on the box showing all the RST that you said were getting through pfsense. I would like to see the sniff showing them and that they came from pfsense mac address on your L2 network.

mislav

@johnpoz:

So you saw the traffic on pfsense via tcpdump?

No, I didn't.

This test was done from "my LAN" to WAN, it was done from my LAN - 192.168.0.197 as seen there - this is my local IP in the office which translates to public IP internet provider give me - and I do test to WAN IP of pfsense. So connection is made WAN to WAN - pfsense is not in the office, but in datacenter, different country.

johnpoz

"and I do test to WAN IP of pfsense. So connection is made WAN to WAN - pfsense is not in the office, but in datacenter, different country."

And so you saw these RST you sent all the way on pfsense wan via tcpdump? Your local firewall should not have let them even out!! So how did they get to pfsense?

Why don't you show us the sniff on pfsense the RST getting there.. And then it not logging them, and as you say letting them through..

mislav

This test machine has no rules set, so I allow all outgoing traffic from it - for testing purposes. This connectivity is indeed done from WAN to WAN and tcdump is empty here.

This RST packages I mentioned that are "getting" through are from LAN traffic in pfsense environment. Here is the package flow:

on the pfsense I have multiple LAN interfaces with different networks - for this example I will try to provide you as much details as possible

there is network 10.0.20.0/24 - interface A on the pfsense (gateway 10.0.20.2)
there is network 10.0.30.0/24 - interface B on the pfsense (gateway 10.0.30.2)
static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A

My LAN machine in 10.0.30.0/24 network has IP 10.0.30.150 with default gateway set to pfsense - interface B - 10.0.30.2. So, when I request something on 10.0.40.0/24, it goes through that static route/interface A. From my point of view, this goes through different interface and rules shouldn't be bypassed, correct?

No. Time Source Destination Protocol Length Info
1 2017-09-29 11:59:48.906297 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 1: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
2 2017-09-29 11:59:48.906311 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 2: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
3 2017-09-29 11:59:48.909711 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 3: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

No. Time Source Destination Protocol Length Info
4 2017-09-29 11:59:48.909770 10.0.30.150 10.0.40.2 TCP 54 443→15863 [RST] Seq=1 Win=0 Len=0

Frame 4: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Vmware_01:2b:2c (00:50:56:01:2b:2c), Dst: Vmware_01:09:86 (00:50:56:01:09:86)
Internet Protocol Version 4, Src: 10.0.30.150 (10.0.30.150), Dst: 10.0.40.2 (10.0.40.2)
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 15863 (15863), Seq: 1, Len: 0

There is no rule in pfsense to allow this, but on the firewall 10.0.40.1 we are able to see this RST packages. I also tried to capture this by setting firewall rule to match my source IP to destination with flags set to any TCP flag and nothing. Also tcpdump is empty there.

johnpoz

"- static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A"

What?

Please draw up this network.. You said wan to wan over public.. But then your not showing anything like that..

So your 10.0.40 is downstream of 10.0.20.. And you have some client sending packets to 10.0.20 to be looped back to 10.0.40?

mislav

@johnpoz:

"- static route is defined to connect additional local network 10.0.40.0/24 - gateway is interface A"

What?

Please draw up this network.. You said wan to wan over public.. But then your not showing anything like that..

So your 10.0.40 is downstream of 10.0.20.. And you have some client sending packets to 10.0.20 to be looped back to 10.0.40?

Network is very complex there to draw and you won't understand what is what. The problem is that RST is not logged on the pfsense and although it should be if you set such firewall rule on the pfsense - to log all traffic with all TCP flags. Correct?

There is a additional router between this 10.0.20.0/24 and 10.0.40.0/24 which routes this traffic and enable us to even connect there. pfsense doesn't see this traffic by default, that's why static route is there. Without that route, my test machine wouldn't see this network.

I might have confused you a bit with this, I went too much into details with our environment. Imagine any combination you like with local LANs and local interfaces - the main point is that traffic in every scenario will go through pfsense - because pfsense is always a default gateway for all machines. There is no direct access between machines, no direct connectivity from WAN or to the internet - all internal LAN traffic, incoming/outgoing will always go through pfsense (pfsense has multiple interfaces for every LAN network).

Derelict

Network is very complex there to draw and you won't understand what is what. The problem is that RST is not logged on the pfsense and although it should be if you set such firewall rule on the pfsense - to log all traffic with all TCP flags. Correct?

There is a additional router between this 10.0.20.0/24 and 10.0.40.0/24 which routes this traffic and enable us to even connect there. pfsense doesn't see this traffic by default, that's why static route is there. Without that route, my test machine wouldn't see this network.

I might have confused you a bit with this, I went too much into details with our environment. Imagine any combination you like with local LANs and local interfaces - the main point is that traffic in every scenario will go through pfsense - because pfsense is always a default gateway for all machines. There is no direct access between machines, no direct connectivity from WAN or to the internet - all internal LAN traffic, incoming/outgoing will always go through pfsense (pfsense has multiple interfaces for every LAN network).

So don't test principles like this on your "very complex" network that "confuses" us.

Test it with a simple network in a simple environment (maybe a virtual environment) that you can easily diagram so our simple brains can understand.

johnpoz

hehehehehe - oh fantastic Derelict, you been hanging with dok? ;)

Yeah downstream router and static routes - those can be confusing ;) hehehe

Here is a simple test for you.. Take a box on 10.0.20 and send your RST traffic to another box on 10.0.30, both hanging off pfsense. Sniffing on this 10.0.30 box do you see these RST packets? That you sent from your 20.x box? If so what rule do you have on the 20 interface that allows traffic it is a default any any? Did you set something funny in the flags on the rule? Do you have some rule in floating that would allow it that has something set for flags?

Is 20 and 30 on the same physical interface vis a hairpin to from some downstream router, and you set the do not evaluate rules on traffic that enters and leaves the same interface via a transit network..

Use some crayons to draw this up with so we can understand it ;)

mislav

Thank you all for having patience with me on this one, I really appreciate it.

Latest changes did something and now I've no idea what is going on anymore there. I went to "System-Advanced-Firewall & NAT" and I disabled option under - Static route filtering - Bypass firewall rules for traffic on the same interface.

Now, after disabling this, I was able to see this RST traffic with both tcdump and directly inside pfsense (Status ->System Logs ->Firewall -> Normal View) - and now it makes no difference whether I disable or enable this rule under advanced setting - I notice now this traffic, like this change triggered something that was not there before (as per your suggestion - traffic was done directly from 30 to 20, no routers or anything between, direct communication between LAN machine and pfsense). I even notice now this RST packages as blocked traffic even in my complicated setup with gateways/routes I was trying to explain.

So, I'm confused now! One thing that I noticed is messages in general log "promiscuous mode enabled/disabled" - I don't think was seeing this before, could this effect somehow?

johnpoz

"Bypass firewall rules for traffic on the same interface."

So 20 and 30 are vlans on the same physical interface I take it.. Or your coming in an interface just to hairpin out of it to get to a downstream router? Why wold you have set that checkbox? If you don't understand what it does? That is only there for BS setups that don't use transits, fix asymmetrical messes, etc.

mislav

Could it be that it's set by default? If not, then I probably set it for unknown reason few years back when I was configuring this. Meanwhile new interfaces were added and everybody forgot about this option. The server is in "cloud" and I have around 6-7 NICs added, so every interface is like em0, em1, em2….etc (or eth0, however you prefer) - they're not added like 1 interface and then eth0, eth0:0, eth0:1, eth0:2 - this is not the case. But anyway, I'm start seeing this RST packages inside pfsense now, so I guess this did the trick anyway.
Thank you very much for the patience and guidance!

Derelict

No. It is not set by default. Ever.

mislav

Thank you!