ip address silently blocked

lifeboy · Sep 9, 2021, 11:26 AM

pfSense latest stable version: 2.5.2-RELEASE (amd64)

I'm experiencing this since yesterday. Two separate pfSense installations at different locations.

Traffic from 102.218.142.79 is not allowed. The client gets a timeout.
If I turn the firewall off, i.e. just allow routing, then traffic is passed from that address. However, with the firewall on, nothing is allowed (ping, tcp, etc) but nothing it logged either.

The default deny rule logs denied traffic, but this address is not in the log.

I have also inspected all the tables (suspected virusprot), but nothing in there either on both firewalls.

The client at 102.218.142.79 can access other websites perfectly.

How do I find out what is blocking this address?

johnpoz · Sep 9, 2021, 11:31 AM

@lifeboy said in ip address silently blocked:

The client at 102.218.142.79 can access other websites perfectly.

Your using public IP space internally? Is that your space? It was assigned and routed to you?

What site(s) are not working.. If it was not logged by the default deny it was allowed.. Unless your running something like snort?

Do you show traffic in your state table, if it was allowed then state would of been created. Sniff on pfsense - do you see the traffic going out your wan? Maybe where your trying to go just doesn't want to talk to that IP..

Gertjan · Sep 9, 2021, 11:40 AM

@lifeboy said in ip address silently blocked:

How do I find out what is blocking this address?

By default, everything is blocked, and "102.218.142.79" isn't an exception.

I'm missing your NAT story.

Also, your WAN IP is still the same ?
And if so, does the traffic arrive on the WAN interface of pfSense ?
( use the Diagnostics >Packet Capture to find out )

lifeboy · Sep 9, 2021, 12:13 PM

@johnpoz, I'm definitely not using that address space. This is a client that used to connect to services running behind that firewalls in question.

If I sniff that traffic on the WAN, that address doesn't show up, so the traffic is not reaching WAN port according to the sniffer.

However (and this is the perplexing issue), if I disable the firewall, the traffic is routed / NAT'ed to the destination.

How can that be?

johnpoz · Sep 9, 2021, 1:01 PM

@lifeboy said in ip address silently blocked:

I'm definitely not using that address space.

What?? Dude how do you have a client behind pfsense with a public IP..

So that is an external IP trying to get to your resources? Your going to have to be a bit more clear on what is happening.. From your OP.. sounded like that was a client behind pfsense trying to go somewhere..

If that is some external IP out on the internet trying to hit your pfsense IP.. And you don't see it on a sniff of your wan.. Then NO its never going to be able to get to something behind pfsense..

lifeboy · Sep 9, 2021, 1:01 PM

@johnpoz said in ip address silently blocked:

@lifeboy said in ip address silently blocked:

I'm definitely not using that address space.

What?? Dude how do you have a client behind pfsense with a public IP..

So that is an external IP trying to get to your resources? Your going to have to be a bit more clear on what is happening.. From your OP.. sounded like that was a client behind pfsense trying to go somewhere..

Not sure why anyone would assume that a public IP is on the LAN side... but apologies for not stating that explicitely.

If that is some external IP out on the internet trying to hit your pfsense IP.. And you don't see it on a sniff of your wan.. Then NO its never going to be able to get to something behind pfsense..

Indeed. Let's just explore what happens with imcp packets.

When I ping the gateway of the WAN network from the client's machine that cannot reach any services on the inside of our firewalled network, the response is normal.
When I ping the firewall WAN address, I get no response (timeout)
When I ping that same address from anywhere else (I tried quite a few networks), I get a response (I have a rule allowing icmp)
There is no log entry of the failed ping, but the others (successful ones) are logged.
My default deny rule logs, so I expect that failed ping to be in there if it were blocked, but it's not
If I disable the firewall under https://<myfirewall>/system_advanced_firewall.php by ticking the "disable firewall" checkbox, the ping response is correct.

So I can only conclude that some built-in check blocks that address, but I can't find which one. Which pfSense services block traffic silently?

johnpoz · Sep 9, 2021, 1:14 PM

@lifeboy said in ip address silently blocked:

My default deny rule logs, so I expect that failed ping to be in there if it were blocked, but it's not

Its not going to be there if it never gets there!

Sniff on your wan when you do your ping test - do you see the ping get there?

The only way it would not be in the firewall log is it never gets there, or you have something like snort or something blocking it. So the firewall never actually saw it to say block or allow.

Other thing that could cause what your seeing is you have a rule blocking - set to not log. Do you have any rules in floating? Sniff is always best place to start - so you KNOW for sure if traffic is getting to pfsense or not. I would guess you have a floating rule blocking it set to not log. Maybe a pfblocker rule? Based on geoip or reputation or some other list your loading?

lifeboy · Sep 9, 2021, 1:19 PM

@johnpoz I sniff the WAN port, but it's doens't get there.

It's clear it doesn't get there. But if it gets there when I disable the firewall, then it's something the firewall does that blocks it, not so?

I don't have snort installed.

If it was one firewall doing this I'd suspect some rule or config, but 2 different ones makes me suspect it's something else.

johnpoz · Sep 9, 2021, 1:28 PM

@lifeboy said in ip address silently blocked:

I sniff the WAN port, but it's doens't get there.

Sniff has zero to do with firewall on or off.. If its not getting there, its not getting there - nothing pfsense can do - can not answer, can not forward if it doesn't see the traffic. Can not allow, can not block.

Sniff would see the traffic before it ever gets to the firewall. The traffic is viewed via a sniff before it would be passed up the stack to the firewall. It is via the interface and what the interface sees or puts on the wire.. In front of any firewall your running on that interface.

network -- interface - sniff -- firewall (pfsense)

As traffic enters the interface from the network - if your sniffing you would see traffic no matter what your firewall or any application up the stack would do with the traffic. Outbound, firewall could stop the traffic from being put on the network before the sniffer would see it.. But if you see it via the sniff - then it was put on the wire outbound, etc.

Gertjan · Sep 9, 2021, 1:58 PM

@lifeboy
Example of what @johnpoz said above :
When I snif on my WAN with these settings :

= I selected WAN, IPv4 and "ICMP"

I have a lot of results, and it tells me that 'something' locally is pinging some one on the net.
The something is the dpinger process that constantly pings a remote device so it can 'mesure' my uplink (WAN) connection.
This is a default configuration case.
The remote device is a host that I choose myself : one of my own web server, hosted somewhere on the Internet.

15:45:35.050123 IP 198.23.119.36 > 192.168.10.3: ICMP echo request, id 21026, seq 24371, length 8
15:45:35.050158 IP 192.168.10.3 > 198.23.119.36: ICMP echo reply, id 21026, seq 24371, length 8
15:45:35.164477 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46975, length 9
15:45:35.187540 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46975, length 9
15:45:35.666470 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46976, length 9
15:45:35.689982 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46976, length 9
15:45:36.167471 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46977, length 9
15:45:36.190547 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46977, length 9
15:45:36.669470 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46978, length 9
15:45:36.692864 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46978, length 9
15:45:37.170470 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46979, length 9
15:45:37.193307 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46979, length 9
15:45:37.437407 IP 169.45.119.36 > 192.168.10.3: ICMP echo request, id 2101, seq 48249, length 8
15:45:37.437426 IP 192.168.10.3 > 169.45.119.36: ICMP echo reply, id 2101, seq 48249, length 8
15:45:37.672470 IP 192.168.10.3 > 188.165.53.87: ICMP echo request, id 2264, seq 46980, length 9
15:45:37.694874 IP 188.165.53.87 > 192.168.10.3: ICMP echo reply, id 2264, seq 46980, length 9

198.23.119.36 = 24.77.17c6.ip4.static.sl-reverse.com : dono who that is, but he pinged 'me', and 'I' replied. He pings me a lot .....

"192.168.10.3" (192.168.10.1) is my upstream ISP router. 192.168.10.3 is my pfSense WAN IP, it's an RFC1918.

My ISP routes all traffic, everything, to 192.168.1.3, my pfSense WAN IP.

188.165.53.87 is the IP my dpinger process "WAN checker" is using as the gateway check device.

When I start to ping to my WAN IP from some remote location (a remote dedicated server) using 188.165.201.123, I see :

15:50:15.500120 IP 188.165.201.123 > 192.168.10.3: ICMP echo request, id 28246, seq 11, length 64
15:50:15.500132 IP 192.168.10.3 > 188.165.201.123: ICMP echo reply, id 28246, seq 11, length 64

I understand that you don't care about all this ping stuff. I'm just showing what I send on one side, and receive on the other side.

So :
Do you have an upstream router ?
What are the NAT rules you use ?

edit : Troubleshooting NAT Port Forwards

lifeboy · Sep 9, 2021, 2:15 PM

@Gertjan

I'm now contacting the client to run a ping from her side again to do another packet capture on the WAN port. I'll do that with the firewall enabled and disabled.

The fact is that she cannot even ping the WAN ip address, whereas the rest of the world apparently can.
When the firewall is disabled, she can ping the WAN port. Forget all the rest: The problem is essentially just that.

Can you ping 197.214.117.194 ? She can't (this is the one firewall)
Can you ping 197.214.119.130 ? Sha can't (this is the other firewall)

She can ping the first FW's gateway address, 197.214.117.193 as well as the second FW's gateway 197.214.119.129.

ALSO: This was not a problem yesterday, but today it suddenly is.

johnpoz · Sep 9, 2021, 2:28 PM

Your firewall being on or off would have zero to do with traffic getting to that IP...

In like 30 some years in the business - have never seen such a thing.. A sniff always happens before the firewall in the stack.. If not seeing it via sniff, its not getting there. Or you sniff is messed up - wrong filtering, wrong interface, etc.

lifeboy · Sep 9, 2021, 3:11 PM

@johnpoz said in ip address silently blocked:

Your firewall being on or off would have zero to do with traffic getting to that IP...

In like 30 some years in the business - have never seen such a thing..

neither have I in about as many years :-)

A sniff always happens before the firewall in the stack.. If not seeing it via sniff,
its not getting there. Or you sniff is messed up - wrong filtering, wrong
interface, etc.

Here's the pfSense packet capture with the firewall enabled:

16:54:40.059933 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32265, length 9
16:54:40.060261 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32265, length 9
16:54:40.561928 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32266, length 9
16:54:40.562254 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32266, length 9
16:54:41.063930 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32267, length 9
16:54:41.064256 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32267, length 9
16:54:41.565932 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32268, length 9
16:54:41.566259 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32268, length 9

And here it is with the firewall disabled:

16:59:30.846247 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32843, length 9
16:59:31.045781 IP 102.218.142.79 > 197.214.117.194: ICMP echo request, id 1, seq 167, length 40
16:59:31.045795 IP 197.214.117.194 > 102.218.142.79: ICMP echo reply, id 1, seq 167, length 40
16:59:31.347906 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32844, length 9
16:59:31.348233 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32844, length 9
16:59:31.849904 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32845, length 9
16:59:31.850229 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32845, length 9
16:59:32.050159 IP 102.218.142.79 > 197.214.117.194: ICMP echo request, id 1, seq 168, length 40
16:59:32.050168 IP 197.214.117.194 > 102.218.142.79: ICMP echo reply, id 1, seq 168, length 40
16:59:32.351906 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32846, length 9
16:59:32.352389 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32846, length 9

Here's a screenshot from the client, first part where the firewall is enabled, second part where it's disabled.

So clearly the firewall is blocking the icmp packets from that address 102.218.142.79.

But not from other addresses:
(with the firewall packet filter enabled)

johnpoz · Sep 9, 2021, 3:16 PM

@lifeboy said in ip address silently blocked:

So clearly the firewall is blocking the icmp packets from that address.

That is not what I see - how can firewall block what it doesn't see? Again if pfsense doesn't see the traffic how can it do anything with it.. The firewall being on or off has nothing to do with what traffic is seen by the interface.

Run your wan traffic through a switch/tap and mirror/span the port so you can sniff outside of pfsense. To see if traffic is actually put on the wire towards pfsense wan.

But have never on any device let alone pfsense ever see firewall have any effect on what your sniffing. Sniff is taken from the interface before the "firewall" even sees it to block or allow.

lifeboy · Sep 9, 2021, 3:34 PM

@johnpoz, fair enough, but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

Also, why is only that ip address getting a timeout, when every other address I have tried is passed by the filters and a response sent back to a ping?

Also, the on firewall is a physical metal (Sun X2100) machine, whereas the other is a VM on Proxmox. Both have the exact same behaviour.

I'll see what I can do to sniff the traffic on the metal box from the switch.

NogBadTheBad · Sep 9, 2021, 3:40 PM

@lifeboy Why are you seeing traffic from 197.214.117.193 going to 197.214.117.194 through the firewall, aren't they both on the same subnet past the pfSense interface?

johnpoz · Sep 9, 2021, 3:42 PM

@lifeboy said in ip address silently blocked:

but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

I don't - all I can say in 30 some years doing this stuff have never in my life seen a firewall have anyway to control what a sniff sees or doesn't see. If your sniff is not seeing it - then its not there.. Why its not there.. No idea. But nothing can happen with that traffic on pfsense if the traffic is not there.

197.214.117.193 going to 197.214.117.194 through the firewall,

That is not going through the firewall - he is sniffing on the wan.. And that is his dpinger pinging his gateway would be my take on that.. Better filtering on his sniff so he doesn't see that noise, and only sees the source of the pings would of been better idea. For all we know he didn't see the traffic because he filled up his 100 packets before pings even started ;)

lifeboy · Sep 9, 2021, 6:11 PM

@johnpoz said in ip address silently blocked:

@lifeboy said in ip address silently blocked:

but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

I don't - all I can say in 30 some years doing this stuff have never in my life seen a firewall have anyway to control what a sniff sees or doesn't see. If your sniff is not seeing it - then its not there.. Why its not there.. No idea. But nothing can happen with that traffic on pfsense if the traffic is not there.

Well, the traffic is there the moment I disable the packet filter. So the packet filter is somehow involved in this, that much is clear.

197.214.117.193 going to 197.214.117.194 through the firewall,

That is not going through the firewall - he is sniffing on the wan.. And that is his dpinger pinging his gateway would be my take on that.. Better filtering on his sniff so he doesn't see that noise, and only sees the source of the pings would of been
better idea.

Yes, it is dpinger. I didn't filter it out on purpose, simple because I want to see everthing for the duration of the capture, since I'm dealing with an unknown here.

For all we know he didn't see the traffic because he filled up his 100 packets before pings even started ;)

Nope, not the case. I first started the ping (with "ping -t"), then I started capturing and stopped it again. I then disabled the packet filter on pfSense, the ping started getting replies and I captured packets again. In the second capture, the address is present, in the first not. But I did say this all quite clearly and presented the details. It seems people don't read clearly?

My suspicion is that this is some regression that has been introduced in the 2.5.2. This address is pretty close to a bogon network range 102.218.0.0/17. Are bogon network packets just silently dropped?

As a test I have now disabled "Block bogon networks" on the WAN port and lo and behold, pings now are being responded to. As soon as I enable it again, they time out.

I suspect that instead of 102.218.0.0/17 being blocked, 102.218.0.0/16 is being blocked. I cannot find an authoritive list of bogon networks though. Can someone help with that?

lifeboy · Sep 9, 2021, 6:49 PM

To answer my own question: https://<firewall>/diag_tables.php and select bogon to view the list. In there 102.218.0.0/17 is listed, but no matches for 102.218.142.79, so there seems to be a buggy bogon filter.

johnpoz · Sep 9, 2021, 6:48 PM

@lifeboy said in ip address silently blocked:

there seems to be a buggy bogon filter.

And again even if was bogon - firewall is after sniff!! Nothing you can do in the firewall has anything to do with what you see on the interface via a sniff.. Just doesn't work that way..

Here
https://docs.netgate.com/pfsense/en/latest/diagnostics/packetcapture/index.html

"For incoming traffic, captures will show traffic that arrives on an interface on the firewall regardless of whether that traffic will be blocked by the firewall configuration"

If you say it works when not doing bogon - has your bogon be updated recently? Look in diagnostic tables for what cidrs are listed in bogon for pfsense..