ip address silently blocked

lifeboy

I'm now contacting the client to run a ping from her side again to do another packet capture on the WAN port. I'll do that with the firewall enabled and disabled.

The fact is that she cannot even ping the WAN ip address, whereas the rest of the world apparently can.
When the firewall is disabled, she can ping the WAN port. Forget all the rest: The problem is essentially just that.

Can you ping 197.214.117.194 ? She can't (this is the one firewall)
Can you ping 197.214.119.130 ? Sha can't (this is the other firewall)

She can ping the first FW's gateway address, 197.214.117.193 as well as the second FW's gateway 197.214.119.129.

ALSO: This was not a problem yesterday, but today it suddenly is.

johnpoz

Your firewall being on or off would have zero to do with traffic getting to that IP...

In like 30 some years in the business - have never seen such a thing.. A sniff always happens before the firewall in the stack.. If not seeing it via sniff, its not getting there. Or you sniff is messed up - wrong filtering, wrong interface, etc.

lifeboy

@johnpoz said in ip address silently blocked:

Your firewall being on or off would have zero to do with traffic getting to that IP...

In like 30 some years in the business - have never seen such a thing..

neither have I in about as many years :-)

A sniff always happens before the firewall in the stack.. If not seeing it via sniff,
its not getting there. Or you sniff is messed up - wrong filtering, wrong
interface, etc.

Here's the pfSense packet capture with the firewall enabled:

16:54:40.059933 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32265, length 9
16:54:40.060261 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32265, length 9
16:54:40.561928 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32266, length 9
16:54:40.562254 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32266, length 9
16:54:41.063930 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32267, length 9
16:54:41.064256 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32267, length 9
16:54:41.565932 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32268, length 9
16:54:41.566259 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32268, length 9

And here it is with the firewall disabled:

16:59:30.846247 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32843, length 9
16:59:31.045781 IP 102.218.142.79 > 197.214.117.194: ICMP echo request, id 1, seq 167, length 40
16:59:31.045795 IP 197.214.117.194 > 102.218.142.79: ICMP echo reply, id 1, seq 167, length 40
16:59:31.347906 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32844, length 9
16:59:31.348233 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32844, length 9
16:59:31.849904 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32845, length 9
16:59:31.850229 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32845, length 9
16:59:32.050159 IP 102.218.142.79 > 197.214.117.194: ICMP echo request, id 1, seq 168, length 40
16:59:32.050168 IP 197.214.117.194 > 102.218.142.79: ICMP echo reply, id 1, seq 168, length 40
16:59:32.351906 IP 197.214.117.194 > 197.214.117.193: ICMP echo request, id 36805, seq 32846, length 9
16:59:32.352389 IP 197.214.117.193 > 197.214.117.194: ICMP echo reply, id 36805, seq 32846, length 9

Here's a screenshot from the client, first part where the firewall is enabled, second part where it's disabled.

So clearly the firewall is blocking the icmp packets from that address 102.218.142.79.

But not from other addresses:
(with the firewall packet filter enabled)

johnpoz

@lifeboy said in ip address silently blocked:

So clearly the firewall is blocking the icmp packets from that address.

That is not what I see - how can firewall block what it doesn't see? Again if pfsense doesn't see the traffic how can it do anything with it.. The firewall being on or off has nothing to do with what traffic is seen by the interface.

Run your wan traffic through a switch/tap and mirror/span the port so you can sniff outside of pfsense. To see if traffic is actually put on the wire towards pfsense wan.

But have never on any device let alone pfsense ever see firewall have any effect on what your sniffing. Sniff is taken from the interface before the "firewall" even sees it to block or allow.

lifeboy

@johnpoz, fair enough, but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

Also, why is only that ip address getting a timeout, when every other address I have tried is passed by the filters and a response sent back to a ping?

Also, the on firewall is a physical metal (Sun X2100) machine, whereas the other is a VM on Proxmox. Both have the exact same behaviour.

I'll see what I can do to sniff the traffic on the metal box from the switch.

NogBadTheBad

@lifeboy Why are you seeing traffic from 197.214.117.193 going to 197.214.117.194 through the firewall, aren't they both on the same subnet past the pfSense interface?

Screenshot 2021-09-09 at 16.31.35.png

johnpoz

@lifeboy said in ip address silently blocked:

but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

I don't - all I can say in 30 some years doing this stuff have never in my life seen a firewall have anyway to control what a sniff sees or doesn't see. If your sniff is not seeing it - then its not there.. Why its not there.. No idea. But nothing can happen with that traffic on pfsense if the traffic is not there.

197.214.117.193 going to 197.214.117.194 through the firewall,

That is not going through the firewall - he is sniffing on the wan.. And that is his dpinger pinging his gateway would be my take on that.. Better filtering on his sniff so he doesn't see that noise, and only sees the source of the pings would of been better idea. For all we know he didn't see the traffic because he filled up his 100 packets before pings even started ;)

lifeboy

@johnpoz said in ip address silently blocked:

@lifeboy said in ip address silently blocked:

but how do you explain that the minute the firewall packet filter is disabled, the traffic is seen and a ping returned?

I don't - all I can say in 30 some years doing this stuff have never in my life seen a firewall have anyway to control what a sniff sees or doesn't see. If your sniff is not seeing it - then its not there.. Why its not there.. No idea. But nothing can happen with that traffic on pfsense if the traffic is not there.

Well, the traffic is there the moment I disable the packet filter. So the packet filter is somehow involved in this, that much is clear.

197.214.117.193 going to 197.214.117.194 through the firewall,

That is not going through the firewall - he is sniffing on the wan.. And that is his dpinger pinging his gateway would be my take on that.. Better filtering on his sniff so he doesn't see that noise, and only sees the source of the pings would of been
better idea.

Yes, it is dpinger. I didn't filter it out on purpose, simple because I want to see everthing for the duration of the capture, since I'm dealing with an unknown here.

For all we know he didn't see the traffic because he filled up his 100 packets before pings even started ;)

Nope, not the case. I first started the ping (with "ping -t"), then I started capturing and stopped it again. I then disabled the packet filter on pfSense, the ping started getting replies and I captured packets again. In the second capture, the address is present, in the first not. But I did say this all quite clearly and presented the details. It seems people don't read clearly?

My suspicion is that this is some regression that has been introduced in the 2.5.2. This address is pretty close to a bogon network range 102.218.0.0/17. Are bogon network packets just silently dropped?

As a test I have now disabled "Block bogon networks" on the WAN port and lo and behold, pings now are being responded to. As soon as I enable it again, they time out.

I suspect that instead of 102.218.0.0/17 being blocked, 102.218.0.0/16 is being blocked. I cannot find an authoritive list of bogon networks though. Can someone help with that?

lifeboy

To answer my own question: https://<firewall>/diag_tables.php and select bogon to view the list. In there 102.218.0.0/17 is listed, but no matches for 102.218.142.79, so there seems to be a buggy bogon filter.

johnpoz

@lifeboy said in ip address silently blocked:

there seems to be a buggy bogon filter.

And again even if was bogon - firewall is after sniff!! Nothing you can do in the firewall has anything to do with what you see on the interface via a sniff.. Just doesn't work that way..

Here
https://docs.netgate.com/pfsense/en/latest/diagnostics/packetcapture/index.html

"For incoming traffic, captures will show traffic that arrives on an interface on the firewall regardless of whether that traffic will be blocked by the firewall configuration"

alt text

If you say it works when not doing bogon - has your bogon be updated recently? Look in diagnostic tables for what cidrs are listed in bogon for pfsense..

lifeboy

@johnpoz, look I really don't care about the semantics of what you're explaining (which is not wrong) ...

The fact is that when I turn bogon filtering off, the packets are allowed, and when I turn it back on, the packets silently are blocked. pfSense does that. It has nothing to do with how or whether I sniff the traffic or not.

So, how to do file a bug report?

johnpoz

So did you validate what is in pfsense bogon table, look under diag, tables bogon. maybe yours is old..

I don't see that listed..

Should show you when last updated

lifeboy

@johnpoz, I explained in detail in my previous reply that I did interrogate the list and didn't find a match. I even speculated that it the /17 might be a filtered as a /16, so again: Why are why going in circles?

bogon filter on: nothing is "seen" by pfSense' packet capture tool. pings time out.
bogon filter off: packets are recording by the packet filter. pings get a reply.

Conclusion: There's a problem with the bogon filter.

Clear enough?

Please don't be obnoxious about this, even if you're a global moderator.

johnpoz

@lifeboy I just PM you my IP, have that IP ping my IP.. I have bogon blocked, I have it set to log and I allow ping. And running a sniff. If can show its being blocked more than happy to help you fill out a bug report.

johnpoz

Ok.. Now we are getting somewhere with info we can work with for bug report.

From my quick look at my bogon table - that should not be blocked. But clearly it is being blocked..

But also seeing it via sniff - so have no idea what going on why your not seeing sniff..

edit: Ok it should be blocked by bogon - with this entry
102.218.128.0/19
102.218.128.0 - 102.218.159.255

So need to see if bogon is just not updated..

https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt

102.218.0.0/18
102.218.64.0/19
102.218.96.0/21
102.218.104.0/22
102.218.128.0/23
102.218.131.0/24
102.219.56.0/22
102.221.132.0/22
102.221.136.0/21
102.223.180.0/22
102.223.184.0/21

edit: Doh I can not read this afternoon - looks like updated list it is removed. That /19 changed to a /23

lifeboy

@johnpoz I just had a look at my log setting and see that "Log packets blocked by 'Block Bogon Networks' rules" is not turned on, which explains why I didn't see them in the log. I see the bogon block log now.

johnpoz

@lifeboy From what I can tell its still listed as bogon.. So yeah not really a bug of any sort.. Who ever the ISP is that is using that IP needs to get it removed off bogon list.

lifeboy

@johnpoz You're quite right, 102.218.128.0/19 does include that ip.

I'll report it to the ISP.

Thanks for your effort and assistance. Much appreciated!

johnpoz

@lifeboy Wait... I had a misread - I don't see it listed.. The updated list seems to be good. Just need to get pfsense list to update.. Mine hasn't been updated since aug 20th

well hmmm - that doesn't seem to be working. Still showing aug20th

edit: OK - problem is the bogon list from pfsense/netgate is dated..

https://files.netgate.com/lists/fullbogons-ipv4.txt
last updated 1629435001 (Fri Aug 20 04:50:01 2021 GMT)

We can file a bug report on that.. Or can just flag maybe @stephenw10 or @Derelict and maybe they can get the bogon list that is hosted by pfsense updated.. Since I do not show that 102 IP address on the actual current list.

edit: Glad we got to the bottom of this ;) Sniffing sent us on a wild goose chase it seems.. Not sure what is going on with your sniffing.. But sniffing always before firewall, nothing the firewall can do to prevent sniff from seeing traffic that is getting to the interface.

lifeboy

@johnpoz Hmmm. How do we know that 102.218.128.0/19 is not on the current list anymore? Mine is also from 20 August, but I was not able to find the authorative source by searching the web.