PASS rule as an exception to a REJECT rule doesn't match

[Manatee]

Hello everyone,

I'm running 2.5.0.a.20210111.1027 on x86_64 hardware and I have an issue I seem to cannot solve myself.

There are currently two rules, in this order for the LAN device (not floating rules):

In the Facebook_WhatsApp alias there's this address:

In the Facebook alias there's this address range:

Relevant parts of the config file:
https://paste.ee/p/fMnGp

What I expected would happen:
All connections to the hosts in Facebook_WhatsApp passed via the VPN_PIA_US_VPNV4 gateway.
All connections to hosts that are in the Facebook alias but not in the Facebook_WhatsApp alias will be rejected, just because of the order of the rules.

What actually happens:
All connections to hosts that are in the Facebook alias are rejected, wether or not they're in the Facebook_WhatsApp alias.

There's nothing suspicious in the log files at all.

What am I missing?

[Manatee]

Because I can't edit my previous post anymore, I have to post a reply to my own thread.

I now realize that pfSense fails to populate the table:

That's despite there being copious log entries that would make me believe otherwise:

The Facebook_WhatsApp table remains empty even in the rules.debug file:

I still can't find any log entry relating to this alias with an explanation of what's happening.

Again, I'd be very thankful for anyone who could point me in the right direction.

[serbus]

Hello!

My limited experience with aliases is that they can be finicky.

While probably not directly on target, https://redmine.pfsense.org/issues/9296 might give you some insights.

Maybe you could divide the the Facebook_Whatsapp alias into separate ip/fqdn aliases and see if it helps with the table generation.

John

[Manatee]

@serbus Thank you very much for replying! Seems that I'm experiencing Bug #9296 indeed!

Splitting Facebook_WhatsApp into Facebook_WhatsApp_IP and Facebook_WhatsApp_FQDN makes it seem that pfSense has an issue resolving the FQDNs for the table as the table Facebook_WhatsApp_FQDN seems to remain empty still:

[serbus]

Hello!

A workaround from left field would be to do your own fqdn lookups and build your own ip alias table/feed. I am just spitballing here...

It might look something like:

• make a file /usr/local/www/mytable.fqdn that contains one fqdn per line
• run something like

dig @8.8.8.8 +noall +answer -f /usr/local/www/mytable.fqdn | awk '$4 == "A" {print $5}' | tee /usr/local/www/mytable.ips

Make a system URL Table Alias using http://127.0.0.1/mytable.ips or pfb feed using /usr/local/www/mytable.ips

You would probably want to cron the dig.
Prodding the alias to reload the url table might be problematic. Using pfb to refresh the alias might be easier.

John

[Manatee]

@serbus Thank you very much, again!

I'll use your workaround in the meantime, until there's a proper fix. I added some parallelism and sorting tho :

find /usr/local/www/ -name '*.fqdn.txt' | parallel --plus 'parallel -I // -j 30 dig // A @1.1.1.1 +time=2 +tries=1 +noall +answer :::: {} | awk '\''$4 == "A" {print $5}'\'' | tee {%.fqdn.txt}.ip.txt; sort -t. +0n -1 +1n -2 +2n -3 +3n -u -o {%.fqdn.txt}.ip.txt {%.fqdn.txt}.ip.txt'

(requires pkg add http://pkg.freebsd.org/FreeBSD:12:amd64/release_2/All/parallel-20200922.txz)

[Gertjan]

@manatee said in PASS rule as an exception to a REJECT rule doesn't match:

What I expected would happen:

@manatee said in PASS rule as an exception to a REJECT rule doesn't match:

I now realize that pfSense fails to populate the table:

Read me first.

Take note of this note :

Using Hostnames in Aliases

Hostnames can also be used in aliases. Any hostname can be entered into a host or network alias and it will be periodically resolved and updated by the firewall. If a host name returns multiple IP addresses, all of the returned IP addresses are added to the alias. This is useful for tracking dynamic DNS entries to allow specific users into services from dynamic IP addresses.

Big public sites like the Google, Apple, Facebook, Twitter, etc etc etc use many thousands of IP's, they change all the time. You never have them all listed, even if you try to resolve them every second.

edit : Your alias only shows a couple of IPs at best. The ones that were valid at that moment.
What happens is : in the facebook == Watsapp app have a list with build host names and/or IP's (carefully hidden of course). These hosts are CDN's, and these have the actual lists with all the real domain names of the domain servers that resolves Facebook IPs.

And I guess you'll agree with : Do you think Facebook has the financial and technical resources to make it possible so that their users can access Facebook's services as long as they have a 'working' Internet connection that doesn't have half of the possible IP addresses and half of the port blocked ??
Well, guess what : they did just that : investing heavily in make it your task more difficult, they threw in multiple of millions. I even suspect Facebook has a department called "anti-patental-control". And that department does not communicate its findings on the net. It's their "NSA".

Blocking these guys needs special treatment, it's a real sport.
It's one of the reasons packages like this one has been created.

use the search function of this forum, and you'll find out that many posted questions on the forum already. Come of them even have answers, about can be done.
Or open a big search engine and type : "how to block Facebook ?"
Read the first 10 hits. You'll get the picture. A simple pfSense alias won't pull it.

Good news is : Facebook is planning to help you blocking Whatsapp, as they (were forced to) to "tell the public" their intentions. (and the fun part is that there were people that were surprised, or at least, this is what the media is trying to to tell us ..... as we all know that share holders will have their annual bonus, what ever happens, and the money comes from ..... etc etc ).
In short : the fading out process is started.

@serbus : nice !
But it's still like finding a path throughout moving sable.
Servers are being maintained at FB's side : they are shut down, upgraded, replaced, etc. FB, and the others like Microsoft, Google, etc etc do not remap the used IP's to other servers while doing so - un purpose. They have crafted their clients to find a way to reach them.

[Manatee]

@gertjan Thanks for your reply, too.

Of course I agree with you (and I'm looking forward to a world with less WhatsApp and more Signal, Threema and the likes) and I'm very grateful for you pointing out this flaw in my thinking, but I feel like you're missing the point of this thread a bit. I'm afraid focusing on the example provided by me may distract from the real problem at hand.
The fact that some tables aren't being populated / FQDNs in rules aren't resolved properly (thus firewall rules not matching) has nothing to do with Facebook in particular (at least I don't think it does?). I don't see how this could possibly be intended behavior. So it still seems to me like a bug in pfSense / filterdns(?) that absolutely needs to be fixed asap and independently of what an idiot like myself is trying to achieve specifically.
The Facebook_WhatsApp alias is not the only case that exhibits this problem, it was just the one that made me aware of the issue, because I still have to use WhatsApp frequently . That is the only reason Facebook and WhatsApp are mentioned in this thread at all.

But again, thank you very much for your time writing that detailed reply, it's honestly much appreciated! I'll act accordingly!

[NogBadTheBad]

@manatee Your best bet would be use pfBlocker-NG and use ASN numbers to create an alias to use in your firewall rules.