"Bug": Threatgate list management?

[JeGr]

I just tried using Threatgate in a VM to download a big normal IP blocklist. E.g. Firehol1.
Download it did and it showed up in Aliases.

BUT: It did show up in the normal aliases tab, NOT in URL
Also while the list is blocked in the normal GUI with a red lock, in Nexus you can click edit and it can crash your browser.

Because what I thought TG is doing is exactly that: the list you enter is parsed and then simply entered into a network type alias which is arguably not the best way to deal with it. Why? Because Network Type Aliases are directly fed into "pf" via the pf rule file. You can see the generated one in /tmp/rules.debug. But "pf" (since coming from OpenBSD and having worked with it) dislikes big lists and even pfSense's own docs states that you should run it via external list files and why you should use URL Table style aliases instead of other variants:
--> https://docs.netgate.com/pfsense/en/latest/firewall/aliases-types.html#url-table-aliases

So Threatgate is simply throwing a list with over 100k entries simply into a network type alias which in turn gets thrown into the pf.conf. So when we compare e.g. a Firehol1 BL via URL Table IP (or pfBlocker). Also in Nexus the list IS editable and it slows down my browser to a crawl trying to load the contents of the list :/

Comparison:

# manual list alias
table <TestFH1> persist file "/var/db/aliastables/TestFH1.txt"
TestFH1 = "<TestFH1>"

# pfB list alias
table <pfB_Firehol12_v4> persist file "/var/db/aliastables/pfB_Firehol12_v4.txt"
pfB_Firehol12_v4 = "<pfB_Firehol12_v4>"

while TG does:

table <TG_TestFH1> {   1.10.16.0/20  1.19.0.0/16  1.32.128.0/18  2.26.75.0/24  2.27.5.0/24  2.56.192.0/22  2.57.122.0/24  2.57.232.0/22  2.58.56.0/24  2.59.152.0/23  5.42.92.0/24  5.101.86.0/24  5.105.220.0/24  5.175.189.0/24  5.183.60.0/22  5.188.236.0/23  5.230.201.0/24  5.231.63.0/24  5.252.153.0/24  14.102.240.0/20  14.128.32.0/20  14.128.48.0/21  (... 1000s more ...)
TG_TestFH1 = "<TG_TestFH1>"

Was that a deliberate decision and why? The external file loading has been in pf for quite some time and AFAIK in case of >1000 entries is the better way then hardcore pushing them into pf.conf (or rules.debug). It's also far easier to debug in single files. So why did no one check how that function is implemented?

Also could we please stop those "unlicensed" error messages on a valid licensed plus VM just because "it's not secure"? Why shouldn't it be when it's my lab and I installed it by hand knowing what I'm doing? :)

Thanks and Cheers :)
\jens

[JeGr]

Wanting to add another ThreatGate Problem:

Adding other TG lists wanting to test the limits what TG can import or manage I renamed the ThreatGate list I created earlier (TestFH1) and before adding Firehol2 to test, I renamed it again to steamline the naming of the lists. In TG it looks fine:

but in the normal Alias view it shows a big problem:

TG didn't rename the list, but created it anew. And again. So now I actually have 3 TH firehol1 Feeds without the ability to delete them, as they are "managed by TG" so the normal UI can't modify/delete them and TG doesn't see a problem.

How do I delete them now?

Cheers
\jens

[JeGr]

Another addition:

• when you disable Threatgate, the lists get deleted. Yay! :)
• when re-enabling only firehol2 was created. I checked again and accidentally had the same URI on both fh1 and fh2, so it seems TG is smart enough to only download once? But it didn't create both Aliases. So OK, I changed the URI for FH2, made sure to save again and ... nothing. FH1 isn't being created, FH2 didn't change. Nothing happened besides multiple popups in Nexus showing it should work and everything is green and fine.
• I then went into both lists again, clicked save there and hit save in the little round bubble again
• Only then was FH1 created and FH2 actually updated with different contents.

Then afterwards I tried one of our bigger lists:

• Trying to add the QFeeds IP blocklist failed completely. It shows in Nexus it's downloading, then the popups for saving and reloading popup, all is green but in the old UX the list never shows up.

I had to save it multiple times again while no error to make it download and finally show up in both GUIs.

Besides that, rules.debug is now ~250kb in size only with those 3 lists, normally the rules only being around 20k so roughly 10-12 times the size of normally fetching those lists via URLtables.
But because of that I was curious why it was "only" 250kb as the Qfeeds list is quite big. So I re-checked rules.debug and found TG actually cuts lists bigger then 4999 entries due to using normal Aliases instead of URL Tables:

TG managed list

Manually added list (same as pfB also uses):

So without telling any error TG currently just cuts lists after 5k entries and forgets the rest. Unfortunately that makes it unusable until it's fixed to creating tables instead of inline aliases.

Hope that helps.

@marcosm / @stephenw10
If I can add any other context or analysis, let me know and I'll gladly test :)

Cheers
\jens

[JeGr]

Quick addition:

I actually ran into an API (from Qfeeds) Limit so at one time where I tried saving to re-download, it actually tried but got an error but showed NO feedback in the popups, errors or anything else. Only after browsing Nexus Logs did I see a API restriction answer from a curl call. The Nexus GUI itself showed all happy greens and no errors. That's also not that nice for debugging.

Also not seeing anywhere in the TG UI how many "items" in a list were actually downloaded or loaded (in terms of DNS lists) at all makes it also hard to check if something went right or wrong.

Ah and a final thing: TG doesn't "remove" lists when you disable TG or the list in TG. When the Alias is in use, you'll get no info whatsoever and wonder, why you're disabling everything and it doesn't vanish. Even when you rename stuff, it doesn't get renamed (in my test at least).

[lmahieu75]

@JeGr Your "TG currently just cuts lists after 5k entries and forgets the rest" is exactly right, and I can put hard numbers on it. But I think the cause you propose -- network-type aliases instead of URL Table aliases -- is not the one. I tested it, and it isn't.

What I measured (Netgate 6100, 26.07-RELEASE, Nexus in production)

The cap is exactly 4999, and it is the same number everywhere:

Remote text feed, ~15000 entries    -> TG table holds 4999
MaxMind Country feed, 37 countries  -> TG table holds 4999
Same source split into 3 local feeds of 4900 -> all 14700 load

So the cap is per feed, not global. That last line is what makes chunking a viable workaround today.

Why it is not a source-format problem

I built the input myself to remove any doubt: 6000 unique IPv4 addresses, one per line, no comments, no duplicates, no blank lines, served over plain HTTP from the firewall itself.

pfctl -t TG_<feed> -T show | wc -l
4999

A malformed line cannot produce exactly 4999 on a file I generated line by line. Format is ruled out.

Why it is not the alias type (this is the part I think you got wrong)

I ran the same feed twice, changing only the Mode field:

mode = block   -> 4999 entries
mode = alias   -> 4999 entries

Identical. If the network-alias-vs-URL-Table distinction were the cause, those two paths would not truncate at the same point. The truncation happens upstream of how the list is applied, during ingestion.

I do agree with your wider point that network-type aliases inlined into rules.debug are the wrong mechanism for lists this size -- pfSense's own docs say URL Table aliases are the right tool and mention country lists "easily surpassing 40,000 entries". But that is a separate defect about rules.debug size and GUI performance. It is not what produces 4999.

Where the number probably comes from

4999 = 5000 - 1. In the open pfSense repository:

src/usr/local/pfSense/include/www/alias-utils.inc
  line 297   $max_alias_addresses = 5000;
  line 506   for ($x = 0; $x < ($max_alias_addresses - 1); $x++) {

That loop reads address0..address4998 -- exactly 4999 slots -- and emits no error when the input array is longer. Same bound at line 405, and the constant is duplicated in firewall_aliases_edit.php line 49.

Caveat I want to state plainly: in the open repo saveAlias() is only called by the legacy PHP alias editor, and Nexus is Go. So I cannot prove pfnet-controller takes that path. But it is the only place in the entire public pfSense codebase that produces exactly 4999, and the constant 5000 is a long-standing internal convention for non-URL-Table aliases.

Why it matters more than a missing tail

With 37 countries selected, the retained entries follow numeric order. Low ranges survive, high ranges vanish. Concretely:

pfctl -t TG_<feed> -T test 220.181.38.148   -> 0/1 addresses match
pfctl -t TG_<feed> -T test 175.45.176.1     -> 0/1 addresses match

China and North Korea -- the first two countries in the selection -- are not blocked at all. The UI shows 37 countries configured, green, no warning, nothing in /var/log/pfnet-controller.log.

That is the real problem. Not "some entries are missing", but "the console says protected and the firewall is not". Your own observation that the TG UI never shows how many items a list actually loaded is the same defect seen from the other side: there is no way to notice.

What I think needs to happen

Two separate fixes, and I would argue the second matters more than the first:

1. Raise or remove the per-feed cap, and move ThreatGate lists to file-backed URL Table style aliases as you suggest.
2. Independently of 1: fail loudly. A silent truncation in a security control is a defect in its own right. Even a corrected cap will be hit by someone.

For anyone arriving here from a search

Working around it today, in order of preference:

• Preprocess your source into chunks of <= 4900 entries, serve them locally, one Remote Feed per chunk. Verified working.
• Or bypass ThreatGate ingestion entirely with a native Firewall > Aliases > URL Table (IPs) alias. Note this is URL Table (IPs), not URL (IPs) -- the latter has its own 3000-entry cap.
• Always verify with pfctl -t <table> -T show | wc -l. Do not trust the feed status.

Related tickets

• #17029 ThreatGate remote feed silently truncated at 4999 (mine, New, Urgent, unassigned, no target version, no response in 6 days)
• #17027 GeoIP alias generated via ThreatGate fails to match traffic for valid country IPs -- different reporter, different hardware (4100). I believe 17029 is the mechanical explanation for it: the test address falls in the numerically high, truncated part.
• #17030 ThreatGate remote feeds never refreshed by refresh_interval

@marcosm @stephenw10 -- one specific question rather than a general complaint: System > Advanced > Netgate Nexus has a Custom Options field, documented as "do not use unless directed by Netgate developers or TAC". Is there an option name that raises this per-feed limit? If there is, it unblocks everyone reading this thread today, independently of when 17029 gets fixed.

For context on the gap between spec and behaviour, @jwt wrote in the Nexus/ThreatGate/CoreDNS thread: "we can now hit sub-second retrieval for 1M rows even under heavy concurrent load. We've tested well beyond 5M rows." Announced 5M, delivered 4999.

[lmahieu75]

Correction to my previous post -- @JeGr you were right and my objection was not valid.

I claimed the block/alias Mode test ruled out your "normal Aliases instead of URL Tables" explanation. It does not. Both Modes produce the same kind of object: an ordinary network alias. They differ in what is done with it, not in how it is stored. On my box, mode=alias produces

table <TG_Pays_Bloques> { 1.0.1.0/24 1.0.2.0/23 ... }

inlined in rules.debug, and /var/db/aliastables/ contains no TG file at all. So both Modes go down the network-alias path and both hit 4999. That is evidence for your explanation, not against it. I compared two variants of the same branch and drew a conclusion about the other branch. My mistake.

Worse for my own argument: the code pointer I gave -- alias-utils.inc line 297 $max_alias_addresses = 5000; and line 506 for ($x = 0; $x < ($max_alias_addresses - 1); $x++) -- sits in the ordinary-alias save path. URL Table aliases are file-backed and do not go through that loop. So it supports your attribution too.

What my Mode test does establish, and it is a small point: the cap does not depend on the blocking behaviour, only on the storage path.

Everything else in my post stands: the 6000 synthetic well-formed addresses still loading 4999 rules out source formatting, the numeric-order truncation leaving China and North Korea entirely unblocked, and the silence in UI and logs.

Your fix is the right one: ThreatGate should write file-backed URL Table aliases. And it should fail loudly when it truncates.

[JeGr]

@lmahieu75 said in "Bug": Threatgate list management?:

Your fix is the right one: ThreatGate should write file-backed URL Table aliases. And it should fail loudly when it truncates.

Sadly noone even has replied yet to that post and after Netgate posting their Blogs about how Threatgate is THE thing to "replace" pfBlocker with: it could be WAY later, but right now it's a hot mess and not working the way it's advertised. Worse, it's not even doing the basics, that the docs tell you to manually do: use URL Tables for large lists.

A bit disappointed no bugfix was made via the Nexus package yet while that feature still gets promoted high and low and I have customers asking to switch to it. It's not nice to tell them "nope, it's broken".

Also the whole TG thing not even properly logging or giving visual and factual feedback about when it's

• downloading data
• failed to create an alias properly
• if it's created an alias
• how many entries are in said alias
• the last state when that alias was refreshed

or even more simply

• setting refresh times <3h
• enabling more verbose logging

for the whole section is disappointing for such a feature. Compared with pfB yes the latter one is slower and memory hungry. But it logs so gods-damned much you can easily find what's wrong with it. All this missing from TG is making me wonder, why it had to make its appearance right now and not waiting and fleshing it our more until an October or November release.

[keyser]

@JeGr wholeheartedly agree. It’s very buggy and definitely not customer ready yet. But we can hope for a much faster turnaround then regular pfsense+ releases. It’s included in the Nexus package which can be updated individually and out of band from regular pfsense+ updates

EDIT: But this “bundling” og UI, Threatgate, CoreDNS and Snort in one package will soon also become a massive liability… It will likely cause quick updates to any one of those products to become massively delayed, due to all the other components in the package that needs release coordination with Nexus

[pfsjap]

@keyser said in "Bug": Threatgate list management?:

EDIT: But this “bundling” og UI, Threatgate, CoreDNS and Snort in one package will soon also become a massive liability… It will likely cause quick updates to any one of those products to become massively delayed, due to all the other components in the package that needs release coordination with Nexus

I think this “bundling” is bad design. Threatgate, CoreDNS and Snort should all come as independent packages/services.