Network performance issue using pfSense v.2.7.0 running as router

mauro.tridici · Jun 11, 2024, 10:03 PM

Dear Users,

during the last year I set up a physical server (64 cores + 256GB RAM) to act as router using pfSense v.2.7.0.

Everything worked very well during this year with excellent network perfomance (we have two WAN interfaces, each one with a 10Gbps link to the ISP).

During the last two weeks, network seems to be very slow during upload and download activities. So, I started checking everything:

every router hardware components are ok;
no cables issues;
no firewall issues (firewall is a different server behind the router);
no changes on the router;
no changes on the firewall.

I executed multiple tests directly from the router shell and it seems that the perfomance problem is on the router or in the ISP links.
I tried to bring down the primary ISP link, but the problem is still here also using the secondary/backup ISP link.

I already opened a support case request to the ISP, but I have also some doubt about the router.
I noticed that pfSense is not seeing the latest available update (2.7.2):

2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT

The system is on the latest version.
Version information updated at Tue Jun 11 22:38:53 CEST 2024

So, I can't update it.
Do you have any interesting idea to troubleshoot this issue?

Thank you in advance,
Mauro

DBMandrake · Jun 12, 2024, 10:33 AM

@mauro-tridici If you suspect the performance issue is on the ISP side, if possible I would run a quick test where you unplug PFSense from your upstream router and substitute a PC configured with your public IP in the place of PFSense.

Of course this might not be feasible if you are not able to take the network connection down for other people during a test.

By the way your server is massively over speced for even a 10Gb connection.

Our main PFSense box is a 1U rack mounted Intel Core i7-10700 with 8 cores, 16GB of ram and a dual 10Gb Fibre network adaptor.

We have pretty much every feature you can enable turned on and it is easily able to route traffic between VLAN's at near wire speed, so 64 cores and 256GB of ram is just crazy.

stephenw10 · Jun 12, 2024, 1:43 PM

@mauro-tridici said in Network performance issue using pfSense v.2.7.0 running as router:

So, I can't update it.
Do you have any interesting idea to troubleshoot this issue?

That's a known issue coming from 2.7.0. At the CLI run: certctl rehash, then recheck.

JKnott · Jun 12, 2024, 2:17 PM

@stephenw10 said in Network performance issue using pfSense v.2.7.0 running as router:

That's a known issue coming from 2.7.0. At the CLI run: certctl rehash, then recheck.

I just tried that and it still shows 2.7.0 as current.

BTW, I tried to post this in the thread that lead me here, but it's locked.

stephenw10 · Jun 12, 2024, 6:15 PM

There were 3 duplicate threads, I locked the others.

certctl rehash is by far the most common cause coming from 2.7.0 but nit the only one.

What does pkg-static -d update show?

mauro.tridici · Jun 12, 2024, 3:52 PM

Hi @stephenw10 , thank you for your reply.

This is the output of the debug:

DBG(1)[41599]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[41599]> PkgRepo: verifying update for pfSense-core
DBG(1)[41599]> PkgRepo: need forced update of pfSense-core
DBG(1)[41599]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/meta.conf
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/meta.conf

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/meta.txz
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/meta.txz

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
repository pfSense-core has no meta file, using default settings
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/packagesite.pkg
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/packagesite.pkg

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/packagesite.txz
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-core/packagesite.txz

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
DBG(1)[41599]> PkgRepo: verifying update for pfSense
DBG(1)[41599]> PkgRepo: need forced update of pfSense
DBG(1)[41599]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/meta.conf
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/meta.conf

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/meta.txz
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/meta.txz

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
repository pfSense has no meta file, using default settings
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.pkg
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.pkg

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
DBG(1)[41599]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.txz
DBG(1)[41599]> curl_open
DBG(1)[41599]> Fetch: fetcher used: pkg+https
DBG(1)[41599]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.txz

DBG(1)[41599]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 2

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg00-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.207:443...
* Connected to pkg00-atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
DBG(1)[41599]> CURL> attempting to fetch from , left retry 1

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
pkg-static: An error occured while fetching package
Unable to update repository pfSense
Error updating repositories!

stephenw10 · Jun 12, 2024, 3:54 PM

@mauro-tridici said in Network performance issue using pfSense v.2.7.0 running as router:

CAfile: none

CApath: /etc/ssl/certs/

SSL certificate problem: unable to get local issuer certificate

That looks like the error that should be fixed by running certctl rehash. Did you try running that?

JKnott · Jun 12, 2024, 5:58 PM

@stephenw10 said in Network performance issue using pfSense v.2.7.0 running as router:

What does pkg-statc -d update show?

/root: pkg-statc -d update
pkg-statc: Command not found.

JKnott · Jun 12, 2024, 6:00 PM

@JKnott
Never mind. I just changed the update branch to 2.7.2 and it appears ready to update.

stephenw10 · Jun 12, 2024, 6:15 PM

@JKnott said in Network performance issue using pfSense v.2.7.0 running as router:

/root: pkg-statc -d update
pkg-statc: Command not found.

Sorry I typo'd that. Should be: pkg-static -d update

AK_4_Life · Jul 23, 2024, 1:49 PM

@stephenw10 worked for me! thanks for the tip.

georgelza · Oct 28, 2024, 9:13 AM

@stephenw10 said in Network performance issue using pfSense v.2.7.0 running as router:

pkg-static -d update

hi all...
got redirected to this thread. upgrading my pfSense device
did backup 2.7.2, install new software 2.7.0. restored config file.
rebooted... had to get some renaming of interfaces resolved.

packages don't want to installed, see error below

The below was executed after having executed.

certctl rehash

Fetching meta.conf: 100%    178 B   0.2kB/s    00:01
* Connection #0 to host pkg00-atx.netgate.com left intact
DBG(1)[2436]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.pkg
DBG(1)[2436]> curl_open
DBG(1)[2436]> Fetch: fetcher used: pkg+https
DBG(1)[2436]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.pkg

DBG(1)[2436]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg00-atx.netgate.com in the .netrc file; using defaults
* Found bundle for host: 0x8219d24d0 [serially]
* Re-using existing connection with host pkg00-atx.netgate.com
> GET /pfSense_v2_7_2_amd64-pfSense_v2_7_2/packagesite.pkg HTTP/1.1
Host: pkg00-atx.netgate.com
User-Agent: pkg/1.20.8
Accept: */*
If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT

< HTTP/1.1 200 OK
Fetching packagesite.pkg:   0%< Server: nginx
< Date: Mon, 28 Oct 2024 09:10:44 GMT
< Content-Type: application/octet-stream
< Content-Length: 160404
< Last-Modified: Mon, 21 Oct 2024 21:31:27 GMT
< Connection: keep-alive
< ETag: "6716c82f-27294"
< Strict-Transport-Security: max-age=31536000; preload
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: all
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Accept-Ranges: bytes
<
Fetching packagesite.pkg: 100%  157 KiB 160.4kB/s    00:01
* Connection #0 to host pkg00-atx.netgate.com left intact
DBG(1)[2436]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[2971]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[2436]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
Processing entries:   0%
Newer FreeBSD version for package xmlstarlet:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1400094
- running kernel: 1400085

georgelza · Oct 28, 2024, 9:16 AM

rinse and repeat... eventually package list showing...

but it's not showing/displaying original installed packages...
whats the thinking. do a upgrade to 2.7.0 -> 2.7.2 and then re load the config file.

nothing to loose, here it goes...
G

georgelza · Oct 28, 2024, 10:07 AM

@georgelza said in Network performance issue using pfSense v.2.7.0 running as router:

pkg-static -d update

ok... we're back cooking with gas, as the saying goes, got haproxy installed and my external exposed services are working again.

Would have been great if the restore allowed me to re-attempt the installation of previously installed packages.

G