System updated to 2.3.4-p1, now cant list packages?

docwho76

Something is wrong with my pfsense, this is on a SG-2440. I upgraded it to 2.3.4-p1 two weeks ago and I noticed I cant list any packages on it. So I investigated some below:

[2.3.4-RELEASE][root@docwho76.changeip.org]/root: ls -al /var/db/pkg/
total 4176
drwxr-xr-x 2 root wheel 512 Aug 11 18:31 .
drwxr-xr-x 15 root wheel 1024 Aug 11 21:16 ..
-rw-r–r-- 1 root wheel 4210688 Jul 25 21:50 local.sqlite
-rw-r--r-- 1 root wheel 246 Jul 20 09:52 pfSense-core.meta
-rw-r--r-- 1 root wheel 246 Aug 6 08:19 pfSense.meta

[2.3.4-RELEASE][root@docwho76.changeip.org]/root: pkg -v
1.10.1

[2.3.4-RELEASE][root@docwho76.changeip.org]/root: host -t srv _https._tcp.pkg.pfsense.org
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files01.netgate.com.
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files00.netgate.com.

Enter an option: 13

Updating repositories metadata…
Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
pkg: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/meta.txz: Operation timed out
repository pfSense-core has no meta file, using default settings
pkg: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/packagesite.txz: Operation timed out
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
pkg: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-pfSense_factory-v2_3_4/meta.txz: Operation timed out
repository pfSense has no meta file, using default settings
pkg: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-pfSense_factory-v2_3_4/packagesite.txz: Operation timed out
Unable to update repository pfSense
Error updating repositories!

How do I fix this? I'd rather not reinstall the whole OS here, seems like using a sledgehammer

Gertjan

Looks like pfSense can't create the file repo-pfSense-core.sqlite (among others !) here /var/db/pkg/
I have this :

[2.3.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: cd /var/db/pkg/
[2.3.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/var/db/pkg: ls -al
total 49828
drwxr-xr-x   2 root  wheel       512 Aug 11 10:26 .
drwxr-xr-x  18 root  wheel      1536 Aug 12 11:43 ..
-rw-r--r--   1 root  wheel       246 May 17  2016 FreeBSD.meta
-rw-r--r--   1 root  wheel   4055040 Aug  2 14:59 local.sqlite
-rw-r--r--   1 root  wheel       246 Jul 23 12:35 pfSense-core.meta
-rw-r--r--   1 root  wheel       246 Aug 11 07:31 pfSense.meta
-rw-r--r--   1 root  wheel  45684736 May 17  2016 repo-FreeBSD.sqlite
-rw-r--r--   1 root  wheel    212992 Jul 14 22:08 repo-pfSense-core.sqlite
-rw-r--r--   1 root  wheel    897024 Aug 11 07:31 repo-pfSense.sqlite

(running on a classic desktop PC using a hard drive …..)

What does the command "df" tells you ?

Re-installing is a 5 minutes job (pfSense is NOT "Windows" ^^)

PiBa

The missing .sqlite files are not a issue, they are downloaded automatically again..(if a proper internet connection can be made)
The FreeBSD ones shouldnt even exist i suppose Gertjan has been installing packages from native FreeBSD repositories..(unsopported anyhow)

What remains is the "Operation timed out" which sounds like the downloading of those files is failing..

Can you run this on pfSense?:```
fetch https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/meta.txz

Gertjan

@PiBa:

…
The FreeBSD ones shouldnt even exist i suppose Gertjan has been installing packages from native FreeBSD repositories..(unsopported anyhow)

Interesting …. because I have a pretty clean pfSense setup.
But true : I use "nano" and "munin" also.

docwho76

@PiBa:

The missing .sqlite files are not a issue, they are downloaded automatically again..(if a proper internet connection can be made)
The FreeBSD ones shouldnt even exist i suppose Gertjan has been installing packages from native FreeBSD repositories..(unsopported anyhow)

What remains is the "Operation timed out" which sounds like the downloading of those files is failing..

Can you run this on pfSense?:```
fetch https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/meta.txz

Nope, which is weird

[2.3.4-RELEASE][root@docwho76.changeip.org]/root: fetch https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/meta.txz
fetch: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_4_amd64-core/meta.txz: Operation timed out
[2.3.4-RELEASE][root@docwho76.changeip.org]/root: ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
–- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
[2.3.4-RELEASE][root@docwho76.changeip.org]/root: ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: icmp_seq=0 ttl=56 time=52.441 ms
64 bytes from 8.8.4.4: icmp_seq=1 ttl=56 time=39.676 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=56 time=37.524 ms
^C
–- 8.8.4.4 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.524/43.214/52.441/6.584 ms
[2.3.4-RELEASE][root@docwho76.changeip.org]/root: ping www.yahoo.com
PING atsv2-fp.wg1.b.yahoo.com (206.190.36.45): 56 data bytes
^C
–- atsv2-fp.wg1.b.yahoo.com ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss
[2.3.4-RELEASE][root@docwho76.changeip.org]/root: ping www.google.com
PING www.google.com (216.58.194.196): 56 data bytes
^C
–- www.google.com ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss

I think this is related to the fact I have a multi-WAN setup. Currently my WAN1 is down and I'm running on my WAN2 (8.8.8.8 is pinned to WAN1) so it seems like maybe a routing issue of some sort.

PiBa

Check what your default route is under diagnostics/routes?
And possibly enable automatic gateway switching to maybe avoid this.?. Or at least set it to the working WAN for the moment.(under system/routing)

docwho76

Yeah, this doesnt look right at all:

default 192.168.100.1 UGS 432 1500 igb0
8.8.4.4 22.166.220.50 UGHS 526 1500 igb1
8.8.8.8 192.168.100.1 UGHS 8936 1500 igb0
22.166.220.48/30 link#2 U 0 1500 igb1
22.166.220.49 link#2 UHS 0 16384 lo0
127.0.0.1 link#8 UH 1268012 16384 lo0
172.16.0.0/24 link#4 U 8516 1500 igb3
172.16.0.1 link#4 UHS 0 16384 lo0
172.16.0.2 link#9 UH 0 1500 ovpns1
192.168.1.0/24 link#3 U 448192669 1500 igb2
192.168.1.1 link#3 UHS 0 16384 lo0
192.168.100.0/24 link#1 U 17 1500 igb0
192.168.100.11 link#1 UHS 0 16384 lo0

The default shouldnt be pointing to 192.168.100.1 (my currently down cablemodem connection) it should be pointed at 22.166.220.50. See my LAN firewall rules here:

Protocol Source Port Destination Port Gateway Queue Schedule Description

* * LAN Address 666/80/22 * * Anti-Lockout Rule
IPv4* * * 192.168.100.1 * WAN_DHCP none Allow access to cablemodem status via LAN
IPv4* LAN net * * * Failover none Default allow LAN to any rule
IPv6* LAN net * * * * none Default allow LAN IPv6 to any rule

docwho76

@PiBa:

Check what your default route is under diagnostics/routes?
And possibly enable automatic gateway switching to maybe avoid this.?. Or at least set it to the working WAN for the moment.(under system/routing)

Automatic gateway switch IS on. Hmmm, wth

docwho76

[2.3.4-RELEASE][root@docwho76.changeip.org]/root: route add default 22.166.220.50
route: writing to routing socket: File exists
add net default: gateway 22.166.220.50 fib 0: route already in table

PiBa

try 'route change default 22.166.220.50'

docwho76

@PiBa:

try 'route change'

Aha, great success now! However, some very troubling things here:

1. Automatic gateway switching is ON, so why did it never change the default route?
2. I tried disabling WAN1, this did not affect the default route!
3. I tried changing the gateway for the LAN default rule from the Failover group to WAN2, this did not affect the default route!

How can this be?

PiBa

Ok so viewing packages works? thats great :D.

1. that i dont know, status/gateways shows that gw1 is down i suppose?
2. strange, but not sure if any logic is written that would take care of this updating the routes where needed..
3. this is by design, traffic from pfsense itself does not pass through your firewallrule on the lan interface. And its possible to have multiple groups with different orders of tier1 gateways so its (currently) not possible to have these update the default route..

docwho76

@PiBa:

Ok so viewing packages works? thats great :D.

1. that i dont know, status/gateways shows that gw1 is down i suppose?
2. strange, but not sure if any logic is written that would take care of this updating the routes where needed..
3. this is by design, traffic from pfsense itself does not pass through your firewallrule on the lan interface. And its possible to have multiple groups with different orders of tier1 gateways so its (currently) not possible to have these update the default route..

Yes, it shows my WAN1 GW as being down

Derelict

If your WAN1 gateway is down hard for the duration, I would just make the WAN2 gateway the default gateway until WAN1 is back up.

Any of these remarkable checkboxes checked on either of your gateways?

![Screen Shot 2017-08-12 at 2.44.34 PM.png](/public/imported_attachments/1/Screen Shot 2017-08-12 at 2.44.34 PM.png)
![Screen Shot 2017-08-12 at 2.44.34 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2017-08-12 at 2.44.34 PM.png_thumb)