Unable to update repository pfSense

patient0

@elvisimprsntr can you resolve other domain names?

elvisimprsntr

@patient0

From clients, yes.

From pfSense GUI or CLI, no.

Using Q9 for upstream DNS

patient0

@elvisimprsntr said in Unable to update repository pfSense:

From pfSense GUI or CLI, no.
Using Q9 for upstream DNS

How have you configured DNS, resolver or forward? And your clients use pfSense for DNS lookup?

SteveITS

@elvisimprsntr If forwarding, ensure DNSSEC is disabled.

Any restrictions on unbound or is it set to listen on All interfaces?

I would next restart DNS Resolver/unbound, or maybe pfSense.

elvisimprsntr

@patient0

DNS Resolver
Yes Clients use pfSense

I tried starting unbound. same issue.
I remotely rebooted my fiber modem (NVG599), and released and renewed WAN interface.

Seems to be working again.

Running 2.7.2 with all the latest patches.

This just started happening since I applied the latest swath of patches.

It is behaving like something on pfSense crashed or stops working.

@SteveITS

When I disable DNSSEC can't resolve static mappings.

https://forum.netgate.com/topic/196269/att-fiber-dns-issue/13

patient0

@elvisimprsntr said in Unable to update repository pfSense:

This just started happening since I applied the latest swath of patches.

I know it is obvious but I'd revert patches regarding unbound and see what happens.

stephenw10

Are clients, which can resolve, using pfSense for DNS or being passed quad9 to use directly?

What is the DNS behaviour setting in General Setup?

elvisimprsntr

@stephenw10 said in Unable to update repository pfSense:

Are clients, which can resolve, using pfSense for DNS or being passed quad9 to use directly?

Client are served pfSense LAN IP address

What is the DNS behaviour setting in General Setup?

Screenshot 2025-02-08 at 10.53.22 AM.png

stephenw10

Hmm, and Diag > DNS Lookup just fails for all configured severs? For any fqdn?

elvisimprsntr

@stephenw10

Correct.

After rebooting my fiber modem, and releasing and renewing the WAN interface it is working again.

To see if it has any affect, I also deleted the IPv6 Q9 DNS servers.

stephenw10

Hmm, weird. Hard to imagine what could have caused that for pfSense but not clients behind it.

elvisimprsntr

@stephenw10

When I restart both kea-dhcp4 and unbound the problem persists.

When I head to Status -> Interfaces -> Click on WAN Release then Renew, everything starts working again.

Not sure if this error is related when pfSense is trying to get an IP address from my fiber modem, which is in DHCP passthrough mode.

Feb  9 13:45:55 pfSense dhclient[9122]: unknown dhcp option value 0x7d


Feb  9 13:45:55 pfSense dhclient[9122]: DHCPREQUEST on igc0 to 192.168.1.254 port 67
Feb  9 13:45:55 pfSense dhclient[9122]: DHCPACK from 192.168.1.254
Feb  9 13:45:55 pfSense dhclient[9122]: unknown dhcp option value 0x7d
Feb  9 13:45:55 pfSense dhclient[11337]: RENEW
Feb  9 13:45:55 pfSense dhclient[12245]: Creating resolv.conf
Feb  9 13:45:55 pfSense dhclient[9122]: bound to [redacted] -- renewal in 300 seconds.

The behavior almost seems like it is losing a WAN route.

I piped the following script to a log file while it is working.

Will run again once symptoms reoccur.

#!/bin/sh

ifconfig igc0
netstat -rWn
cat /var/db/dhclient.leases.igc0

What other logs should I comb through for any evidence?

patient0

@elvisimprsntr said in Unable to update repository pfSense:

Feb 9 13:45:55 pfSense dhclient[9122]: unknown dhcp option value 0x7d

There exists a bug report for FreeBSD regarding this error: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281361

Seems to be about unknown DHCP options send from upstream, more annoying then a real issue.

elvisimprsntr

@patient0 said in Unable to update repository pfSense:

Seems to be about unknown DHCP options send from upstream, more annoying then a real issue.

Unless it is somehow affecting KEA

patient0

@elvisimprsntr said in Unable to update repository pfSense:

Unless it is somehow affecting KEA

Can't see how, two different applications on different interfaces, dhclient and KEA. But then it's computers and nothing is impossible.

stephenw10

Yup Kea and Unbound are not dhclient so that's probably unrelated.

Is unbound actually starting correctly?

@elvisimprsntr said in Unable to update repository pfSense:

The behavior almost seems like it is losing a WAN route.

Is it actually losing a route? Is there a default route in the routing table? Can pfSense still ping out by IP address?

Gertjan

@patient0 said in Unable to update repository pfSense:

Feb 9 13:45:55 pfSense dhclient[9122]: unknown dhcp option value 0x7d
There exists a bug report for FreeBSD regarding this error: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281361

Seems to be about unknown DHCP options send from upstream, more annoying then a real issue.

So shown on "bugs.freebsd.org" :

The option in question is better known as 'DHCP Option 125' -- used for 'Vendor Specific Information'

This options comes from an upstream DHCP (ISP ?!), server that gives this 'extra' option the the DHCP client (pfSense), who didn't asked for it. So, do panic, it will discarded.
But a log line will tell the admin it did so. Not an error at all, imho, this is something between an INFO and WARNING messages, not an error.

Because we're all networks admins, why not telling us what this option is all about ?
Easy to find out (we have pfSense, remember ?!) :

The settings (I presume none of them needs any introduction) :

and hit the start button.
( in short : WAN interface, DHCP is UDP, and the ports are 67 and 68 - and we want all the details)

After after a while (and you can know upfront when it will happen) :
The device (pfSense DHCP client) sends out a lease request :

16:05:06.715857 a4:bb:6d:ba:16:a1 > 90:ec:77:29:39:2c, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 128, id 31136, offset 0, flags [none], proto UDP (17), length 328)
    192.168.1.6.68 > 192.168.1.1.67: [udp sum ok] BOOTP/DHCP, Request from a4:bb:6d:ba:16:a1, length 300, xid 0x7dd77758, Flags [none] (0x0000)
	  Client-IP 192.168.1.6
	  Client-Ethernet-Address a4:bb:6d:ba:16:a1
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: Request
	    Client-ID (61), length 7: ether a4:bb:6d:ba:16:a1
	    Hostname (12), length 7: "Gauche2"
	    FQDN (81), length 10: "Gauche2"
	    Vendor-Class (60), length 8: "MSFT 5.0"
	    Parameter-Request (55), length 14: 
	      Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Domain-Name (15)
	      Router-Discovery (31), Static-Route (33), Vendor-Option (43), Netbios-Name-Server (44)
	      Netbios-Node (46), Netbios-Scope (47), Unknown (119), Classless-Static-Route (121)
	      Classless-Static-Route-Microsoft (249), Unknown (252)

Take note of the somewhat standard DHCP options 1,3,6,15,31,33,44,46,47,119,121.
You saw the 43 ? 119 ? 252 ?

a couple of ms later you'll see the answer from the server :

16:05:06.722112 90:ec:77:29:39:2c > a4:bb:6d:ba:16:a1, ethertype IPv4 (0x0800), length 359: (tos 0x10, ttl 128, id 0, offset 0, flags [DF], proto UDP (17), length 345)
    192.168.1.1.67 > 192.168.1.6.68: [udp sum ok] BOOTP/DHCP, Reply, length 317, xid 0x7dd77758, Flags [none] (0x0000)
	  Client-IP 192.168.1.6
	  Your-IP 192.168.1.6
	  Client-Ethernet-Address a4:bb:6d:ba:16:a1
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: ACK
	    Subnet-Mask (1), length 4: 255.255.255.0
	    Default-Gateway (3), length 4: 192.168.1.1
	    Domain-Name-Server (6), length 4: 192.168.1.1
	    Domain-Name (15), length 20: "bhf.tld"
	    Vendor-Option (43), length 6: 1.4.192.168.1.6
	    Lease-Time (51), length 4: 21600
	    Server-ID (54), length 4: 192.168.1.1
	    FQDN (81), length 11: [N] "gauche2."

So the DHCP server answered with option 1,3,6,15,43 (!) and 81.
So some requested option from the client went unanswered.
Other options, like 81 and 54 are sent without being asked for.
Option 51 : the lease time, was unasked, but was given anyway ^^

Btw : I faked somewhat the example, as the server is here pfSEnse, and the client one of my LAN DHCP clients, a MS PC. hence the typical MS option requests.
I could stop the pfSense WAN DHCOP client lease, but that would 'break' my Internet connection, and I've a load of colleagues using it right now ...

For every option, as these are extremely well documented, go and look them up.

Now, your turn : what did you see ?

Btw : my Kea pfSEnse server offers option 43, or by default, pfSense doesn't know about it.

So I had to do this : https://redmine.pfsense.org/issues/15321 as 'extra' the option '43' is very known when you use Unifi equipment.

elvisimprsntr

@Gertjan said in Unable to update repository pfSense:

Now, your turn : what did you see ?

Kit:

ATT NVG599 RG in passthrough mode
Protectli FW4C running 2.7.2 with all the latest patches.

This is what I captured

16:47:18.056463 64:62:66:21:95:XX > d4:04:cd:83:e9:XX, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 59801, offset 0, flags [none], proto UDP (17), length 328)
    69.111.183.XX  > 192.168.1.254.67: [udp sum ok] BOOTP/DHCP, Request from 64:62:66:21:95:XX, length 300, xid 0x60716ba8, Flags [none] (0x0000)
	  Client-IP 69.111.183.XX
	  Client-Ethernet-Address 64:62:66:21:95:XX
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: Request
	    Client-ID (61), length 7: ether 64:62:66:21:95:XX
	    Hostname (12), length 7: "pfSense"
	    Parameter-Request (55), length 10: 
	      Subnet-Mask (1), BR (28), Time-Zone (2), Classless-Static-Route (121)
	      Default-Gateway (3), Domain-Name (15), Domain-Name-Server (6), Hostname (12)
	      Unknown (119), MTU (26)
16:47:18.098217 d4:04:cd:83:e9:XX > 64:62:66:21:95:XX, ethertype IPv4 (0x0800), length 382: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 368)
    192.168.1.254.67 > 69.111.183.XX: [udp sum ok] BOOTP/DHCP, Reply, length 340, xid 0x60716ba8, Flags [none] (0x0000)
	  Client-IP 69.111.183.XX
	  Your-IP 69.111.183.XX
	  Client-Ethernet-Address 64:62:66:21:95:XX
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: ACK
	    Server-ID (54), length 4: 192.168.1.254
	    Lease-Time (51), length 4: 600
	    RN (58), length 4: 300
	    RB (59), length 4: 525
	    Subnet-Mask (1), length 4: 255.255.252.0
	    Default-Gateway (3), length 4: 69.111.180.1
	    Domain-Name (15), length 12: "attlocal.net"
	    Domain-Name-Server (6), length 4: 192.168.1.254
	    Unknown (125), length 38: 0,3561,8452,1584,12337,17716,13829,3890,13107,12593,14128,14642,14643,12600,13622,1542,20054,18229,14649

Gertjan

@elvisimprsntr

I check first :
69.111.183.xx is your pfSense WAN IP and 192.168.1.254 is the DHCP server ?
I guess so.... so your DHCP server is nearby, like feet or so away - and you don't control it ?

Maybe its normal, but a RFC1918 IP DHCP server handing over public IP addresses, feels strange to me.
IPv4 leases of 600 seconds ... wow.

edit : oh wait, let me guess : you have a modem ISP device ?

Funny : the requesting DHCP client (pfSense) is asking for a option "119" but the packet (pfSense) capture decoding doesn't' know about it.

I saw the "125" DHCP server answer.
If you know the name of your ISP, and their 'end user device' you can probably find out what it wants / does / means.
( and don't be surprised if they can't answer you neither ^^ )

elvisimprsntr

@Gertjan

69.111.183.xx is my public IP assigned to pfSense
192.168.1.254 is the IP of the NVG599 and built in DHCP server.
600 seconds is the default lease time in the NVG599

Screenshot 2025-02-11 at 4.35.18 PM.png