Yes, I know! Another OPT1 and OPT2 no internet connection!

stephenw10

Run ifconfig at the command line and see which NICs are actually linked. Unplug one of the OPT ports and make sure the one that goes down if what you think it is.

MidGe48

@MidGe48

Some progress being made.

On my test (for this exercise) machine running Linux. Changing the connection type from "Automatic (DHCP)" to "Automatic (DHCP) addresses only" the bringing eth0 down and back up and it connects correctly and gets an ip address, of course. That is all on a working version before last upgrade and before switching to Kea.

So, now the issue is that one of the failing interfaces connect all sorts of IOT and media devices that I do not want to get into other subnets on my network (paranoia about privacy :) ). I am not sure if can change the way they request a DHCP connection. Or I may need change the way the DHCP server handles that. But there must have been a change very recently because I have had no such issues for a few years at least with my current network topology.

But thanks all for your help.

johnpoz

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

but disable (red cross) on the gui.

What would your client dhcp setting have to do with an interface not coming up? Make no sense.. If you connect something to that interface it should come up.. Regardless what dhcp or bootp or whatever protocol you might want to be using for dhcp, maybe the client is static.. None of that has anything to do with the interface coming up on your router.

Maybe just the bounce on the client interface is what brought up the interface on the router, that would make more sense then what you set its dhcp client to do.

MidGe48

@MidGe48

Since that last message and restoring a two month old backup, the interface comes up on the router as per my previous post.

Still have problems with the clients not being to connect unless I change the connection type on the client. This is possible on my Linux machines but a lot of IOT devices I use do not have that facility and do not connect readily, or connect but get no internet (DNS issue, I guess).

Note that all of this started happening after a complete new install of pfSense, following a crash whilst moving to the new Kea DHCP as suggested on the GUI.

It has now taken me three days in trying to resolve this issue. Three days, given my age, likely to be a too big proportion of the rest of my live. It may be alright for most, or youger ones, but its too long for me. I came to pfSense a number of years ago after experiencing too many crashes with Opnsense. I think I may have to start looking at another alternative.

Gertjan

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

(DNS issue, I guess)

Don't guess. Be sure ;) Some tests :

When you start pfSense, the console shows a lot of useful info.
Like these :

....
igc0: <Intel(R) Ethernet Controller I225-V> mem 0x81300000-0x813fffff,0x81400000-0x81403fff at device 0.0 on pci4
...
igc1: <Intel(R) Ethernet Controller I225-V> mem 0x81100000-0x811fffff,0x81200000-0x81203fff at device 0.0 on pci5
...
igc2: <Intel(R) Ethernet Controller I225-V> mem 0x81100000-0x811fffff,0x81200000-0x81203fff at device 0.0 on pci5
....

etc.
I have the "igc" family of network cards, yours might be different. Like 'em'. These lines mean : the kernel found the hardware.

These found interfaces have to be assigned in the GUI :

When booted into pfSense, go console, or SSH - option 8 :

Check what processes are using (listening) on port 67 (DHCP):

[23.09-RELEASE][root@pfSense.bhf.tld]/root: sockstat -4l | grep ':67'
root     kea-dhcp4  23282 19  udp4   192.168.1.1:67        *:*
root     kea-dhcp4  23282 21  udp4   192.168.2.1:67        *:*
root     kea-dhcp4  23282 23  udp4   192.168.100.1:67      *:*

As I have 3 LANs on pfSense, 192.168.1.1/24, 192.168.2.1/24 and 192.168.100.1/24, and these 3 LANs are being set up on my pfSense using "DHCP-server" - or, actually KEA (same thing for me) I know on the pfSense side all is ok.

The same test can be done for the resolver (unbound) (note that I'm testing for IPv4 and IPv6 now :

[23.09-RELEASE][root@pfSense.bhf.tld]/root: sockstat -l | grep ':53'
unbound  unbound    63341 3   udp6   *:53                  *:*
unbound  unbound    63341 4   tcp6   *:53                  *:*
unbound  unbound    63341 5   udp4   *:53                  *:*
unbound  unbound    63341 6   tcp4   *:53                  *:*

which means : for all interfaces (yep, even WAN), for both IPv4 and IPv6, both TCP and UDP ( ! ) unbound listens on port '53'.

If DNS traffic doesn't reach unbound on pfSEnse now, then this can be only one reason : a firewall rule on the interface blocks TCP or UDP port 53 traffic - that this interface.

I presume you don't use smart-switches or something like that.

MidGe48

Hello Gertjan,

OK, I did what you suggested.

On the GUI i have all the interface assignment showing with the correct (I hope) network port.
There are NO processes using or listening on port 67
The resolver (unbound) shows me a single process for tcp4 and udp4. I have disable ipv6 everywhere so I suppose this is correct.

The issue now, is with having no processes listening on port 67! But, now that I have reverted to a backup of two months ago when everything was dandy, I am not using Kea at the moment.

I am using unmanaged switches, so that should not be an issue.

Thanking you very much for trying to help me.

MidGe48

@MidGe48

My boot log shows the ports working as well, I presume:

igb3: netmap queues/slots: TX 4/1024, RX 4/1024
igb3: Ethernet address: 00:e0:67:21:65:5f
igb3: Using MSI-X interrupts with 5 vectors
igb3: Using 4 RX queues 4 TX queues
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: NVM V0.6 imgtype6
igb3: <Intel(R) I210 Flashless (Copper)> port 0xb000-0xb01f mem 0xb1200000-0xb121ffff,0xb1220000-0xb1223fff irq 19 at device 0.0 on pci4
pci4: <ACPI PCI bus> on pcib4
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
igb2: Ethernet address: 00:e0:67:21:65:5e
igb2: Using MSI-X interrupts with 5 vectors
igb2: Using 4 RX queues 4 TX queues
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: NVM V0.6 imgtype6
igb2: <Intel(R) I210 Flashless (Copper)> port 0xc000-0xc01f mem 0xb1300000-0xb131ffff,0xb1320000-0xb1323fff irq 18 at device 0.0 on pci3
pci3: <ACPI PCI bus> on pcib3
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: Ethernet address: 00:e0:67:21:65:5d
igb1: Using MSI-X interrupts with 5 vectors
igb1: Using 4 RX queues 4 TX queues
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: NVM V0.6 imgtype6
igb1: <Intel(R) I210 Flashless (Copper)> port 0xd000-0xd01f mem 0xb1400000-0xb141ffff,0xb1420000-0xb1423fff irq 17 at device 0.0 on pci2
pci2: <ACPI PCI bus> on pcib2
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb0: Ethernet address: 00:e0:67:21:65:5c
igb0: Using MSI-X interrupts with 5 vectors
igb0: Using 4 RX queues 4 TX queues
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: NVM V0.6 imgtype6
igb0: <Intel(R) I210 Flashless (Copper)> port 0xe000-0xe01f mem 0xb1500000-0xb151ffff,0xb1520000-0xb1523fff irq 16 at device 0.0 on pci1
pci1: <ACPI PCI bus> on pcib1
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0

Gertjan

@MidGe48

Good news : your device has supported network hardware.
Not really a surprise, as it worked before.

Btw : you are using the pfSense CE 2.7.1, right ?

What did the sockstat test show ?

MidGe48

@Gertjan

Yes, using pfSense CE 2.7.1

On the GUI i have all the interface assignment showing with the correct (I hope) network port.

There are NO processes using or listening on port 67

The resolver (unbound) shows me a single process for tcp4 and udp4. I have disable ipv6 everywhere so I suppose this is correct.

Gertjan

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

There are NO processes using or listening on port 67

that means : no DHCP server is serving 'DCP leases'.
Check the system log and dhcp log page ( Status > System Logs> DHCP ), and see what happened.

Yo said ; "Another OPT1 and OPT2 no internet connection!" so I dedecut that LAN is ok.
But without a DHCP server running on LAN, you should also have DHCP issues on your LAN interface.
Except if you have shut down the DHCP server on your LAN .....

If there is no DHCP servers running on a LAN, or OPT or OPT2, etc then it's normal that DHCP clients, your devices, ,don't get any DHCP lease => NO IP, no gatway, no DNS, so no connection what so ever.

Btw : If you were using dhcpd-isc (not KEA) ;

[23.09-RELEASE][root@pfSense.bhf.tld]/root: sockstat -4l | grep 'dhcpd'
dhcpd    dhcpd      76530 13  udp4   *:67                  *:*

which means that there is one process listing to 'all' interfaces.
Using UDPv4 of course, on port 67.

MidGe48

@Gertjan

[2.7.1-RELEASE][admin@trakka.midge.home.arpa]/root: sockstat -4l | grep 'dhcpd'
dhcpd dhcpd 32156 13 udp4 *:48463

So, no TCP?!

Unfortunately I cannot bring my network down now. There are important, to me, processes running, and the router is my perimeter firewall and router.. I will restart the router and pull the logs of boot, system general, firewall and dhcp a few minutes after a reboot. That will be in about 12 -13 hours from now.

The log are so long now and I can't quite remember at which times I made what changes. The only thing I can say is that they look normal.

stephenw10

DHCP is a UDP protocol, it shouldn't be listening for TCP.

But why yours is not listening on port 67 is very odd.

You shouldn't need to change the dhcp client settings on the hosts and in fact setting it to 'address only' is probably going to break stuff as they won't be passed dns or time servers etc.

Do you see leases in Status > DHCP Leases? Perhaps you have a rogue DHCP server somewhere else.

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

the correct (I hope) network port

You need to know they are the correct ports because they may be the cause of all the problems here. Check with ifconfig as you connect or disconnect links.

Steve

Gertjan

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

dhcpd dhcpd 32156 13 udp4 *:48463

As above : this explains everything.

A dhcpd server has to listen on port '67' - UDP.
Bacause DHCP clients will use that port '67' as their destination.

Why your dhcpd uses port 48463 is .... never seen that before.

I've checked my dhcpd config file ( here /var/dhcpd/etc/dhcpd.conf ) and 'use port 67' command isn't even an option or mentioned, it the the de facto standard.
Looking at the GUI => Services > DHCP Server > LAN page, I couldn't even find a settings that permits me to change that.

If dhcpd elected to use another port, like your 48463, it should at least mention this its log file.
Just restart it, and look the the DHCP logs
Don't worry to break your networks, they are already pretty 'down' any way.

DHCP functionality is of course non mandatory, I can image that if you set up all network devices with a static IP, static gateway, and static DNS, and the network (/24) used, you don't any DHCP server to work.

stephenw10

Indeed I've never seen that before. Speculation but perhaps it uses a random port if 67 is in use? Maybe Kea was somehow still running?
I've switched back and forth between ISC and Kea many times and never managed to trigger anything like that but....

I would restart the dhcpd service. Failing that reboot.

Steve

MidGe48

@Gertjan @stephenw10

OK, I re-installed 4 times from the downloaded image.

I finally found out about one problem and got that fixed on the last and current install.

sockstat -l | grep ':53'

unbound unbound 4119 3 udp6 *:53 :
unbound unbound 4119 4 tcp6 *:53 :
unbound unbound 4119 5 udp4 *:53 :
unbound unbound 4119 6 tcp4 *:53 :

sockstat -4l | grep ':67'

root kea-dhcp4 7840 19 udp4 10.10.10.1:67 :
root kea-dhcp4 7840 21 udp4 10.88.88.1:67 :
root kea-dhcp4 7840 23 udp4 10.44.44.1:67 :

But, whereas everything works fine on LAN, OPT1 and OPT2 are still giving me some problems.

They both serve me via DHCP, the same subnet when they are different in the settings.

johnpoz

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

They both serve me via DHCP, the same subnet when they are different in the settings.

You don't have your networks actually isolated then..

You can't just plug into the same dumb switch.

MidGe48

@johnpoz

It is not a switch it is a pfSense router with different physical ports assigned to different interfaces, different DHCP servers and different subnets. So where in pfSense do I see/change the isolation??

johnpoz

@MidGe48 what do you have plugged into pfsense interfaces? Your saying you have a single device plugged into Opt1 and it gets an IP from dhcp on a lan or opt2 network?

What plugs into pfsense interfaces where you setup these networks and are running dhcp.. You can not just plug in lan, opt and opt2 into the same dumb switch.. And then plug a PC into one of the other ports.

MidGe48

@johnpoz

I am well aware of what you say.

I have a relatively complex topology for my whole network including 3 routers and 4 switches. But what plugs in the ports of my perimeter router (pfsense) are physical networks totally isolated from each other. I don't know where you got the idea that I was converging things via a dumb switch.

Anyway, I am done for now, here.

I have other tasks awaiting and will use a workaround for the time being.

Thanks all.

johnpoz

@MidGe48 said in Yes, I know! Another OPT1 and OPT2 no internet connection!:

totally isolated from each other. I

Clearly they are not - because there is NO possible way for you to get a dhcp address from some other network that is on different layer 2 network.

Or your network is so complex you don't know what is plugged into where, and what you think is connected to opt1 is connected to lan or opt2 or whatever network you device is suppose to get dhcp from opt1 is on