Single NIC machine not getting WAN IP with PFsense

KFaust

Okay, I had pfsense run another package capture with promiscuous mode enabled and filtered for port 67. Captured 18 packets, here's what it looks like:

All the packets pretty much look like this, a longer time interval passes between each one as well (MAC address removed):

Frame 3: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Apr 30, 2023 05:28:24.169625000 Central Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1682850504.169625000 seconds
    [Time delta from previous captured frame: 1.018097000 seconds]
    [Time delta from previous displayed frame: 1.018097000 seconds]
    [Time since reference or first frame: 16.809947000 seconds]
    Frame Number: 3
    Frame Length: 342 bytes (2736 bits)
    Capture Length: 342 bytes (2736 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:dhcp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: [Ethernet MAC Address], Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: [Ethernet MAC Address]
        Address: [Ethernet MAC Address]
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
    Total Length: 328
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x3996 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 0.0.0.0
    Destination Address: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 308
    Checksum: 0xa2ea [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
    UDP payload (300 bytes)
Dynamic Host Configuration Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x78581545
    Seconds elapsed: 1
    Bootp flags: 0x0000 (Unicast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: [Ethernet MAC Address]
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
        Length: 1
        DHCP: Discover (1)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet (0x01)
        Client MAC address: [Ethernet MAC Address]
    Option: (12) Host Name
        Length: 7
        Host Name: pfSense
    Option: (55) Parameter Request List
        Length: 10
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (26) Interface MTU
    Option: (255) End
        Option End: 255
    Padding: 0000000000000000000000000000000000000000000000000000

johnpoz

@kfaust yeah that is pfsense sending discover - asking for lease, and you don't seem to be getting any answers, which would explain why no IP address.

stephenw10

That was a pcap on LAN, re0 directly?

I expect to see those DHCP packets tagged with VLAN99 if so.

KFaust

@stephenw10

Whoops, no, that was pcap on the WAN interface. I'm out of time to troubleshoot this today as people are getting up now, so i'll have to try running it that way tomorrow.

Assuming there's not a huge difference though-it seems that would at least confirm that VLAN setup/configuration isn't the issue here? If so, why might pfsense not be getting a DHCP lease from my ISP? FTR I rebooted both the modem and pfsense before connecting them to the switch.

stephenw10

The ISP might require a VLAN or priority tag. Or something additional in the DHCP request. I would expect to see that in the ISP router config though. If they give you access to that.

Connecting a laptop to the modem dircetly to make sure that pulls a lease would rule that out.

KFaust

Alright, i'm back with some more info from this morning's troubleshooting.

First off, here's a packet for a DHCP lease request from running a packet trace on the LAN interface in PFsense:

Frame 12533: 346 bytes on wire (2768 bits), 346 bytes captured (2768 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  1, 2023 02:58:46.771593000 Central Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1682927926.771593000 seconds
    [Time delta from previous captured frame: 0.004210000 seconds]
    [Time delta from previous displayed frame: 22.820040000 seconds]
    [Time since reference or first frame: 40.044565000 seconds]
    Frame Number: 12533
    Frame Length: 346 bytes (2768 bits)
    Capture Length: 346 bytes (2768 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:vlan:ethertype:ip:udp:dhcp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: [Ethernet MAC], Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: Giga-Byt_6d:ec:e4 [Ethernet MAC]
        Address: Giga-Byt_6d:ec:e4 [Ethernet MAC]
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 99
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0110 0011 = ID: 99
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
    Total Length: 328
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x3996 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 0.0.0.0
    Destination Address: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 308
    Checksum: 0xafde [unverified]
    [Checksum Status: Unverified]
    [Stream index: 5]
    [Timestamps]
    UDP payload (300 bytes)
Dynamic Host Configuration Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x06f679b4
    Seconds elapsed: 0
    Bootp flags: 0x0000 (Unicast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: Giga-Byt_6d:ec:e4 [Ethernet MAC]
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
        Length: 1
        DHCP: Discover (1)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet (0x01)
        Client MAC address: [Ethernet MAC]
    Option: (12) Host Name
        Length: 7
        Host Name: pfSense
    Option: (55) Parameter Request List
        Length: 10
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (26) Interface MTU
    Option: (255) End
        Option End: 255
    Padding: 0000000000000000000000000000000000000000000000000000

Note: Pfsense's diagnostic tool actually didn't report any packets when I tried running a trace on port 67 for the LAN. I had to remove that setting and then filter for DHCP in wireshark to see a list of requests. Including that in case it means something.

It just keeps making these lease requests forever without getting a response. It actually does this even if the modem isn't plugged into the (I guess the switch can tell if there's something in the port but pfsense can't because it's a VLAN on a connected NIC?)

The VLAN tag is in the packet there, so that seems like it's working correctly. Plugging the laptop to the modem directly got an immediate IP assignment as well:

Laptop=DHCP.PNG

So it really seems like pfsense is getting ignored for some reason. At this point i'm scratching my head on why the ISP won't assign this machine a lease when it gives one to everything else.

One thing I noticed that seemed of interest to me is that the IP pfsense assigns to my laptop when it's connected to the switch looks a bit weird:

Pfsense is at 192.168.1.1 so i'd expect the default gateway to match, but instead it's IPv6 only. I was expecting the config to look similar to when the laptop is connected to the Archer AX1800:

Just to cover my bases, I disconnected the Archer from the modem and rebooted it before connecting the laptop. Same result.

At this point I'm not sure what else to explore for why pfense isn't getting a WAN IP. Rebooting the modem doesn't do it, and multiple devices don't have an issue pulling a lease from the ISP. I'll probably try to call my ISP in the morning when I have time and see if they can see anything from their end while i'm trying to connect pfsense if no one else has any ideas. Appreciate all the thoughts and feedback so far.

stephenw10

As a test try just assigning re0 as WAN directly and connect it to the modem. Remove the VLAN and switch entirely as a possibility.

KFaust

@stephenw10 thanks for suggesting this, when I assigned re0 to the WAN and connected the machine running pfsense directly to the modem it pulled an IP in seconds. The issume does seem to be in the switch/VLAN, though it's weird because i've followed steps other people have done exactly and it doesn't work. From my captures I can see that DHCP discovers pfsense is sending has the VLAN tag on it, so I'm wondering if tag somehow isn't getting stripped when it leaves the switch.

stephenw10

@kfaust Yes or the replies are not being tagged coming back in. That would be the PVID on the port. It looks correct but maybe never applied or similar?

You could just use WAN untagged and LAN tagged and reconfigure the switch to match that. But that would likely expose the switch GUI on the WAN. So...sub optimal!

johnpoz

@kfaust said in Single NIC machine not getting WAN IP with PFsense:

sending has the VLAN tag on it,

Well yeah its tagged on that port so you would need to send the tag.. But on the port connected to the modem it should be untagged on vlan 99, and yes the pvid should be 99.. So the answer to your dhcp would hit the port on the switch and the switch would say oh this is vlan 99, and send it on to pfsense tagged on 99.

stephenw10

The switch config posted in screenshots above looks correct. It is a TPLink switch of course....

KFaust

Well if nothing else I've learned a good bit about wireshark monitoring! I spent the whole evening just figuring out how keep my windows laptop from stripping the tags off the packet when trying to monitor them with port mirroring, but I finally got it.

That said, it does appear the switch is correctly applying and removing the tag when the DHCP request ingresses on port 1 and egresses from port 2. Good to know, but it doesn't exactly solve my problem. The only other thing I can think is that if the response from the modem is getting dropped when it enters the switch somehow.

As one last attempt, I do have a managed netgear switch lying around with same number of ports. I'll swap them out next chance I get and see if that makes a difference.

stephenw10

Yes, if the PVID wasn't applying for some reason the replies from the ISP would not make it back onto the VLAN.
However it looks to be correctly configured and LAN was working as expected when you had that configured as VLAN 10 in the same way.

If you still have the pcap of your laptop pulling a lease successfully, or pfSense doing so on re0, then you might look at those reply packets and see if they have some other tags. Every cheap switch I have seen just strips priority tags but it's possible this one uses them and drops the packets. Does it have tagging enabled on the QoS tab?

Steve

KFaust

Made some more head way this morning. I need to amend my previous statement:

That said, it does appear the switch is correctly applying and removing the tag when the DHCP request ingresses on port 1 and egresses from port 2.

This true, but I wasn't taking these packet captures while connected to the modem. Since I can't take the modem down during the day, I moved my set up and connected my desktop to the port where the modem would usually go, then mirrored/monitored port traffic in my laptop. In this scenario, pfsense was sending tagged DHCP requests through port 1 and they were passing through port 2 with the tag stripped off, all correct.

I tried this again in the morning with the modem connected, and found that the DHCP requests aren't being routed after ingressing on port 1. Just to be sure, I connected my PC again and verified everything by monitoring just the port 1 ingress, then egress, and the same for port 2 . I then did the same with the modem, also setting up capture on pfsense to track the packets it's sending to port 1, then specifically monitoring port 1 egress on the laptop.

All that's to say: when the modem is connect to the switch, PFsense will send a VLAN-tagged DHCP Request (ID: 99) to the switch, but the switch does not route the package anywhere, it never leaves the port and thus never reaches the modem. If I change the connected end device to desktop PC, the packages suddenly route correctly. Same VLAN/PVID setup I've already demonstrated in this thread. No idea what could cause this, but i'll keep researching while awaiting any ideas here.

stephenw10

Hmm, does the modem have some odd MAC address perhaps?

It shouldn't matter though broadcast traffic should be sent everywhere.

You might try passing the untagged traffic from re0 directly through the switch and see if that fails.
Either with the switch just passing it as untagged on a VLAN or in port vlan mode if it has that.

Steve

KFaust

There's truly no accounting for the physical layer. I tried it with my netgate switch the morning and pfsense pulled a WAN IP on VLAN 99 almost instantly just as every guide I've followed would expect. The VLAN/PVID setup is exactly the same, all that changed is a the switch. Maybe some weird interaction between the TP-Link switch and the cable modem? In any case, I'm finally unblocked on this and can start moving forward with everything else.

johnpoz

@kfaust said in Single NIC machine not getting WAN IP with PFsense:

TP-Link switch

Not on the top of my recommendation lists for low cost switches.. Ever since their fisaco with vlan 1 not being able to be removed, and it taking them like 2 years to fix it.. And them saying on their own forums that it was normal and required, etc. it was clear they don't actually understand vlans..

They did finally fix it, but they didn't officially back port the firmware change to the older models, etc.. I have one with the fixed firmware in use, its behind my tv and have some vlans on it and not had any issues, and it does let me remove vlan 1 now. And some of the leak testing I did it seems to be actually isolating the vlans.. But I wouldn't buy that brand if there was any other choice.. At least not their entry level smart switches.

stephenw10

Hmm, yeah TPLink have had some... interesting.... switch issues in older models but nothing that would really explain this. Weird. Glad you were able to resolve it.

johnpoz

@stephenw10 yeah its odd for sure - what I would do for testing and validation of the problem would be to duplicate dhcp and vlan without the modem.

Fire up something running dhcp and let pfsense pretend its a wan interface getting dhcp and run it through the switch with the same sort of setup. untagged towards your dhcp server/gateway for pfsense, and tagged towards pfsense..

You would use anything for the test, some laptop or pi would work just fine for such testing..