Single NIC machine not getting WAN IP with PFsense

KFaust

Hi all, I am new to pfsense and am attempting to get it working on a small form server with a single NIC through the use of VLANs and a managed switch. All the setup and VLAN parts appear to be working right, but i'm not getting a WAN IP address assigned and therefore no internet access.

Here is the hardware I am using:

PC
CPU - AMD Ryzen 3 2200G
Motherboard - Gigabyte AB350N Gaming Wifi
Memory - 16 GB DDR4

Switch
TL-SG105E

Router
TP-Link AX 1800 WiFi 6

Modem
DOCSIS 3.1 EU2251 (Spectrum ISP)

My goal is to get pfsense working succesfully when connected to my modem and set the router into bridge mode and use it as a wireless access point only. Here's some screenshots of my setup in both the pfsense and switch management screens:

VLAN setup in PFsense:

VLAN 10 setup in TL-SG105E:

VLAN 99 setup in TL-SG105E:

PVIDs in TL-SG105E:

When I disconnect the modem from the TP-Link router, reboot it, then plug it into the correct port on the TL-SG105E, I eventually see this on PFsense:

Incoming traffic on WAN but no outgoing, and no assigned IP (it just stays at 0.0.0.0).

Here's some output from a data capture I ran on the WAN connection:

I don't really know what this means, but hopefully someone can help me understand it. Ran 100 packets and they're all queries like this. Here's a detailed description of a packet if it helps:

I've tried switching the interfaces around in PFsense and the switch, rebooting both the cable modem and pfsense, and re-installing everything and starting again, nothing seems to get around this issue. I already had my own router, but I contacted my ISP and they said I don't need to give them anything if I'm changing routers. Hoping someone here can advise next steps based on everything i've provided.

stephenw10

I assume the LAN VLAN works as expected?

The switch config looks fine. I would certainly reboot the modem if you have not.

Try connecting, say, a laptop to the modem directly and make sure that can pull a DHCP lease.
If that fails check the TP-Link router is not using something additional like a VLAN on the WAN.

Are you running pfSense 2.6? You test a 2.7 snapshot. If a laptop does pull a lease you might be hitting this: https://redmine.pfsense.org/issues/12070
To verify that you would need to run a packet capture directly on re0 for the DHCP packets.

Steve

johnpoz

@kfaust where exactly is the modem connected too? Where is pfsense?

You have multiple untagged vlans on ports..

Vlan 1 is untagged on all the ports, and then you have 10 on 3-5 and 99 on port 2

If the modem is on port 1, your pvid would put untagged on vlan 1, and not 99..

Where you modem comes in should untagged vlan 99, and then port on pfsense should be tagged. Vlan 1 should be removed from these ports. To manage that port you would need untagged on vlan 1. I would prob use untagged for you lan and just leave it on vlan 1.. And then have your modem come in on a tagged vlan 99 with untagged 99 where the modem comes in and then tagged to port connected on pfsense. where your lan is untagged vlan 1. This will allow you to easy manage your switch from your lan devices.

stephenw10

Actually forget that VLAN0 issue, the switch would be removing that anyway.

johnpoz

@stephenw10 where is vlan0 coming in - from the modem? I run my modem through a switch, before it gets to pfsense. I run it untagged, on vlan 99 as well ;) But its untagged into pfsense wan port as well.

KFaust

@johnpoz

Sorry, I was so busy getting the screenshots up I forgot to mention where things are hooking up. For reference:

Port 1 is the trunking port receiving tagged traffic for WAN and LAN connections. PFsense is connected to this port.
The cable modem is connected to port 2.
Ports 3-5 are for any wired devices connecting to PFsense. When testing this all out I have a laptop in port 3.

@stephenw10 as far as I can tell the VLANs are functioning correctly, I've been through this enough times I'm pretty confident that's not the issue and something else with DHCP leasing is going on. Connecting a laptop to the cable modem directly DOES yield a lease and internet access, so it seems to just be pfsense that's being treated differently.

I'll look at the issue linked when I have time to review it fully, but at a glance it seems to be about fiber connection? My ISP isn't providing fiber.

Jarhead

@kfaust Did you power cycle the modem?
Cable modems "remember" the directly connected MAC and need to be power cycled to forget it when changing that directly connected device.

johnpoz

@kfaust yeah as @Jarhead mentions when you change the connection to a cable modem you need to power cycle it.

So for example you connect your laptop and get an IP... You would then need to power cycle the modem before connecting it to your switch and then pfsense.

KFaust

@johnpoz @Jarhead I've power cycled the modem, yes, and pfsense still doesn't get an IP in this setup. I'll try removing some complexity and keep the modem on a VLAN 99 (And also removing the port it connects on from VLAN 1) and have the LAN just use VLAN 1 as suggested, then see if I can get an IP. I can usually test this in the mornings before my wife is up since it requires taking down the internet, but i'll report what my results.

johnpoz

@kfaust well the nice thing about having the connection go through the switch is via span/mirror port setting you can easy see the dhcp traffic..

if you have no luck tmrw - that would prob be the next logical step to see what is actually going on.

stephenw10

Failing to pull a dhcp lease because the ISP is sending back all the dhcp responses priority tagged and pfSense 2.6 just drops those is a (relatively) common failure.
However I didn't think Spectrum was one of those.
And the switch being configured in 802.1Q mode like that would strip those tags anyway.

Steve

KFaust

@johnpoz this morning I made the following changes to my configuration:

In pfsense I deleted VLAN 10 and assigned the LAN interface directly to the physical NIC.

Also deleted VLAN 10 in the switch interface and put all ports back on VLAN 1, except for port 2 which was removed and isolated to VLAN 99:

PVIDs are also set to 1 for all ports except 2, which remains the same as in the original post.

Rebooting the cable modem and connecting it to port 2, I still wasn't able to get an IP assigned to WAN:

Here are the port stats when all pfsense, modem, and laptop are all connected to their respective ports, IDK if relevant:

(I'm going back and forth between the web interface and management app for the switch here, that's why the screenshots look different).

I set up port mirroring on port 3 and selected ports 1 and 2 to mirror to try and capture all traffic between PFsense and the modem with wireshark. I haven't done this before and am not experienced with wireshark so i'm not sure if I did something wrong here, but wireshark only captures 1 packet when I have it monitor the laptop's ethernet connection:

So that's where i'm at this morning, any other suggestions appreciated.

stephenw10

With re0 assigned you should be able to pcap on that in promiscuous mode and see the VLAN tagged traffic on it also. Filter by port 67 and you should see the dhcp requests and responses (or lack of) on WAN.

KFaust

Okay, I had pfsense run another package capture with promiscuous mode enabled and filtered for port 67. Captured 18 packets, here's what it looks like:

All the packets pretty much look like this, a longer time interval passes between each one as well (MAC address removed):

Frame 3: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Apr 30, 2023 05:28:24.169625000 Central Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1682850504.169625000 seconds
    [Time delta from previous captured frame: 1.018097000 seconds]
    [Time delta from previous displayed frame: 1.018097000 seconds]
    [Time since reference or first frame: 16.809947000 seconds]
    Frame Number: 3
    Frame Length: 342 bytes (2736 bits)
    Capture Length: 342 bytes (2736 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:dhcp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: [Ethernet MAC Address], Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: [Ethernet MAC Address]
        Address: [Ethernet MAC Address]
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
    Total Length: 328
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x3996 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 0.0.0.0
    Destination Address: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 308
    Checksum: 0xa2ea [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
    UDP payload (300 bytes)
Dynamic Host Configuration Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x78581545
    Seconds elapsed: 1
    Bootp flags: 0x0000 (Unicast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: [Ethernet MAC Address]
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
        Length: 1
        DHCP: Discover (1)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet (0x01)
        Client MAC address: [Ethernet MAC Address]
    Option: (12) Host Name
        Length: 7
        Host Name: pfSense
    Option: (55) Parameter Request List
        Length: 10
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (26) Interface MTU
    Option: (255) End
        Option End: 255
    Padding: 0000000000000000000000000000000000000000000000000000

johnpoz

@kfaust yeah that is pfsense sending discover - asking for lease, and you don't seem to be getting any answers, which would explain why no IP address.

stephenw10

That was a pcap on LAN, re0 directly?

I expect to see those DHCP packets tagged with VLAN99 if so.

KFaust

@stephenw10

Whoops, no, that was pcap on the WAN interface. I'm out of time to troubleshoot this today as people are getting up now, so i'll have to try running it that way tomorrow.

Assuming there's not a huge difference though-it seems that would at least confirm that VLAN setup/configuration isn't the issue here? If so, why might pfsense not be getting a DHCP lease from my ISP? FTR I rebooted both the modem and pfsense before connecting them to the switch.

stephenw10

The ISP might require a VLAN or priority tag. Or something additional in the DHCP request. I would expect to see that in the ISP router config though. If they give you access to that.

Connecting a laptop to the modem dircetly to make sure that pulls a lease would rule that out.

KFaust

Alright, i'm back with some more info from this morning's troubleshooting.

First off, here's a packet for a DHCP lease request from running a packet trace on the LAN interface in PFsense:

Frame 12533: 346 bytes on wire (2768 bits), 346 bytes captured (2768 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  1, 2023 02:58:46.771593000 Central Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1682927926.771593000 seconds
    [Time delta from previous captured frame: 0.004210000 seconds]
    [Time delta from previous displayed frame: 22.820040000 seconds]
    [Time since reference or first frame: 40.044565000 seconds]
    Frame Number: 12533
    Frame Length: 346 bytes (2768 bits)
    Capture Length: 346 bytes (2768 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:vlan:ethertype:ip:udp:dhcp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: [Ethernet MAC], Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: Giga-Byt_6d:ec:e4 [Ethernet MAC]
        Address: Giga-Byt_6d:ec:e4 [Ethernet MAC]
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 99
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0110 0011 = ID: 99
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
    Total Length: 328
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x3996 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 0.0.0.0
    Destination Address: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 308
    Checksum: 0xafde [unverified]
    [Checksum Status: Unverified]
    [Stream index: 5]
    [Timestamps]
    UDP payload (300 bytes)
Dynamic Host Configuration Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x06f679b4
    Seconds elapsed: 0
    Bootp flags: 0x0000 (Unicast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: Giga-Byt_6d:ec:e4 [Ethernet MAC]
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
        Length: 1
        DHCP: Discover (1)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet (0x01)
        Client MAC address: [Ethernet MAC]
    Option: (12) Host Name
        Length: 7
        Host Name: pfSense
    Option: (55) Parameter Request List
        Length: 10
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (26) Interface MTU
    Option: (255) End
        Option End: 255
    Padding: 0000000000000000000000000000000000000000000000000000

Note: Pfsense's diagnostic tool actually didn't report any packets when I tried running a trace on port 67 for the LAN. I had to remove that setting and then filter for DHCP in wireshark to see a list of requests. Including that in case it means something.

It just keeps making these lease requests forever without getting a response. It actually does this even if the modem isn't plugged into the (I guess the switch can tell if there's something in the port but pfsense can't because it's a VLAN on a connected NIC?)

The VLAN tag is in the packet there, so that seems like it's working correctly. Plugging the laptop to the modem directly got an immediate IP assignment as well:

Laptop=DHCP.PNG

So it really seems like pfsense is getting ignored for some reason. At this point i'm scratching my head on why the ISP won't assign this machine a lease when it gives one to everything else.

One thing I noticed that seemed of interest to me is that the IP pfsense assigns to my laptop when it's connected to the switch looks a bit weird:

Pfsense is at 192.168.1.1 so i'd expect the default gateway to match, but instead it's IPv6 only. I was expecting the config to look similar to when the laptop is connected to the Archer AX1800:

Just to cover my bases, I disconnected the Archer from the modem and rebooted it before connecting the laptop. Same result.

At this point I'm not sure what else to explore for why pfense isn't getting a WAN IP. Rebooting the modem doesn't do it, and multiple devices don't have an issue pulling a lease from the ISP. I'll probably try to call my ISP in the morning when I have time and see if they can see anything from their end while i'm trying to connect pfsense if no one else has any ideas. Appreciate all the thoughts and feedback so far.

stephenw10

As a test try just assigning re0 as WAN directly and connect it to the modem. Remove the VLAN and switch entirely as a possibility.