VLAN tag on WAN not working



  • My ISP (Fieber, Netherlands) supplied me with a Draytek router, this has always worked well, but i want better VPN support so am now replacing it with a pfsense box (QOTOM). i have take all the settings from the Draytek and set up the WAN the same way (except for the VLAN bit) but can't get it to work.

    ISP says WAN needs to have VLAN tag ID 128. In the Draytek modem this was just a simple text field to fill in "WAN VLAN SETTING - WAN VLAN ID". I have searched these forums and googled but have not found anything that works for me. MAC address is spoofed with the WAN MAC of the Draytek router. I have tried creating a VLAN with tag ID 128 on igb0 and connected that with WAN, doing nothing else with the igb0 interface. This didn't work either.

    Has anyone with similar conditions from ISP have a working setup with pfsense?

    thanks for any help or nudges in the right direction.




  • Netgate Administrator

    If they only need tagged traffic then creating a VLAN on igb0 and assigning that as WAN should work.

    What are you actually seeing? Should it pull an IP via DHCP? What do the logs show?

    Steve



  • stephenw, thanks for the reply. i had already tried what you suggested (and just tried again). from what i glanced from other posts that seemed to be what was needed, but i wasn't sure if that was it, since it didn't work for me.

    i did a packet capture on wan to see what the dhcp request packets look like, but don't actually have any idea what they should look like, so i can't see if anything is wrong there. in case someone else does, below is the captured data:

    Frame 13: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Jan 20, 2009 03:37:29.013311000 CET
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1232419049.013311000 seconds
    [Time delta from previous captured frame: 6.179544000 seconds]
    [Time delta from previous displayed frame: 6.179544000 seconds]
    [Time since reference or first frame: 47.215069000 seconds]
    Frame Number: 13
    Frame Length: 342 bytes (2736 bits)
    Capture Length: 342 bytes (2736 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:bootp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
    Ethernet II, Src: Draytek_03:7b:xx (00:1d:aa:03:7b:xx), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: Draytek_03:7b:xx (00:1d:aa:03:7b:xx)
    Type: IPv4 (0x0800)
    Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
    0001 00.. = Differentiated Services Codepoint: Unknown (4)
    .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 328
    Identification: 0x0000 (0)
    Flags: 0x00
    0... .... = Reserved bit: Not set
    .0.. .... = Don't fragment: Not set
    ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 128
    Protocol: UDP (17)
    Header checksum: 0x3996 [validation disabled]
    [Header checksum status: Unverified]
    Source: 0.0.0.0
    Destination: 255.255.255.255
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
    User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 308
    Checksum: 0xb295 [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    Bootstrap Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x9c4a8303
    Seconds elapsed: 28
    Bootp flags: 0x0000 (Unicast)
    0... .... .... .... = Broadcast flag: Unicast
    .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: Draytek_03:7b:xx (00:1d:aa:03:7b:xx)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
    Length: 1
    DHCP: Discover (1)
    Option: (61) Client identifier
    Length: 7
    Hardware type: Ethernet (0x01)
    Client MAC address: Draytek_03:7b:xx (00:1d:aa:03:7b:xx)
    Option: (12) Host Name
    Length: 9
    Host Name: pfsense
    Option: (55) Parameter Request List
    Length: 9
    Parameter Request List Item: (1) Subnet Mask
    Parameter Request List Item: (28) Broadcast Address
    Parameter Request List Item: (2) Time Offset
    Parameter Request List Item: (121) Classless Static Route
    Parameter Request List Item: (3) Router
    Parameter Request List Item: (15) Domain Name
    Parameter Request List Item: (6) Domain Name Server
    Parameter Request List Item: (12) Host Name
    Parameter Request List Item: (119) Domain Search
    Option: (255) End
    Option End: 255
    Padding: 000000000000000000000000000000000000000000000000...


  • Netgate Administrator

    I don't see any VLAN information in that packet. Did you capture on the VLAN interface itself?

    If so try assigning igb0 also and pcap on that in promiscuous mode. You should see the VLAN tagging in the packet you're sending and whatever the ISP is sending back.

    Steve



  • thanks for the info. i will try that and hopefully have more info to get it working.

    thanks



  • i did the capture in pfsense itself (Diagnostics -> Packet Capture). selected WAN (doesn't allow to select port of virtual port), and WAN is conntected on igb0 on VLAN 128. i redid the capture and it is the same. not sure if pfsense captures before tagging or maybe i set it up wrong. Will try it again tonight when no one should be using anything behind the router. thanks for the help.

    0_1537779674115_Screenshot_20180924_082725.png

    0_1537779694988_Screenshot_20180924_082742.png


  • Netgate Administrator

    Right. You need to assign igb0 as another interface so that it is available to pcap on. That will show the VLAN tagged traffic on it.

    Steve



  • thanks. i added igb0 to OPT3, enabled OPT3 and then did the capture on there and i can see the vlan tag. it looks ok. i will contact the isp to release the MAC and try to connect without spoofing the mac and see if that works.


  • Netgate Administrator

    Sounds like a good plan.
    Did you see traffic coming back from them at all? On the correct VLAN?

    Steve



  • no traffic coming back at all. i am going to try and capture some packets form the draytek and see what is different



  • managed to capture dhcp discover packets from draytek, there are some differences, will look into trying to make pfsense do the same

    draytek modem:
    Option: (55) Parameter Request List
    Length: 11
    Parameter Request List Item: (1) Subnet Mask
    Parameter Request List Item: (3) Router
    Parameter Request List Item: (6) Domain Name Server
    Parameter Request List Item: (12) Host Name
    Parameter Request List Item: (15) Domain Name
    Parameter Request List Item: (28) Broadcast Address
    Parameter Request List Item: (42) Network Time Protocol Servers
    Parameter Request List Item: (121) Classless Static Route
    Parameter Request List Item: (33) Static Route
    Parameter Request List Item: (212) 6RD
    Parameter Request List Item: (150) TFTP Server Address

    pfsense:
    Option: (55) Parameter Request List
    Length: 9
    Parameter Request List Item: (1) Subnet Mask
    Parameter Request List Item: (28) Broadcast Address
    Parameter Request List Item: (2) Time Offset
    Parameter Request List Item: (121) Classless Static Route
    Parameter Request List Item: (3) Router
    Parameter Request List Item: (15) Domain Name
    Parameter Request List Item: (6) Domain Name Server
    Parameter Request List Item: (12) Host Name
    Parameter Request List Item: (119) Domain Search


  • Netgate Administrator

    The thing I would be looking for if that doesn't work is packets tagged with a VLAN PRIO number or some other QoS type marking. We seen other ISPs require that.

    Steve



  • still stumped. pfsense was missing option-60 vendor-class-identification (googling showed some ISPs require this), so i added this and set it to exactly the same as on the draytek, spoofed the mac, made sure hostname is also spoofed. still not working. vlan id and prio are the same. it seems the packets are not reaching the isp at all or i would at least see some packets coming back.
    the only difference now is the request list, but so far i can't get it to work. when i put something in pfsense gui request options field, it doesn't send option 55 at all anymore. i still have to try further, maybe i am doing it wrong.
    but the only difference between pfsense parameter request list en draytek's is that the draytek asks for a few more things: static route, 6rd and tftp server address. the rest is the same as pfsense.

    i have tried entering "Subnet Mask, Router, Domain Name Server, Host Name, Domain Name, Broadcast Address, Network Time Protocol Servers, Classless Static Route, Static Route, 6RD, TFTP Server Address", "subnet-mask, router, domain-name-server, host-name, domain-name, broadcast-address, network-time-protocol-servers, classless-static-route, static-route, 6rd, tftp-server-address" and just "router" in the "Request Options" field in WAN interface page of pfsense, but each time the packet capture showed de dhcp discover without any Option 55.

    next test I'll scrounge some hardware and install a squeeky clean pfsense on it and see it it work on that. ISP tech support doesn't know of any special settings that they require, just plain DHCP they told me. But that's tech support, so it could mean anything.

    thanks for the help and suggestions so far



  • ok, so more info now after a few more nights/mornings of testing.

    i am running pfsense 2.4.4 on this install on a Qotom Q515G6 3865U (6 ports, intel gigabit), which doesn't seem to want to get the ip on wan through dhcp and vlan.
    so i installed pfsense 2.4.3 on a PC-Engine APC2C4 (3 ports, intel) and got the same result.
    so i thought i'd try opnsense and installed that on a spare Gigabyte BRIX J1900 which only has one ethernet port (realtek) and i hooked up an usb ethernet dongle (also realtek) for LAN. this actually worked, so I assumed there must be something wrong in the way pfsense translates configuration into dhcp packets. Then i went about and installed opsense on the QOTOM and guess what.... doesn't work. Then i installed pfsense 2.3.5 (2.4.4 would not install, it hung at "booting..." trying to install via USB memstick) on the BRIX and wan instantly gets IP from dhcp on WAN.

    So it seems there must be some hardware thing with boards that have more than 1 nic that is preventing it from receiving the dhcp offer packets?

    hoping someone has run into this kind of problem before and has a solution....


  • Netgate Administrator

    Hmm, well that's an interesting set of results. You do have a variety of NIC types in there though. It looks like you have not had any success in any OS using an Intel NIC? Are those all igb?

    If it appears to hang at "Booting..." you are probably hitting the console error referred to here:
    https://www.netgate.com/docs/pfsense/install/upgrade-guide.html?highlight=upgrade#upgrading-from-versions-older-than-pfsense-2-4-4

    Booting the installer UEFI is probably easiest on a BRIX to get past that.

    Although we tried pcaping in promiscuous mode previously it would be worth trying to set the NIC in promiscuous mode permanently to see it that prevents it dropping packets (if that's what's happening). At the command line:
    ifconfig igb0 promisc

    If you run ifconfig -v igb0 it would be interesting to see what hardware offloading it had enabled. Similarly with ifconfig -v igb0.128.

    Steve



  • the PC-Engines APU2C4 has Intel i210AT nics and the Qotom Q515G6 3865U uses Intel I211AT nics.

    I tried setting the interfaces to promiscuous mode but that didn't yield any different results. I also tried connecting the usb ethernet dongle to the Qotom and using that for WAN, but that nic didn't support VLANs, somehting pfSense detects and OPNsense doesn't, or OPNsense uses a different driver for the dongle that does support VLANs. Anyways, dongle connected to Qotom with OPNsense didn't get an IP, I was able to set a VLAN on the dongle. On the BRIX i tried swapping WAN and LAN, realtek as LAN and the dongle as WAN, but on the BRIX running pfSense 2.3.5 the dongle ue0 was not selectable.
    Now that it seemed more like a driver problem, I did a search and came up with this bug reference https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209581
    A bug from May 2016, could what they are describing be the cause of the problem I am having?

    flags=UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC
    options=VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,TXCSUM_IPV6



  • i tested with disabling -vlanhwfiler, -vlanhwtso and -vlanhwtag on igb0 but none of these made it work.


  • Netgate Administrator

    Ok then I would try to install 2.4.4 on the BRIX by booting UEFI or upgrading from 2.4.3/2.3.5 and confirm that it works as expected with those interfaces in that version.

    That bug appears unrelated as I don't believe we are using that in a bare metal pfSense install. If you were hitting that the packet capture would show the traffic incorrectly tagged. I assume you setup a mirror port to pcap from the Draytek? Did you also use that see actual packets on the wire to/from pfSense?

    Steve



  • ok, more testing.

    i upgraded the BRIX to 2.4.4 and it stopped getting an IP on WAN. so that felt like good news, so I went about to install 2.3.5 on the Qotom. After getting that done, it still didn't get an IP on WAN, so I thought maybe 2.4.4 didn't work on the BRIX because i had messed up the config before when switching re0 and ue0 for WAN and LAN. So, i created an installer stick with 2.4.4 and installed that on the BRIX which went very smoothly once I disabled CSM support in the BIOS like you suggested. Installed and configured, everything looked ok. Then I took it down to where the ISP line comes in and it didn't seem to boot ok (no screen over there) so wen up again 4 stories and tested it and the damn things boots fine. So i take it down again and same problem !#@##$%%! So I decided to stop for today. Tomorrow I will prepare a small monitor to bring so I don't have to keep climbing stairs. Will also prepare a ddwrt router and see if that works. I did install OpenWRT on the Qotom but did not get it to get an ip on WAN, but not sure if I configured it wrong maybe, I had never used OpenWRT/LuCI before.

    I captured the packets from the Draytek by configuring one of the ports on the Draytek as a mirror/monitor port, hooked up my laptop with wireshark.
    All captures for pfsense and opnsense were done with the packet capture option under diagnostics.


  • Netgate Administrator

    Hmm, frustrating!

    If you can use a actual mirror port on a switch that will show for certain that packet are being tagged correctly.

    I have no reason think they would not be however, VLANs are used extensively with igb NICs.

    Steve



  • finally had time to do some more testing.

    • i did a clean install (twice) on the BRIX (re0) with 2.3.5 and can consistently get an dhcp IP
    • clean install 2.3.5 on the Qotom (igb0) dit not work
    • reset'ed back to default and old Firebox (sk0) with 2.4.1 also got the dhcp IP
    • clean install 2.3.5 on an PC-Engines APU2C4 (igb0) did not work
    • put a smart switch (TP-Link) between router and fiber converter on port 1 and 2 and set them to PVID 128
    • both draytek and pfsense can then connect without any problems
    • comparing the packets from pfsense qotom going to ISP with and without the smart switch in between, both packets are identical except for datagram protocol checksum.

    but this doesn't help me, because i still don't know what's causing the problem. i can see the packets with vlan id 128 tagged going through the tp-link switch. but since i don't have any packets that are coming in not through the tp-link switch, i can't compare packets to see if anything is different. i am new to vlan, managed switches, so if i shoudl set it up differently for capturring, please let me know.

    how can i use the smart switch to capture raw packets coming in? i.e. the raw packets that are responding to the DHCP Dicover coming from pfsense without setting PVID 128 on port 1 and 2? without PVID i could not get an IP from the isp testing both with teh Qotom and Draytek.


  • Netgate Administrator

    Did you disable checksum off-loading in System > Advanced > Networking?

    You can probably configure a mirror port on the switch to send all the packets going to/from the ISP to a capture device.

    Steve