tunneling VLAN trunk help needed

jpyeron

I am trying to avoid a double NAT situation shown below which is working, except for the issues of double NAT. My tunnel solution is not working.

The desired architecture should be something similar to:

On pfSense tunnel the vlan trunk comes in tagged on vtnet2, and goes out gif0 (on vtnet1) via bridge0. On pfSense NAT the vlan trunk comes in via gif0 and is bridged to bridge0. The vlan interfaces are created as bridge0.1, bridge0.2, ...., bridge0.nnnn. The bridge was needed because GIF interfaces do not support VLAN.

At this point ARP seems to be working, but ping or TCP do not.

I have search the forum (shown below), with no conclusive results.

https://forum.netgate.com/topic/113534/bridge-and-gif-tunnel
https://forum.netgate.com/topic/133446/which-tunnel-to-use
https://forum.netgate.com/topic/138272/gif-monitoring-traffic-goes-via-wrong-parent-interface-on-pfsense-2-4-4
https://forum.netgate.com/topic/119752/bridging-problem

JKnott

@jpyeron

You can't tunnel VLANs, unless you use TAP mode. VLANs are a layer 2 function and tunnels layer 3. What you do is just route the various subnets through the VLAN.

jpyeron

@JKnott said in tunneling VLAN trunk help needed:

@jpyeron

You can't tunnel VLANs, unless you use TAP mode. VLANs are a layer 2 function and tunnels layer 3. What you do is just route the various subnets through the VLAN.

Can you please define "can't"? I am not trying to be pedantic, but clearly on the receiving side I get the VLAN tagged frames, and then when broken out I get only the untagged frames for the corresponding VLAN ID, which is desired and contrary to your statement.

Here are 4 frames of ARP, request/response on bridge0 (Tagged) and bridge0.2314 (Untagged VLAN 2314)

bridge0

ARP request

No.     Time           Source                Destination           Protocol Length Info
   5224 11431.773861   Tp-LinkT_b4:50:c6     Broadcast             ARP      96     Who has 10.23.14.254? Tell 10.23.14.10

Frame 5224: 96 bytes on wire (768 bits), 96 bytes captured (768 bits) on interface -, id 0
    Interface id: 0 (-)
        Interface name: -
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 18, 2020 20:30:13.591997000 Eastern Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1582075813.591997000 seconds
    [Time delta from previous captured frame: 148.389468000 seconds]
    [Time delta from previous displayed frame: 148.389468000 seconds]
    [Time since reference or first frame: 11431.773861000 seconds]
    Frame Number: 5224
    Frame Length: 96 bytes (768 bits)
    Capture Length: 96 bytes (768 bits)
    [Frame is marked: True]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:etherip:eth:ethertype:vlan:ethertype:arp]
    [Coloring Rule Name: ARP]
    [Coloring Rule String: arp]
Ethernet II, Src: REDACTED:83:b6), Dst: REDACTED:bd:ef)
    Destination: REDACTED:bd:ef)
        Address: REDACTED:bd:ef)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: REDACTED:83:b6)
        Address: REDACTED:83:b6)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: REDACTED_lower_IP, Dst: REDACTED_upper_IP
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 82
    Identification: 0x380f (14351)
    Flags: 0x0000
        0... .... .... .... = Reserved bit: Not set
        .0.. .... .... .... = Don't fragment: Not set
        ..0. .... .... .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 30
    Protocol: Ether in IP (97)
    Header checksum: 0xd15c [validation disabled]
    [Header checksum status: Unverified]
    Source: REDACTED_lower_IP
    Destination: REDACTED_upper_IP
EtherIP, Version 3
    0011 .... .... .... = Version: 3
    .... 0000 0000 0000 = Reserved: 0x000
Ethernet II, Src: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        Address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 2314
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 1001 0000 1010 = ID: 2314
    Type: ARP (0x0806)
    Trailer: 0000000000000000000000000000
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
   Protocol size: 4
    Opcode: request (1)
    Sender MAC address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Sender IP address: 10.23.14.10
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 10.23.14.254

REDACTEDDATABLOCK

ARP response

No.     Time           Source                Destination           Protocol Length Info
   5225 11431.774056   REDACTED:be:00     Tp-LinkT_b4:50:c6     ARP      82     10.23.14.254 is at REDACTED:be:00

Frame 5225: 82 bytes on wire (656 bits), 82 bytes captured (656 bits) on interface -, id 0
    Interface id: 0 (-)
        Interface name: -
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 18, 2020 20:30:13.592192000 Eastern Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1582075813.592192000 seconds
    [Time delta from previous captured frame: 0.000195000 seconds]
    [Time delta from previous displayed frame: 0.000195000 seconds]
    [Time since reference or first frame: 11431.774056000 seconds]
    Frame Number: 5225
    Frame Length: 82 bytes (656 bits)
    Capture Length: 82 bytes (656 bits)
    [Frame is marked: True]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:etherip:eth:ethertype:vlan:ethertype:arp]
    [Coloring Rule Name: ARP]
    [Coloring Rule String: arp]
Ethernet II, Src: REDACTED:bd:ef), Dst: REDACTED:83:b6)
    Destination: REDACTED:83:b6)
        Address: REDACTED:83:b6)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: REDACTED:bd:ef)
        Address: REDACTED:bd:ef)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: REDACTED_upper_IP, Dst: REDACTED_lower_IP
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 68
    Identification: 0x39b5 (14773)
    Flags: 0x0000
        0... .... .... .... = Reserved bit: Not set
        .0.. .... .... .... = Don't fragment: Not set
        ..0. .... .... .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 30
    Protocol: Ether in IP (97)
    Header checksum: 0xcfc4 [validation disabled]
    [Header checksum status: Unverified]
    Source: REDACTED_upper_IP
    Destination: REDACTED_lower_IP
EtherIP, Version 3
    0011 .... .... .... = Version: 3
    .... 0000 0000 0000 = Reserved: 0x000
Ethernet II, Src: REDACTED:be:00 (REDACTED:be:00), Dst: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Destination: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        Address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: REDACTED:be:00 (REDACTED:be:00)
        Address: REDACTED:be:00 (REDACTED:be:00)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 2314
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 1001 0000 1010 = ID: 2314
    Type: ARP (0x0806)
Address Resolution Protocol (reply)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (2)
    Sender MAC address: REDACTED:be:00 (REDACTED:be:00)
    Sender IP address: 10.23.14.254
    Target MAC address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Target IP address: 10.23.14.10

REDACTED_DATA_BLOCK                                            ..

bridge0.2314

ARP request

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       Tp-LinkT_b4:50:c6     Broadcast             ARP      56     Who has 10.23.14.254? Tell 10.23.14.10

Frame 1: 56 bytes on wire (448 bits), 56 bytes captured (448 bits) on interface -, id 0
    Interface id: 0 (-)
        Interface name: -
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 18, 2020 20:30:13.592133000 Eastern Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1582075813.592133000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 56 bytes (448 bits)
    Capture Length: 56 bytes (448 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:arp]
    [Coloring Rule Name: ARP]
    [Coloring Rule String: arp]
Ethernet II, Src: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        Address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
    Trailer: 0000000000000000000000000000
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Sender IP address: 10.23.14.10
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 10.23.14.254

REDACTED_DATA_BLOCK

ARP response

No.     Time           Source                Destination           Protocol Length Info
      2 0.000040       REDACTED:be:00     Tp-LinkT_b4:50:c6     ARP      42     10.23.14.254 is at REDACTED:be:00

Frame 2: 42 bytes on wire (336 bits), 42 bytes captured (336 bits) on interface -, id 0
    Interface id: 0 (-)
        Interface name: -
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 18, 2020 20:30:13.592173000 Eastern Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1582075813.592173000 seconds
    [Time delta from previous captured frame: 0.000040000 seconds]
    [Time delta from previous displayed frame: 0.000040000 seconds]
    [Time since reference or first frame: 0.000040000 seconds]
    Frame Number: 2
    Frame Length: 42 bytes (336 bits)
    Capture Length: 42 bytes (336 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:arp]
    [Coloring Rule Name: ARP]
    [Coloring Rule String: arp]
Ethernet II, Src: REDACTED:be:00 (REDACTED:be:00), Dst: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Destination: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        Address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: REDACTED:be:00 (REDACTED:be:00)
        Address: REDACTED:be:00 (REDACTED:be:00)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
Address Resolution Protocol (reply)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (2)
    Sender MAC address: REDACTED:be:00 (REDACTED:be:00)
    Sender IP address: 10.23.14.254
    Target MAC address: Tp-LinkT_b4:50:c6 (98:da:c4:b4:50:c6)
    Target IP address: 10.23.14.10

REDACTED_DATA_BLOCK

jpyeron

The GIF tunnel is supposed to support Layer 2. I see no reason it should not support 802.1q. I do know that when tunneling Layer 2, I cannot at the same time "ping" on Layer 3[1] of the foreign GIF IP.

1: https://docs.netgate.com/pfsense/en/latest/interfaces/gif-interfaces.html#limitations

jpyeron

Here is an updated systems view, emphasizing the security perimeter. As a goal I would like to reduce the resources required for the lower (tunnel) pfSense firewall, hence maximizing the throughput.

JKnott

@jpyeron said in tunneling VLAN trunk help needed:

The GIF tunnel is supposed to support Layer 2. I see no reason it should not support 802.1q. I do know that when tunneling Layer 2, I cannot at the same time "ping" on Layer 3[1] of the foreign GIF IP.

1: https://docs.netgate.com/pfsense/en/latest/interfaces/gif-interfaces.html#limitations

Please go back and read what I said. I said that it would require TAP mode, rather than the usual TUN mode. However, there is nothing in your sketch to indicate which you are using. The usual practice for VPNs is to use tunnel mode, with different subnets at each end. With TAP mode, you must have everything in the same subnet, which means you cannot use routing to isolate networks. This also means that things like broadcasts, spanning tree and others are consuming bandwidth of what's usually a narrow bandwidth connection.

jpyeron

I think I see the disconnect. Thank you for your patience on this.

@JKnott said in tunneling VLAN trunk help needed:

Please go back and read what I said. I said that it would require TAP mode, rather than the usual TUN mode.

I am using an unencrypted GIF. Nowhere in the docs, configuration screen of the GIF Configuration screen, or ifconfig is there a mention of TAP or TUN. It is a GIF, like GRE, but can carry Layer 2 encapsulated in Protocol: Ether in IP (97).

Am I misunderstanding your use of TAP/TUN terminology?

Or am I misunderstanding the application of TAP/TUN to a GIF interface?

However, there is nothing in your sketch to indicate which you are using.

Correct, the sketch is an architectural desire. I am "trying avoid a double NAT situation" in the first diagram. I am open to any solution that accomplishes the second diagram, where the VLAN Tagged Trunk is bridged between the upper and lower pfSense boxes.

The text

On pfSense tunnel the vlan trunk comes in tagged on vtnet2, and goes out gif0 (on vtnet1) via bridge0. On pfSense NAT the vlan trunk comes in via gif0 and is bridged to bridge0. The vlan interfaces are created as bridge0.1, bridge0.2, ...., bridge0.nnnn. The bridge was needed because GIF interfaces do not support VLAN.

was used to describe the configuration at present. I first tried with GRE (layer 3 only), then OpenVPN (could not get VLANs to bridge), and then GIF (ARP/Layer 2 good, TCP/ICMP nothing).

The usual practice for VPNs is to use tunnel mode, with different subnets at each end.

NOTE: I am not using OpenVPN, hence the TAP/TUN question seems to be out of place.

With TAP mode, you must have everything in the same subnet, which means you cannot use routing to isolate networks.

Same subnet, that is the intention as the gateway (NAT) would be on the upper pfSense and the hosts are in the untrusted VLANs below.

This also means that things like broadcasts, spanning tree and others are consuming bandwidth of what's usually a narrow bandwidth connection.

This is not a concern, the LAN fabric is 10G, the clients are connected at 1G / wifi, and the Internet NAT is rate limited to 100MB.

jpyeron

@jpyeron said in tunneling VLAN trunk help needed:

Correct, the sketch is an architectural desire. I am "trying avoid a double NAT situation" in the first diagram. I am open to any solution that accomplishes the second diagram, where the VLAN Tagged Trunk is bridged between the upper and lower pfSense boxes.

The text <snip/> was used to describe the configuration at present. I first tried with GRE (layer 3 only), then OpenVPN (could not get VLANs to bridge), and then GIF (ARP/Layer 2 good, TCP/ICMP nothing).

On a side note, I previously tried to get OpenVPN working using the details I found at:
https://forum.archive.openwrt.org/viewtopic.php?id=33678 and https://openvpn.net/community-resources/ethernet-bridging/ and https://forum.netgate.com/topic/107802/bridging-vlans-and-physical-interfaces

JKnott

@jpyeron said in tunneling VLAN trunk help needed:

I am using an unencrypted GIF. Nowhere in the docs, configuration screen of the GIF Configuration screen, or ifconfig is there a mention of TAP or TUN. It is a GIF, like GRE, but can carry Layer 2 encapsulated in Protocol: Ether in IP (97).

GRE is a method for carrying a variety of frame types, just like PPP or PPPoE are used. GRE compares with TAP mode in OpenVPN. VPNs usually include encryption for privacy, as those links often carry internal traffic from the LAN. Not using encryption also increases your security risk, as you're essentially leaving your network wide open.

BTW, I first got involved with packet networks back in the X.25¹ days, through my work. Back then, there was little concern for security, as these were closed systems, compared to the wide open Internet we know today. With X.25 and later with Frame Relay, you had virtual circuits, which were only accessible to those within your organization. These days, we use proper VPNs, with encryption to obtain the same results.

The company I worked for provided the "Telenet" service in Canada. Another popular X.25 service, back in those days, was CompuServe. I had an account with them at home, but Telenet at work. There was no way to connect between them, but someone at CompuServe support mentioned this new thing called "The Internet", which might connect them. It eventually did. When that happened, you'd have 2 "@" in an email address, if you had to use a gateway to connect different services.

jpyeron

@JKnott said in tunneling VLAN trunk help needed:

@jpyeron said in tunneling VLAN trunk help needed:

I am using an unencrypted GIF. Nowhere in the docs, configuration screen of the GIF Configuration screen, or ifconfig is there a mention of TAP or TUN. It is a GIF, like GRE, but can carry Layer 2 encapsulated in Protocol: Ether in IP (97).

GRE is a method for carrying a variety of frame types, just like PPP or PPPoE are used. GRE compares with TAP mode in OpenVPN. VPNs usually include encryption for privacy, as those links often carry internal traffic from the LAN.

I am not using GRE, I am using generic tunnel interface (GIF). It is designed to carry L2 over L3.

Not using encryption also increases your security risk, as you're essentially leaving your network wide open.

I appreciate your concerns, but they have no place in this discussion. If we use OpenVPN, it will be used with the null cipher. The reasons for this are very much out of scope of the issue at hand. Please note the unencrypted traffic is traveling inside a secure (logically and physically) perimeter.

BTW, I first got involved with packet networks back in the X.25¹ days, through my work. Back then, there was little concern for security, as these were closed systems, compared to the wide open Internet we know today. With X.25 and later with Frame Relay, you had virtual circuits, which were only accessible to those within your organization. These days, we use proper VPNs, with encryption to obtain the same results.

The company I worked for provided the "Telenet" service in Canada. Another popular X.25 service, back in those days, was CompuServe. I had an account with them at home, but Telenet at work. There was no way to connect between them, but someone at CompuServe support mentioned this new thing called "The Internet", which might connect them. It eventually did. When that happened, you'd have 2 "@" in an email address, if you had to use a gateway to connect different services.

I remember the all the different gateways. Thats cool, sounds like you got in to things a bit before me.

JKnott

@jpyeron said in tunneling VLAN trunk help needed:

I remember the all the different gateways. Thats cool, sounds like you got in to things a bit before me.

Yep, I was working with LANs before there was such a thing as Ethernet. I worked on a Time Division Multiplexing network in a Rockwell Collins 8500C computer system, that was part of the Air Canada reservation system. This was late '70s - mid '80s. The various devices, such as tape drives and disk drives connected to the CPU over a tri-axial cable at 8 Mb/s. This system was the communications front end for the Univac computers at the heart of the system.