Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Web sites not loading when accessing pfSense through VLAN trunk.

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 3 Posters 9.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      This could be an important result in narrowing down the watchdog timeout errors that have plagued that box. Until now about the best advice we have is to use a managed connected to the box based anecdotal evidence that fragmented packets can cause a problem.

      Anyway I think you will have to bring the interface down before changing the MTU or possibly remove it from the LAGG? Though I have no problem doing that on my box.  :-\ You want to be doing this on the VLAN parent interface but this is the LAGG interface for you. Hmm. Trial and error I think!

      Also I see that the re driver and the chip supports hardware VLAN tagging. It might be interesting to try disabling it, especially vlanmtu:
      @ifconfig:

      vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso
          If the driver offers user-configurable VLAN support, enable
          reception of extended frames, tag processing in hardware, frame
          filtering in hardware, or TSO on VLAN, respectively.  Note that
          this must be issued on a physical interface associated with
          vlan(4), not on a vlan(4) interface itself.

      -vlanmtu, -vlanhwtag, -vlanhwfilter, -vlanhwtso
          If the driver offers user-configurable VLAN support, disable
          reception of extended frames, tag processing in hardware, frame
          filtering in hardware, or TSO on VLAN, respectively.

      Steve

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        Some Realtek NICs have broken long frame support, so when you're trying to pass packets that have the full 1500 MTU and then add the VLAN tag, they refuse to send or receive them. Your symptoms match that scenario 100%. Not sure anyone has done VLANs on those boxes, but it'd be far from the first NIC issues people have seen on them.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          I am having difficulty understanding the various flags reported by ifconfig but it seems to me that the vlanmtu flag is used by the VLAN driver to determine whether or not the interface supports VLAN frames larger than 1500. If it is set and the interface in fact does not support this then it could be the cause of many problems. There is a suggested solution to this:

          The vlan driver automatically recognizes devices that natively support
              long frames for vlan use and calculates the appropriate frame MTU based
              on the capabilities of the parent interface.  Some other interfaces not
              listed above may handle long frames, but they do not advertise this abil-
              ity of theirs.  The MTU setting on vlan can be corrected manually if used
              in conjunction with such a parent interface.

          My own X700 box has died completely so I can't check.  :(
          What MTU size has the VLAN driver determined is correct on your box? What flags are reported by ifconfig on the re interfaces?

          Steve

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Here's a quote from YongHyeon PYUN, the re(4) maintainer/author:
            @http://freebsd.1045724.n5.nabble.com/Abysmal-re-4-performance-under-8-1-STABLE-mid-August-td3946608.html:

            I'm sure this has nothing to do that this issue.
            If you want to disable checksum offloading of VLAN
            interface, use vlan interface instead of parent interface
            of the VLAN interface(i.e. ifconfig vlan0 -txcsum -rxcsum).
            And you can't disable VLAN_MTU on re(4). There is no
            reason to disable supporting VLAN oversized frames.

            So perhaps a manual MTU reduction is necessary.

            Steve

            1 Reply Last reply Reply Quote 0
            • S
              superbob
              last edited by

              Thanks for all the insight!

              So far I tried disabling all the hardware features on the LAGG interfaces and reducing the MTU. I'll try doing one thing at a time this weekend, just wanted to see the result of the extreme set of changes. I had to delete the LAGG and all the VLANs to be able to change the underlying interfaces.

              There are two differences I noticed so far:
              1 - now I can ping VLAN PCs with all the packet sizes with no packet loss. Ie. doing ping -v -c 1 -g 1470 -G 1492 -S 192.168.10.1 192.168.10.64 on pfSense box doesn't exibit packet loss anymore.
              2 - while pinging the pfSense box via the VLAN interfaces (with hw. features disabled), when using packet sizes that didn't work before (~1474, basically MTU - 28), there are no echo replies detected in tcpdump. Previously, echo replies were logged but nothing got out

              Also, no watchdog timeouts logged yet, but I've done no stress testing yet either.

              Certain web sites are still inaccessible when connecting via VLAN interfaces. Actually, I think it even got worse since I can't even load imgur now, while previously it was just a matter of refreshing the page until the main pic loaded.

              Next up I'll try disabling the LAGG. Even though I don't suspect it of inducing errors, it doesn't make changing interface settings any easier. It did inherit all the relevant changes I made to the interfaces, like MTU and hw. features.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                @superbob:

                2 - while pinging the pfSense box via the VLAN interfaces (with hw. features disabled), when using packet sizes that didn't work before (~1474, basically MTU - 28), there are no echo replies detected in tcpdump. Previously, echo replies were logged but nothing got out.

                Hmm, anything in the firewall log? Did you reinstate the firewall rules? Easily overlooked.  ;)
                Looks like you're making some progress.

                Steve

                1 Reply Last reply Reply Quote 0
                • S
                  superbob
                  last edited by

                  What I meant in point 2:

                  PE1900, pinging normal interface with HW acceleration enabled:

                  
                  root@bobeus:~# ping -c3 -s 1474 -I 192.168.2.16 192.168.2.1
                  PING 192.168.2.1 (192.168.2.1) from 192.168.2.16 : 1474(1502) bytes of data.
                  ^C
                  --- 192.168.2.1 ping statistics ---
                  3 packets transmitted, 0 received, 100% packet loss, time 2015ms
                  
                  

                  tcpdump on pfSense box:

                  
                  23:58:29.303739 IP 192.168.2.16 > 192.168.2.1: ICMP echo request, id 22611, seq 1, length 1480
                  23:58:29.303762 IP 192.168.2.16 > 192.168.2.1: icmp
                  23:58:29.303871 IP 192.168.2.1 > 192.168.2.16: ICMP echo reply, id 22611, seq 1, length 1480
                  23:58:29.303875 IP 192.168.2.1 > 192.168.2.16: icmp
                  23:58:30.316443 IP 192.168.2.16 > 192.168.2.1: ICMP echo request, id 22611, seq 2, length 1480
                  23:58:30.316464 IP 192.168.2.16 > 192.168.2.1: icmp
                  23:58:30.316517 IP 192.168.2.1 > 192.168.2.16: ICMP echo reply, id 22611, seq 2, length 1480
                  23:58:30.316521 IP 192.168.2.1 > 192.168.2.16: icmp
                  23:58:31.329564 IP 192.168.2.16 > 192.168.2.1: ICMP echo request, id 22611, seq 3, length 1480
                  23:58:31.329586 IP 192.168.2.16 > 192.168.2.1: icmp
                  23:58:31.329646 IP 192.168.2.1 > 192.168.2.16: ICMP echo reply, id 22611, seq 3, length 1480
                  23:58:31.329650 IP 192.168.2.1 > 192.168.2.16: icmp
                  
                  

                  PE1900, pinging the a VLAN interface with hw acceleration disabled.

                  
                  root@bobeus:~# ping -c3 -s 1468 -I 192.168.10.64 192.168.10.1
                  PING 192.168.10.1 (192.168.10.1) from 192.168.10.64 : 1468(1496) bytes of data.
                  ^C
                  --- 192.168.10.1 ping statistics ---
                  3 packets transmitted, 0 received, 100% packet loss, time 2015ms
                  
                  

                  tcpdump on pfSense box:

                  
                  [2.0.1-RELEASE][root@pfsense.bobnet]/root(19): tcpdump -i re3_vlan128 host 192.168.10.64
                  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
                  listening on re3_vlan128, link-type EN10MB (Ethernet), capture size 96 bytes
                  00:03:14.496355 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22616, seq 1, length 1476
                  00:03:15.500857 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22616, seq 2, length 1476
                  00:03:16.505921 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22616, seq 3, length 1476
                  00:03:19.529010 ARP, Request who-has 192.168.10.1 tell 192.168.10.64, length 42
                  00:03:19.529036 ARP, Reply 192.168.10.1 is-at 00:90:7f:2e:84:db (oui Unknown), length 28
                  ^C
                  5 packets captured
                  5 packets received by filter
                  0 packets dropped by kernel
                  
                  

                  PE1900, pinging the a VLAN interface with a smaller payload.

                  
                  root@bobeus:~# ping -c3 -s 1452 -I 192.168.10.64 192.168.10.1
                  PING 192.168.10.1 (192.168.10.1) from 192.168.10.64 : 1452(1480) bytes of data.
                  1460 bytes from 192.168.10.1: icmp_req=1 ttl=64 time=0.503 ms
                  1460 bytes from 192.168.10.1: icmp_req=2 ttl=64 time=0.443 ms
                  1460 bytes from 192.168.10.1: icmp_req=3 ttl=64 time=0.434 ms
                  
                  --- 192.168.10.1 ping statistics ---
                  3 packets transmitted, 3 received, 0% packet loss, time 1998ms
                  rtt min/avg/max/mdev = 0.434/0.460/0.503/0.030 ms
                  
                  

                  tcpdump on pfSense box:

                  
                  [2.0.1-RELEASE][root@pfsense.bobnet]/root(25): tcpdump -i re3_vlan128 host 192.168.10.64
                  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
                  listening on re3_vlan128, link-type EN10MB (Ethernet), capture size 96 bytes
                  00:08:47.328097 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22659, seq 1, length 1460
                  00:08:47.328201 IP 192.168.10.1 > 192.168.10.64: ICMP echo reply, id 22659, seq 1, length 1460
                  00:08:48.332135 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22659, seq 2, length 1460
                  00:08:48.332183 IP 192.168.10.1 > 192.168.10.64: ICMP echo reply, id 22659, seq 2, length 1460
                  00:08:49.336806 IP 192.168.10.64 > 192.168.10.1: ICMP echo request, id 22659, seq 3, length 1460
                  00:08:49.336850 IP 192.168.10.1 > 192.168.10.64: ICMP echo reply, id 22659, seq 3, length 1460
                  ^C
                  6 packets captured
                  6 packets received by filter
                  0 packets dropped by kernel
                  
                  

                  I've also looked at the interface itself, no replies visible either.

                  Pinging with large payloads (2000+) works well, whenever the packets are fragmented.

                  Question - if I set the interface/VLAN interface MTU very low, say 300, should a ping of 1200 bytes directed to it be automatically split up into smaller chunks? Or should it always fail (like it does here)? I think I need to read up on the basics…

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    If you don't have 'do not fragment' set then it should simply fragment the packets into suitably sized frames. The problem is how it decides whether it needs to do that and how it decides what a suitable size is.
                    To be honest this is now well outside my own experience!  ;)

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • C
                      cmb
                      last edited by

                      @superbob:

                      Question - if I set the interface/VLAN interface MTU very low, say 300, should a ping of 1200 bytes directed to it be automatically split up into smaller chunks? Or should it always fail (like it does here)? I think I need to read up on the basics…

                      It'll get dropped, can't accept frames larger than your MTU. Nothing in the path to fragment it.

                      1 Reply Last reply Reply Quote 0
                      • S
                        superbob
                        last edited by

                        I tried, tested, tuned and couldn't get it to work in a reasonable amount of time. So I dropped the idea of using VLANs on the pfSense/firebox combo.

                        Instead of aggregating four ports into a LAGG and passing VLANs through that, I just mapped four ports on the switch to different VLANs and setup interfaces normally. It works fine this way, but I really liked the idea of having a theoretical throughput of 400Mb/s to play with, along with a flexible amount of VLAN interfaces to control.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          A dissapointing result but hopefully save someone else some time.  ::)
                          I'm sure it could be made to work but whether it would be worth the effort or not is debatable. It would probably be easier to just put an Intel gigabit card in the PCI slot with the case mods that requires.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.