Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PFSense Plus 23.09 Certain Websites not working on VLAN - Working fine on LAN

    General pfSense Questions
    2
    7
    543
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      ddCorazon
      last edited by ddCorazon

      Hello everyone, (Happy new Year, best wishes)

      I have a problem with pfsense Plus in version 23.09 (I already had the problem in version 23.01)

      On my VLANs it is impossible to access some sites like https://rh.silae.fr (not all sites ). Other sites take a very long time to load. At first I thought it came from my firewall rules but I don't think so. Just to be sure I created a new VLAN with a new subnet and at the firewall level I set a single rule which authorizes all traffic without restrictions but still no access to certain sites.
      Still being on the VLANs I thought it was the DNS resolution but no more, with an nslookup I resolved the name correctly. Another weird thing is that I ping the IP associated with the DNS name.

      On my native LAN without VLAN it works without problems.

      For example, a phone cannot install applications from the appstore or install IOS updates while being on VLAN networks.

      I would like to point out that I started having the problem when I migrated from pfsense 2.4.5 to pfsense + (it may not have anything to do with it)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        So what error do you see at the client when you try to open the site?

        Check the firewall states when you try to open it. Do you see states on the VLAN and WAN interfaces? Do you see matching traffic numbers on those states?

        Ultimately a packet capture should show what's happening.

        1 Reply Last reply Reply Quote 0
        • D
          ddCorazon
          last edited by

          Thanks for the quick reply

          On the client side it just show ERR_TIME_OUT
          Here is the state on all interfaces: the VLAN interface (named INTERFACE_ADI) and WAN interface named (EURAFIBRE) filtered. The IP of the client is 192.168.200.10 and the IP of rh.silae.fr website server is 20.111.40.206 . The hidden text is my public IP
          654bed85-ba7e-4101-95e5-28383127a4c0-image.png

          This is the result of packet capture directly from pfsense interface when trying to access to website (View options set on high and pomiscuous mode enable)

          18:24:41.267556 IP (tos 0x0, ttl 128, id 56603, offset 0, flags [none], proto UDP (17), length 57)
          192.168.200.10.53323 > 192.168.200.1.53: [udp sum ok] 19021+ A? rh.silae.fr. (29)
          18:24:41.267890 IP (tos 0x0, ttl 128, id 56604, offset 0, flags [none], proto UDP (17), length 57)
          192.168.200.10.54042 > 192.168.200.1.53: [udp sum ok] 26421+ HTTPS? rh.silae.fr. (29)
          18:24:41.270901 IP (tos 0x0, ttl 128, id 56605, offset 0, flags [none], proto UDP (17), length 57)
          192.168.200.10.51296 > 192.168.200.1.53: [udp sum ok] 507+ A? rh.silae.fr. (29)
          18:24:41.276436 IP (tos 0x0, ttl 64, id 8605, offset 0, flags [none], proto UDP (17), length 143)
          192.168.200.1.53 > 192.168.200.10.54042: [bad udp cksum 0x11ea -> 0x65e8!] 26421 q: HTTPS? rh.silae.fr. 0/1/0 ns: silae.fr. SOA ns1-04.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300 (115)
          18:24:41.285296 IP (tos 0x0, ttl 64, id 9243, offset 0, flags [none], proto UDP (17), length 173)
          192.168.200.1.53 > 192.168.200.10.53323: [bad udp cksum 0x1208 -> 0x06ff!] 19021 q: A? rh.silae.fr. 1/2/2 rh.silae.fr. A 20.111.40.206 ns: silae.fr. NS ns2-04.azure-dns.net., silae.fr. NS ns1-04.azure-dns.com. ar: ns2-04.azure-dns.net. A 150.171.21.4, ns1-04.azure-dns.com. A 13.107.236.4 (145)
          18:24:41.285306 IP (tos 0x0, ttl 64, id 1573, offset 0, flags [none], proto UDP (17), length 173)
          192.168.200.1.53 > 192.168.200.10.51296: [bad udp cksum 0x1208 -> 0x573c!] 507 q: A? rh.silae.fr. 1/2/2 rh.silae.fr. A 20.111.40.206 ns: silae.fr. NS ns2-04.azure-dns.net., silae.fr. NS ns1-04.azure-dns.com. ar: ns2-04.azure-dns.net. A 150.171.21.4, ns1-04.azure-dns.com. A 13.107.236.4 (145)
          18:24:41.287572 IP (tos 0x0, ttl 128, id 49954, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [S], cksum 0xa616 (correct), seq 1275719682, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
          18:24:41.293487 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto TCP (6), length 52)
          20.111.40.206.443 > 192.168.200.10.51793: Flags [S.], cksum 0xa92f (correct), seq 3875673576, ack 1275719683, win 64240, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
          18:24:41.293746 IP (tos 0x0, ttl 128, id 49955, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [.], cksum 0xe0d9 (correct), seq 1, ack 1, win 1029, length 0
          18:24:41.294045 IP (tos 0x0, ttl 128, id 49956, offset 0, flags [DF], proto TCP (6), length 557)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [P.], cksum 0x9a63 (correct), seq 1:518, ack 1, win 1029, length 517
          18:24:41.299621 IP (tos 0x0, ttl 51, id 10799, offset 0, flags [DF], proto TCP (6), length 40)
          20.111.40.206.443 > 192.168.200.10.51793: Flags [.], cksum 0xe0e4 (correct), seq 1, ack 518, win 501, length 0
          18:24:41.300105 IP (tos 0x0, ttl 51, id 10802, offset 0, flags [DF], proto TCP (6), length 1216)
          20.111.40.206.443 > 192.168.200.10.51793: Flags [P.], cksum 0x4ce1 (correct), seq 2921:4097, ack 518, win 501, length 1176
          18:24:41.300111 IP (tos 0x0, ttl 51, id 10803, offset 0, flags [DF], proto TCP (6), length 77)
          20.111.40.206.443 > 192.168.200.10.51793: Flags [P.], cksum 0x71db (correct), seq 4097:4134, ack 518, win 501, length 37
          18:24:41.300338 IP (tos 0x0, ttl 128, id 49957, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [.], cksum 0x937d (correct), seq 518, ack 1, win 1029, options [nop,nop,sack 1 {2921:4097}], length 0
          18:24:41.300356 IP (tos 0x0, ttl 128, id 49958, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [.], cksum 0x9358 (correct), seq 518, ack 1, win 1029, options [nop,nop,sack 1 {2921:4134}], length 0
          18:25:08.322836 IP (tos 0x0, ttl 128, id 49959, offset 0, flags [DF], proto TCP (6), length 41)
          192.168.200.10.51791 > 20.111.40.206.443: Flags [.], cksum 0xd0f1 (correct), seq 1615466915:1615466916, ack 1164218768, win 1029, length 1
          18:25:08.329286 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto TCP (6), length 40)
          20.111.40.206.443 > 192.168.200.10.51791: Flags [R], cksum 0x46f1 (correct), seq 1164218768, win 0, length 0
          18:25:11.279868 IP (tos 0x0, ttl 128, id 56606, offset 0, flags [none], proto UDP (17), length 69)
          192.168.200.10.54616 > 192.168.200.1.53: [udp sum ok] 1068+ A? edge-http.microsoft.com. (41)
          18:25:11.280054 IP (tos 0x0, ttl 128, id 56607, offset 0, flags [none], proto UDP (17), length 69)
          192.168.200.10.56647 > 192.168.200.1.53: [udp sum ok] 54205+ HTTPS? edge-http.microsoft.com. (41)
          18:25:11.287286 IP (tos 0x0, ttl 64, id 4286, offset 0, flags [none], proto UDP (17), length 219)
          192.168.200.1.53 > 192.168.200.10.54616: [bad udp cksum 0x1236 -> 0xdf13!] 1068 q: A? edge-http.microsoft.com. 3/2/2 edge-http.microsoft.com. CNAME edge-microsoft-com.b-0005.b-msedge.net., edge-microsoft-com.b-0005.b-msedge.net. CNAME b-0005.b-msedge.net., b-0005.b-msedge.net. A 13.107.6.158 ns: b-msedge.net. NS ns1.b-msedge.net., b-msedge.net. NS ns2.b-msedge.net. ar: ns1.b-msedge.net. A 13.107.6.1, ns2.b-msedge.net. A 13.107.6.2 (191)
          18:25:11.287300 IP (tos 0x0, ttl 64, id 38157, offset 0, flags [none], proto UDP (17), length 168)
          192.168.200.1.53 > 192.168.200.10.56647: [bad udp cksum 0x1203 -> 0xfffe!] 54205 q: HTTPS? edge-http.microsoft.com. 1/1/0 edge-http.microsoft.com. CNAME edge-microsoft-com.b-0005.b-msedge.net. ns: b-msedge.net. SOA ns1.b-msedge.net. msnhst.microsoft.com. 2016090101 1800 900 2419200 240 (140)
          18:25:11.288306 IP (tos 0x0, ttl 128, id 45409, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51794 > 13.107.6.158.80: Flags [S], cksum 0xe229 (correct), seq 2241045763, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
          18:25:11.293979 IP (tos 0x0, ttl 128, id 49960, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [F.], cksum 0xded3 (correct), seq 518, ack 1, win 1029, length 0
          18:25:11.299978 IP (tos 0x0, ttl 113, id 50853, offset 0, flags [DF], proto TCP (6), length 52)
          13.107.6.158.80 > 192.168.200.10.51794: Flags [S.], cksum 0x8f49 (correct), seq 674440609, ack 2241045764, win 65535, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0
          18:25:11.299988 IP (tos 0x0, ttl 51, id 10812, offset 0, flags [DF], proto TCP (6), length 40)
          20.111.40.206.443 > 192.168.200.10.51793: Flags [F.], cksum 0xd0bd (correct), seq 4134, ack 519, win 501, length 0
          18:25:11.300223 IP (tos 0x0, ttl 128, id 49961, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51793 > 20.111.40.206.443: Flags [.], cksum 0x9357 (correct), seq 519, ack 1, win 1029, options [nop,nop,sack 1 {2921:4134}], length 0
          18:25:11.300933 IP (tos 0x0, ttl 128, id 45410, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51794 > 13.107.6.158.80: Flags [.], cksum 0xcc03 (correct), seq 1, ack 1, win 1029, length 0
          18:25:11.324384 IP (tos 0x0, ttl 128, id 45411, offset 0, flags [DF], proto TCP (6), length 584)
          192.168.200.10.51794 > 13.107.6.158.80: Flags [P.], cksum 0x7fc3 (correct), seq 1:545, ack 1, win 1029, length 544: HTTP, length: 544
          GET /captiveportal/generate_204 HTTP/1.1
          Host: edge-http.microsoft.com
          Connection: keep-alive
          Pragma: no-cache
          Cache-Control: no-cache
          Sec-Mesh-Client-Edge-Version: 120.0.2210.91
          Sec-Mesh-Client-Edge-Channel: stable
          Sec-Mesh-Client-OS: Windows
          Sec-Mesh-Client-OS-Version: 10.0.19045
          Sec-Mesh-Client-Arch: x86_64
          Sec-Mesh-Client-WebView: 0
          User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0
          Accept-Encoding: gzip, deflate
          Accept-Language: fr

          18:25:11.336016 IP (tos 0x0, ttl 114, id 50854, offset 0, flags [DF], proto TCP (6), length 40)
          13.107.6.158.80 > 192.168.200.10.51794: Flags [.], cksum 0x8dea (correct), seq 1, ack 545, win 16382, length 0
          18:25:11.351673 IP (tos 0x0, ttl 114, id 50855, offset 0, flags [DF], proto TCP (6), length 299)
          13.107.6.158.80 > 192.168.200.10.51794: Flags [P.], cksum 0xa140 (correct), seq 1:260, ack 545, win 16382, length 259: HTTP, length: 259
          HTTP/1.1 204 No Content
          Cache-Control: max-age=0, no-cache, no-store, must-revalidate
          X-Cache: PRIVATE_NOSTORE
          X-MSEdge-Ref: Ref A: 79617C3EBBC94B239EC1205D73CC6232 Ref B: LON212050711035 Ref C: 2024-01-04T18:25:11Z
          Date: Thu, 04 Jan 2024 18:25:11 GMT

          18:25:11.395880 IP (tos 0x0, ttl 128, id 45412, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51794 > 13.107.6.158.80: Flags [.], cksum 0xc8e1 (correct), seq 545, ack 260, win 1028, length 0
          18:25:11.985449 IP (tos 0x0, ttl 128, id 61868, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51792 > 20.223.35.26.443: Flags [R.], cksum 0x1b1d (correct), seq 4148002191, ack 3740906548, win 0, length 0
          18:25:12.328402 IP (tos 0x0, ttl 128, id 49962, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51795 > 20.111.40.206.443: Flags [S], cksum 0x63d9 (correct), seq 1959266687, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
          18:25:12.331541 IP (tos 0x0, ttl 128, id 56608, offset 0, flags [none], proto UDP (17), length 57)
          192.168.200.10.61894 > 192.168.200.1.53: [udp sum ok] 7428+ A? rh.silae.fr. (29)
          18:25:12.331662 IP (tos 0x0, ttl 64, id 8775, offset 0, flags [none], proto UDP (17), length 73)
          192.168.200.1.53 > 192.168.200.10.61894: [bad udp cksum 0x11a4 -> 0xd8ef!] 7428 q: A? rh.silae.fr. 1/0/0 rh.silae.fr. A 20.111.40.206 (45)
          18:25:12.335133 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto TCP (6), length 52)
          20.111.40.206.443 > 192.168.200.10.51795: Flags [S.], cksum 0x830c (correct), seq 2981572377, ack 1959266688, win 64240, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
          18:25:12.335422 IP (tos 0x0, ttl 128, id 49963, offset 0, flags [DF], proto TCP (6), length 40)
          192.168.200.10.51795 > 20.111.40.206.443: Flags [.], cksum 0xbab6 (correct), seq 1, ack 1, win 1029, length 0
          18:25:12.341013 IP (tos 0x0, ttl 128, id 49964, offset 0, flags [DF], proto TCP (6), length 603)
          192.168.200.10.51795 > 20.111.40.206.443: Flags [P.], cksum 0xd90c (correct), seq 1:564, ack 1, win 1029, length 563
          18:25:12.347466 IP (tos 0x0, ttl 51, id 7741, offset 0, flags [DF], proto TCP (6), length 40)
          20.111.40.206.443 > 192.168.200.10.51795: Flags [.], cksum 0xba93 (correct), seq 1, ack 564, win 501, length 0
          18:25:12.348157 IP (tos 0x0, ttl 51, id 7744, offset 0, flags [DF], proto TCP (6), length 1216)
          20.111.40.206.443 > 192.168.200.10.51795: Flags [P.], cksum 0xdff5 (correct), seq 2921:4097, ack 564, win 501, length 1176
          18:25:12.348166 IP (tos 0x0, ttl 51, id 7745, offset 0, flags [DF], proto TCP (6), length 77)
          20.111.40.206.443 > 192.168.200.10.51795: Flags [P.], cksum 0x0ed4 (correct), seq 4097:4134, ack 564, win 501, length 37
          18:25:12.348401 IP (tos 0x0, ttl 128, id 49965, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51795 > 20.111.40.206.443: Flags [.], cksum 0xa560 (correct), seq 564, ack 1, win 1029, options [nop,nop,sack 1 {2921:4097}], length 0
          18:25:12.348914 IP (tos 0x0, ttl 128, id 49966, offset 0, flags [DF], proto TCP (6), length 52)
          192.168.200.10.51795 > 20.111.40.206.443: Flags [.], cksum 0xa53b (correct), seq 564, ack 1, win 1029, options [nop,nop,sack 1 {2921:4134}], length 0

          stephenw10S 1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator @ddCorazon
            last edited by

            Ok the first thing I'm seeing here is that the state packet counters for the VLAN and WAN states do not match. There are more packets shown on the WAN. So it looks like for some reason packets arriving on the WAN are not being forwarded to the VLAN.
            If you do the same test on LAN those counters should match.

            I'd have to guess it's an MTU issue somewhere. Try a pcap on the WAN and look at the packet sizes. If the largest packets are not passed it's probably that.

            Steve

            D 1 Reply Last reply Reply Quote 1
            • D
              ddCorazon @stephenw10
              last edited by ddCorazon

              @stephenw10
              This is the state when trying to access to the same site but from a client on the LAN interface. the packet count seems to look alike on WAn and LAN
              aa65fe2c-810c-48c8-9a3d-56f0a36bc3aa-image.png

              And bellow the packet lenght stats of WAN from wireshark
              1bba9220-885d-4791-809b-f631378e124d-image.png

              Here from LAN interface
              fe467e0e-8562-46ab-aa0e-75bed62f030b-image.png

              And here from VLAn interface
              47f4bedb-f82a-4668-ada0-e471dc963e69-image.png

              prrety weird most packet are between 40-79 Look like there is a mess somewhere ??

              reviewing on interface status? LAN is 1500, WAN is also 1500 and VLANs interfaces are 1492

              dabb19e1-51cb-40f7-8b2b-02d0255dd5d5-image.png

              841bc133-1979-4ccd-af7a-f81d5c4a2291-image.png

              1051da2e-053a-44ec-ac8a-dbd499955cc3-image.png

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Right the VLAN tags use 8 bytes so the MTU is 1492 there. Except most NICs can handle 1508 byte packets so the MTU can usually be 1500. Those are VMX NICs so I'm unsure. Possibly you'd need to configure them to allow that.

                However you also see 1514B packets on WAN which is larger than I expect to see there. I don't see anything larger than 1506B here.

                So it looks like the Path MTU detection is broken there somewhere because it should simply reduce the packet size to fit the 1492 MTU.

                1 Reply Last reply Reply Quote 0
                • D
                  ddCorazon
                  last edited by

                  Hey, Sorry for the late.

                  After the last tests I tried to map one vlan on the LAN NIC and It's OK. So the problem was either the nic used for the VLANs or the switch port used.

                  Thanks

                  1 Reply Last reply Reply Quote 1
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.