Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Weird issue with OPT1 internet access

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    22 Posts 5 Posters 3.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      Rewt0r
      last edited by

      Hi all,

      Recently setup a new pf installation that has 3 interfaces: WAN, LAN and OPT1 (doubles as a 2nd LAN for virtual machines).

      I've added a rule to allow any traffic from the OPT1 interface and also created a 1:1 NAT mapping for an external IP on WAN, I am able to ping, trace etc from the virtual machine (Win Server2012 R2) but opening web apges does not work.

      There are no entries in the firewall log, I have tried turning the Windows Firewall off and it still isn't working. The automatic outbound NAT rules look as though they should work and I have also tried manual outbound rules but still unable to open web pages, get updates etc.

      Any ideas what I'm missing here?

      1 Reply Last reply Reply Quote 0
      • KOMK Offline
        KOM
        last edited by

        Any ideas what I'm missing here?

        Some screencaps to show us your firewall & NAT rules?  We have no idea what you've actually done.  Outbound NAT is not your problem here so you should nuke anything you created in that area.

        1 Reply Last reply Reply Quote 0
        • R Offline
          Rewt0r
          last edited by

          OPT1 (VMLAN) rules:

          Outbound NAT:

          1:1 NAT (top one works fine which is for a device on the LAN interface):

          1 Reply Last reply Reply Quote 0
          • KOMK Offline
            KOM
            last edited by

            First, for a web server you shouldn't do a 1:1 NAT.  Just create port-forward for 80 and 443.

            Next, for a port-forward to work, you also need a complimentary rule on WAN to allow the access in.  Please post your WAN rules.

            1 Reply Last reply Reply Quote 0
            • johnpozJ Online
              johnpoz LAYER 8 Global Moderator
              last edited by

              on a side note the use of 10.0.0.0/8 really??  You have that many networks/nodes that using all of /8 makes sense?

              An intelligent man is sometimes forced to be drunk to spend time with his fools
              If you get confused: Listen to the Music Play
              Please don't Chat/PM me for help, unless mod related
              SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1

              1 Reply Last reply Reply Quote 0
              • GertjanG Offline
                Gertjan
                last edited by

                @johnpoz:

                on a side note the use of 10.0.0.0/8 really??  You have that many networks/nodes that using all of /8 makes sense?

                Wait to see what happens when he gets a IPv6/48  8)

                No "help me" PM's please. Use the forum, the community will thank you.
                Edit : and where are the logs ??

                1 Reply Last reply Reply Quote 0
                • KOMK Offline
                  KOM
                  last edited by

                  You have that many networks/nodes that using all of /8 makes sense?

                  I use a /16 for organizational purposes.  10.10.0 for servers, 10.10.10 for clients, 10.10.2 for DHCP, 10.10.4 for phones, etc but yes a /8 seems excessive.

                  1 Reply Last reply Reply Quote 0
                  • R Offline
                    Rewt0r
                    last edited by

                    @KOM:

                    You have that many networks/nodes that using all of /8 makes sense?

                    I use a /16 for organizational purposes.  10.10.0 for servers, 10.10.10 for clients, 10.10.2 for DHCP, 10.10.4 for phones, etc but yes a /8 seems excessive.

                    It is excessive and will be adjusted prior to going into production, 10.10 for servers, 10.20 for workstations 10.30 for DHCP. I wanted to sort connectivity first of all, I wasn't expecting to hit this issue as I've already got it working fine on a co-lo server running 2.2.2…

                    1 Reply Last reply Reply Quote 0
                    • R Offline
                      Rewt0r
                      last edited by

                      @KOM:

                      First, for a web server you shouldn't do a 1:1 NAT.  Just create port-forward for 80 and 443.

                      Next, for a port-forward to work, you also need a complimentary rule on WAN to allow the access in.  Please post your WAN rules.

                      This isn't a web server or I'd have forwarded those ports, the issue is getting HTTP access going from this server to the outside world. I can ICMP from the server fine but making any HTTP requests with a web browser or trying to update using Windows Update fails with no error in the firewall log. Workstations are working fine but they're on LAN and not OPT1…

                      1 Reply Last reply Reply Quote 0
                      • R Offline
                        Rewt0r
                        last edited by

                        Right, I've now finished creating any necessary VLANs and restricted the addressing to what is required. I've added duplicate rules for VLAN10 and VLAN20 as has been added in VMLAN (whilst replacing the source to their respective subnet).

                        VLAN10 and VLAN20 can access HTTP but VMLAN still cannot, updated screenshots above.

                        This isn't a DNS issue as performing "nslookup google.co.uk" returns:

                        C:\Windows\system32>nslookup google.co.uk
Server:  UnKnown
Address:  192.168.1.1

Non-authoritative answer:
Name:    google.co.uk
Addresses:  2a00:1450:4009:80d::2003
          31.55.166.213
          31.55.166.217
          31.55.166.218
          31.55.166.215
          31.55.166.219
          31.55.166.212
          31.55.166.216
          31.55.166.214

                        UPDATE: I've tried removing the allow all rules from the OPT1 (VMLAN) interface and the firewall correctly shows it blocking the DNS request, allowing this through a rule then correctly shows a block accessing the websites IP address on port 80, allowing this as a rule still doesn't allow HTTP requests :/

                        1 Reply Last reply Reply Quote 0
                        • X Offline
                          xtofh
                          last edited by

                          And you're sure no squid/transp proxy is installed?

                          If it was me I'd:

                          • create a rule to pass (but log!) all http traffic

                          • get rid of any non-default nat/port forwarding rules

                          • look with tcpdump on the external (WAN) interface. (tcpdump -n -i wanif port 80)

                          1 Reply Last reply Reply Quote 0
                          • johnpozJ Online
                            johnpoz LAYER 8 Global Moderator
                            last edited by

                            "the issue is getting HTTP access going from this server to the outside world"

                            Then why do you have a 1:1 nat setup??  You do understand this is point and click working…  So something stupid your over looking..

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1

                            1 Reply Last reply Reply Quote 0
                            • R Offline
                              Rewt0r
                              last edited by

                              @johnpoz:

                              "the issue is getting HTTP access going from this server to the outside world"

                              Then why do you have a 1:1 nat setup??  You do understand this is point and click working…  So something stupid your over looking..

                              The 1:1 was added to assign this server an inbound/outbound IP for usage later. Yes I understand that once there is an allow any rule it should be all working as expected but it isn't and I know there must be something stupid I am overlooking but I cannot seem to figure out what. Hence my post here.

                              I have started again by demoting the Windows server and destroying the domain.

                              pfsense has 5 interfaces:

                              WAN
                              LAN - 10.0.0.0/24
                              OPT1 (VMLAN) - 10.0.5.0/24
                              OPT2 (VLAN10) -  10.0.10.0/24
                              OPT3 (VLAN20) - 10.0.20.0/24

                              Windows Server is on VMLAN:
                              10.0.5.2
                              GW: 10.0.5.1
                              DNS: 10.0.5.1

                              Outbound NAT is set to automatic:

                              ICMP requests from the Windows server are working fine as are DNS requests but HTTP is not even though the Network and Sharing Center states that there is Internet access.
                              Windows Firewall is turned off for all connection types, IE enhanced security configuration is also disabled for administrators (which is the account in use).

                              If I create a firewall rule on VMLAN to log requests on port 80 I can see that this is passed on the firewall log although the server gets "Page can't be displayed"… I'm stumped because everything seems to be correct.


                              1 Reply Last reply Reply Quote 0
                              • X Offline
                                xtofh
                                last edited by

                                Try tcpdump on your external interface to make sure it requests the page correctly?

                                1 Reply Last reply Reply Quote 0
                                • R Offline
                                  Rewt0r
                                  last edited by

                                  @xtofh:

                                  Try tcpdump on your external interface to make sure it requests the page correctly?

                                  172.16.0.78 being the pfsense LAN address assigned from the modem (WAN)

                                  16:28:48.909893 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 66: (tos 0x2,ECT(0), ttl 127, id 8896, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [SEW], cksum 0x8c89 (correct), seq 655046013, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
16:28:48.928957 00:37:b7:15:a2:66 > aa:b7:69:52:a3:ec, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 57, id 7244, offset 0, flags [none], proto TCP (6), length 52)
    31.55.166.216.80 > 172.16.0.78.4737: Flags [S.], cksum 0x8c1d (correct), seq 3750408064, ack 655046014, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:28:48.929576 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 127, id 8897, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [.], cksum 0xe5a2 (incorrect -> 0x3b00), seq 1, ack 1, win 1024, length 0
16:28:48.929689 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 301: (tos 0x0, ttl 127, id 8898, offset 0, flags [DF], proto TCP (6), length 287)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [P.], cksum 0xe699 (incorrect -> 0x5581), seq 1:248, ack 1, win 1024, length 247
16:28:49.242823 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 301: (tos 0x0, ttl 127, id 8899, offset 0, flags [DF], proto TCP (6), length 287)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [P.], cksum 0xe699 (incorrect -> 0x5581), seq 1:248, ack 1, win 1024, length 247
16:28:49.852286 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 301: (tos 0x0, ttl 127, id 8900, offset 0, flags [DF], proto TCP (6), length 287)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [P.], cksum 0xe699 (incorrect -> 0x5581), seq 1:248, ack 1, win 1024, length 247
16:28:51.055344 aa:b7:69:52:a3:ec > 00:37:b7:15:a2:66, ethertype IPv4 (0x0800), length 301: (tos 0x0, ttl 127, id 8901, offset 0, flags [DF], proto TCP (6), length 287)
    172.16.0.78.4737 > 31.55.166.216.80: Flags [P.], cksum 0xe699 (incorrect -> 0x5581), seq 1:248, ack 1, win 1024, length 247
                                  1 Reply Last reply Reply Quote 0
                                  • R Offline
                                    Rewt0r
                                    last edited by

                                    Seems that my issue may be related to: https://forum.pfsense.org/index.php?topic=85797.0

                                    Reply #4 from cmb suggests that I shouldn't have xn nics when I do, running pfsense 2.2.3 and XenServer 6.5. I have now disabled hardware checksum offload (System -> Advanced -> Networking) and disabled ethtool-tx on all VIFs attached to the pf vm with no luck…

                                    UPDATE: That post mentions to do it for rx as well which has fixed my issue! Thanks all.

                                    1 Reply Last reply Reply Quote 0
                                    • johnpozJ Online
                                      johnpoz LAYER 8 Global Moderator
                                      last edited by

                                      So your issue is you were running on xenserver.. But did not mention that anywhere ;) until now..

                                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                                      If you get confused: Listen to the Music Play
                                      Please don't Chat/PM me for help, unless mod related
                                      SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1

                                      1 Reply Last reply Reply Quote 0
                                      • R Offline
                                        Rewt0r
                                        last edited by

                                        @johnpoz:

                                        So your issue is you were running on xenserver.. But did not mention that anywhere ;) until now..

                                        I did mention it was on a vm on my first post, just forgot to mention the hv… Apologies  :-[

                                        1 Reply Last reply Reply Quote 0
                                        • X Offline
                                          xtofh
                                          last edited by

                                          Those incorrect checkums in your tcpdump do look creepy.. :)

                                          I have no experience with XenServer but can't you use a different type of NIC ? (for example an intel e1000 or so?)

                                          1 Reply Last reply Reply Quote 0
                                          • R Offline
                                            Rewt0r
                                            last edited by

                                            @xtofh:

                                            Those incorrect checkums in your tcpdump do look creepy.. :)

                                            I have no experience with XenServer but can't you use a different type of NIC ? (for example an intel e1000 or so?)

                                            We use Broadcom NetXtreme BCM5720 Gigabit, I have a feeling that this is a XenTools issue…

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.