Port Forwarding Issue in VM



  • Greetings,

    I have a PFSense box set up in an Exsi VM.  The connection is as follows:

    ATT Modem/Router > vswitch1 (WAN setup) > vswitch0 (LAN)

    The pfsense VM is the only machine in the WAN switch.

    Because the ATT modem is also a router, I've allowed the ATT router to assign an IP via DHCP to the pfsense box, and placed it in the DMZ.  I've then unchecked "Block private networks and loopback addresses" in the WAN interface.

    I have also set up an OpenVPN server, which works just fine and I am able to connect outside the network.  However, no amount of port forwarding works.  Regardless of what ports I open, I cannot get through the firewall.

    I have also checked the clients and they are allowing all connections on all ports, so it's something to do with the pfsense machine.


    Thanks for any help.


  • Rebel Alliance Global Moderator

    And have you gone through the port forwarding troubleshooting guide?

    https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

    If so where in in that guide are you failing?

    What are you forwarding too, other vms?  Something on the physical network?  You checked the clients how exactly?  What are the clients are they running firewall - windows firewall by default likes to block stuff outside its own local network for example.



  • Thanks for the reply.  Yes, this is a port forward to other VMs within the same machine.

    Both of those clients are linux VMs, one Ubuntu, the other Debian 8.

    I've run iptables -L, which brings back nothing indicating they do not have any firewall rules set up.  Previously I had an Arch machine that I had installed ufw and had allowed all traffic, with still no luck.

    1. NAT and firewall rules not correctly added (see How can I forward ports with pfSense?)
    Hint: Do NOT set a source port
    Tried this, no source ports set.
    2. Firewall enabled on client machine
    Confirmed via iptables -L
    3. Client machine is not using pfSense as its default gateway
    netroute shows the client machine using the correct gateway
    4. Client machine not actually listening on the port being forwarded
    The programs needing ports are running and listening on the respective ports.  They can be accessed from within the network
    5. ISP or something upstream of pfSense is blocking the port being forwarded
    The router is in DMZ, so it shouldn't need additional configuration
    6. Trying to test from inside the local network, need to test from an outside machine
    Using neighbor's internet to test this
    7. Incorrect or missing Virtual IP configuration for additional public IP addresses
    I don't have additional public IP's
    8. The pfSense router is not the border router. If there is something else between pfSense and the ISP, the port forwards and associated rules must be replicated there.
    DMZ should handle this.
    9. Forwarding ports to a server behind a Captive Portal. An IP bypass must be added both to and from the server's IP in order for a port forward to work behind a Captive Portal.
    I don't have a captive portal
    10. If this is on a WAN that is not the default gateway, make sure there is a gateway chosen on this WAN interface, or the firewall rules for the port forward would not reply back via the correct gateway.
    The clients use the gateway supplied by pfsense
    11. If this is on a WAN that is not the default gateway, ensure the traffic for the port forward is NOT passed in via Floating Rules or an Interface Group. Only rules present on the WAN's interface tab under Firewall Rules will have the reply-to keyword to ensure the traffic responds properly via the expected gateway.
    The WAN is the default gateway.
    12. If this is on a WAN that is not the default gateway, make sure the firewall rule(s) allowing the traffic in do not have the box checked to disable reply-to.
    The WAN is the default gateway
    13. If this is on a WAN that is not the default gateway, make sure the master reply-to disable switch is not checked under System > Advanced, on the Firewall/NAT tab.
    The WAN is the default gateway
    14. WAN rules should NOT have a gateway set, so make sure that the rules for the port forward do NOT have a gateway configured on the actual rule.
    I don't have any rules set for gateway
    15. If the traffic appears to be forwarding in to an unexpected device, it may be happening due to UPnP. Check Status > UPnP to see if an internal service has configured a port forward unexpectedly. If so, disable UPnP on either that device or on the firewall.
    UPnP is not running on these services

    I am not 100% sure on these, especially the gateway items, and am happy to post up outputs of any requested commands or screenshots.

    Thanks.



  • I confirmed with the packet sniffer that traffic was coming to the WAN properly.  It even shows that it's pointing it to the correct internal IP and port.

    The pfsense machine and the ATT router share the same gateway.  Traffic passes in and out, but does not connect to each other leading me to believe the rules are set up incorrectly, especially since the auto-generated OpenVPN one works just fine.

    the local machine:
    sudo netstat -nr
    Kernel IP routing table
    Destination    Gateway        Genmask        Flags  MSS Window  irtt Iface
    0.0.0.0        192.168.1.1    0.0.0.0        UG        0 0          0 eth0
    192.168.1.0    0.0.0.0        255.255.255.0  U        0 0          0 eth0



  • Further testing leads me to believe that the firewall pfsense has is the issue, not the port mapping.

    I enabled UPnP and let Plex enable it's own mapping.  It was able to link the internal ip with its respective port to the external IP and a listen port, but the connection to the internet was not working.  I am not sure if this is an issue with traffic getting in or out.

    Thanks for any help.


  • Rebel Alliance Global Moderator

    So your seeing the traffic on the wan, and its going to your wan IP on the dest port you want to forward.  So sniff on your pfsense lan do you see the traffic going to the IP you wan it to go to?  Do you see a reply?  If you see it sent to your inside IP but no reply then its the device. This device has internet connectivity through pfsense?

    So your behind a nat router, what is this network on the pfsense wan?  Its not the same or overlapping the network you have on your pfsense lan?

    I run multiple vms on esxi in multiple segments behind pfsense.  And have no problems forwarding to them..  What does you esxi network look like.. You say your wan is on its own vswitch, so your lan is setup like that as well?  With the vms connected to this same vswitch and pfsense lan in that vswitch.  See my attached esxi network.

    So I setup a forward for ssh to ubuntu vm.  I then sniff on lan of pfsense and you see the traffic go through and get a reply.  Since you have validated the traffic is getting there.  You don't have any other rules on the pfsense wan that would block?  Doesn't look like your showing the rules above your port forward rules on your wan.  Can not tell from your screen shot that there are no rules above those.

    So you see my esxi setup on the left.  Where pfsense lan and ubuntu vm are connected to the same lan vswitch.  See my forward and rules, and on the packet capture on the pfsense lan you can see the traffic go to 192.168.9.7 and get an answer.




  • So your seeing the traffic on the wan, and its going to your wan IP on the dest port you want to forward.  So sniff on your pfsense lan do you see the traffic going to the IP you wan it to go to?  Do you see a reply?  If you see it sent to your inside IP but no reply then its the device. This device has internet connectivity through pfsense?
    Yes, this machine can get outside the network.  I can ping websites, and the service i'm trying to forward was able to pull from a podcast website no problem.  See packet sniffing below.

    So your behind a nat router, what is this network on the pfsense wan?  Its not the same or overlapping the network you have on your pfsense lan?
    I'm sorry I'm not sure what this means.

    Pictures below of setup and rules and sniffing.




    WAN Sniff
    12:06:52.851946 IP 216.84.175.230.57543 > 108.81.xx.4040: tcp 0
    12:06:52.852005 IP 216.84.175.230.57543 > 196.168.1.165.4040: tcp 0
    12:06:53.854669 IP 216.84.175.230.57543 > 108.81.xx.4040: tcp 0
    12:06:53.854688 IP 216.84.175.230.57543 > 196.168.1.165.4040: tcp 0

    LAN Sniff
    12:18:02.049751 ARP, Request who-has 192.168.1.1 tell 192.168.1.165, length 46
    12:18:02.049765 ARP, Reply 192.168.1.1 is-at 00:0c:29:8d:f8:cf, length 28
    12:18:04.065941 IP 192.168.1.165.33597 > 192.168.1.255.32412: UDP, length 21
    12:18:04.066054 IP 192.168.1.165.56683 > 192.168.1.255.32414: UDP, length 21
    12:18:04.495476 IP 192.168.2.2.55412 > 192.168.1.165.4040: tcp 684
    12:18:04.527141 IP 192.168.1.165.4040 > 192.168.2.2.55412: tcp 101

    These were just snippets of the whole sniff.  The first one was me trying to access the machine via the port from an external network.  The LAN sniff was me inside the network pulling the podcast.
    I also have looked at the firewall logs and summary and the graph shows that the firewall blocks 100% of traffic.  I'm not sure if this is just the last 50 entries or everything…


  • Rebel Alliance Global Moderator

    12:18:04.495476 IP 192.168.2.2.55412 > 192.168.1.165.4040: tcp 684
    12:18:04.527141 IP 192.168.1.165.4040 > 192.168.2.2.55412: tcp 101

    What is that traffic, source of 192.168.2.2 ???  Are you trying to source nat?  What has IP address 192.168.2.2?  Is that were you testing from on pfsense wan? From your wan sniff you show your public IP being 108.81.x.x which I would prob not post on public internet.. Notice in my sniff the traffic is the actual public IP that pfsense sends on, and that is the IP the box answers too.

    I show some sort of response from 192.168.1.165 but its sending it to some 192.168.2 network??  Do you have other segments connected to pfsense?

    On side not, seems your vmkern is sharing same phy nic as your lan vswitch… As a suggestion, it looks like you have a spare nic if me I would use that as your vmkern, its can be on the same network as your lan network.



  • I am away from the machine's network at this time, and used a VPN to get into it to get the screen shots.  My VPN defaults 192.168.2.x to VPN'd machines from the outside, so that was me remotely.

    If I disconnect from the VPN and try to access the public IP with that port, it will time out.

    I see that yours has an external IP > internal IP, which my WAN sniff seems to show, but I don't know why it shows it in two lines:
    12:06:52.851946 IP 216.84.175.230.57543 > 108.81.xx.4040: tcp 0
    12:06:52.852005 IP 216.84.175.230.57543 > 196.168.1.165.4040: tcp 0

    "Notice in my sniff the traffic is the actual public IP that pfsense sends on, and that is the IP the box answers too."
    I am not sure how to make this happen.

    "On side not, seems your vmkern is sharing same phy nic as your lan vswitch… As a suggestion, it looks like you have a spare nic if me I would use that as your vmkern, its can be on the same network as your lan network."

    I agree, and will need to figure out how to make this work.  This machine has five NICs so it should be easy.


  • Rebel Alliance Global Moderator

    well vpn is not a valid test of port forwarding.  It doesn't got through the port forward.  It uses the vpn firewall rules.  Can you access stuff via vpn?  It clearly sent you some response - but might of been a RST?  You would have to open sniff in wireshark to see what it sent back, should of been a syn,ack to your syn.

    Not sure how to make what happen?  You just need to sniff while your testing from outside your network.. Use your phone cell data for example.  If that is your actual IP I can do a test from outside for you.  I see you blocked out your public IP.. I don't recall what it was - send it to me via PM and I will do a port scan to you to see if any of those ports are showing open.



  • Thank you for the replies.  I just use the VPN to get into the network to access machines locally to get those screenshots.  All my testing of ports has been with my phone via LTE or another wifi outside my network.

    I've also used port testing websites and they've come back as closed so I'm not really sure where to go from here…

    I'm not sure what an RST is or syn either.

    Using the VPN, I can get in just fine and access both the machines and their services using their internal IP's and ports.  Using LTE, it just times out.  This is true for all machines, regardless of service or port forwarding.

    Not sure where to go from here...


  • Rebel Alliance Global Moderator

    Well we need to do a sniff on your lan while your testing from the outside to validate pfsense is actually sending traffic..  Your forwards look correct, and you have stated your seeing the traffic to the wan address to correct port.  Now your wan is blocking bogon.  Not sure what IP your coming from on the wan test?

    It shows it in 2 lines because the other is the answer!!

    This is outside talking to your wan IP going to port 4040 that you have forwarded.
    12:06:52.851946 IP 216.84.175.230.57543 > 108.81.xx.4040: tcp 0

    This is some sort of answer.  This seems to show its working.. Ie pfsense sent the traffic on and that box answered.. Or could be pfsense telling the box to F off with a RST.. But you don't show any rules set to reject.  So pfsense would just ignore the inbound traffic and there would be no reply if forward didn't work or the box behind didn't answer.
    12:06:52.852005 IP 216.84.175.230.57543 > 196.168.1.165.4040: tcp 0

    If you want to setup a teamviewer session could take a quick look see.  This really is only a couple of minutes to troubleshoot to see what is not right.  Your not trying to use subsonic over https are you - I believe that default port for subsonic is 4443 which you do not have forwarded.  But sure would work over vpn.

    The answer your seeing could be your box sending a RST telling them hey sorry we not listening on that port..  Or it could be redirect to 4443?  What does your firewall log show when you try and make a connection from outside?



  • PM'd you, but it's not showing up in my sent items so let me know if you didn't get it.

    I'm not using subsonic over HTTPS so it should still be ok.  I'm not 100% sure on how to perform a proper LAN test so let's see if we can try that.