Port-forwarding, UPNP, NAT-PMP issues I can't resolve...



  • Hi,

    I have a Port-Forwarding issue I just can't seem to resolve, whereby port-forwards (both manual and UPNP/NAT-PMP) work fine to one LAN machine but not another, and I can't for the life of me work out why.

    Background: Home network 192.168.0.x all flat, no VLANs.
    Bridged modem > pfsense router doing PPPoE for WAN > Netgear GS110TP switch > SFP > Netgear GS110TP switch > Ubiquiti access point and wired devices.

    I have two Macs on this network: Macbook 192.168.0.16 and Mac Mini 192.168.0.4. I am testing Transmission bittorrent client on both, and only the Macbook works. For convenient remembering I have set the ports to 10016 and 10004. For troubleshooting I have disabled the Mac firewalls completely and disabled Little Snitch. On pfSense I have disabled both PFBlocker and Suricata.

    My pfSense Firewall rules:

    0_1535185403330_Firewall WAN.jpg
    0_1535185420761_Firewall-LAN.jpg

    My NAT Port Forward settings:

    0_1535186402093_NAT Port Forward.jpg

    My NAT Outbound setting:

    0_1535185456319_NAT-outbound.jpg

    UPNP status shows the two mappings operational:

    0_1535185496961_UPNP status.jpg

    Firewall logging shows the packets to both machines being passed:

    0_1535185535357_Firewall Log.jpg

    Test Port for both machines from pFsense is successful:

    1_1535185715054_Test Port 10016.jpg
    0_1535185715054_Test Port 10004.jpg

    Macbook Transmission shows Port Open but Mac Mini Transmission shows port closed:

    0_1535185782377_MacBook Port Open.jpg

    0_1535185797738_Mac Mini port closed.jpg

    Testing from CanYouSeeMe to my public IP address confirms this:

    0_1535185847190_CanYouSeeMe port 10016.jpg
    0_1535185869758_CanYouSeeMe port 10004.jpg

    I should add that I have also tried disabling UPNP and doing manual port forwards following the Netgate guide but it produced the same situation. I just find it bizarre that one machine works and one doesn't. When I try another application Plex that uses UPNP/NAT-PMP on the Mac Mini it faces the same issue.

    Can anyone guide me on how I might troubleshoot this further? I guess it's possible the problem is not in pfSense but in switch or the Mac itself, but given the probematic Mac Mini shows the port open when tested from pfsense or other LAN devices, it seems the problem is more likely to be in pfSense. Many thanks....


  • Rebel Alliance Global Moderator

    Firewall on the mac mini?

    Here is the thing if pfsense forwards the traffic - simple enough you see that in the rules. And can validate with simple packet capture.. And you do not get an answer then its the device you forwarding to.

    Firewall, the device isn't actually listening on the port you think its listening on.. You have the wrong IP for your forward.. Sending your traffic to wrong machine, etc..

    Pfsense is out of the equation once you see it send the traffic to the IP you say to send it to..

    Just because some app uses UPnP to forward traffic to it, doesn't mean the OS firewall allows that.. Or 3rd party security software/firewall on the device.



  • @johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    Thanks, appreciate the help...

    Firewall on the mac mini?

    Is disabled completely. And Test Port to the Mac Mini shows that port as open

    Here is the thing if pfsense forwards the traffic - simple enough you see that in the rules. And can validate with simple packet capture.. And you do not get an answer then its the device you forwarding to.

    Not sure I understand but the firewall log is showing passing that traffic. Here is a packet capture on the LAN interface for the Mac Mini IP as host, when doing the port check in Transmission:

    12:20:18.050294 IP 192.168.0.4.1900 > 239.255.255.250.1900: UDP, length 98
    12:20:18.104690 IP 192.168.0.1.1900 > 192.168.0.4.1900: UDP, length 365
    12:20:18.215560 IP 192.168.0.4.1900 > 239.255.255.250.1900: UDP, length 98
    12:20:18.215609 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
    12:20:18.269606 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
    12:20:18.269633 IP 192.168.0.1.1900 > 192.168.0.4.1900: UDP, length 365
    12:20:18.346785 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
    12:20:18.400678 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
    12:20:18.615051 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
    12:20:18.615303 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
    12:20:18.669252 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
    12:20:18.669276 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
    12:20:18.825153 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
    12:20:18.879325 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
    12:20:19.046467 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
    12:20:19.100646 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
    12:20:19.929573 IP 192.168.0.4.138 > 192.168.0.255.138: UDP, length 198
    12:20:19.929830 IP 192.168.0.4.137 > 192.168.0.255.137: UDP, length 50
    12:20:21.193944 IP 192.168.0.4.56160 > 217.146.21.134.5938: tcp 24
    12:20:21.323048 IP 192.168.0.4.49453 > 170.79.112.26.45609: UDP, length 97
    12:20:21.411679 IP 217.146.21.134.5938 > 192.168.0.4.56160: tcp 0
    12:20:21.676011 IP 170.79.112.26 > 192.168.0.4: ICMP 170.79.112.26 udp port 45609 unreachable, length 133
    12:20:21.933476 IP 192.168.0.4.137 > 192.168.0.255.137: UDP, length 50
    12:20:22.797022 IP 139.162.54.192.443 > 192.168.0.4.64457: tcp 34
    12:20:22.797383 IP 192.168.0.4.64457 > 139.162.54.192.443: tcp 0
    12:20:26.162510 IP 192.168.0.4.50216 > 87.98.162.88.443: tcp 257
    12:20:26.325823 IP 87.98.162.88.52390 > 192.168.0.4.10004: tcp 0
    12:20:26.326102 IP 192.168.0.4.10004 > 87.98.162.88.52390: tcp 0
    12:20:26.339765 IP 87.98.162.88.443 > 192.168.0.4.50216: tcp 0
    12:20:26.507901 IP 87.98.162.88.443 > 192.168.0.4.50216: tcp 269
    12:20:26.508269 IP 192.168.0.4.50216 > 87.98.162.88.443: tcp 0

    12:20:30.679581 IP 192.168.0.4.56150 > 17.188.136.186.5223: tcp 101
    12:20:31.101039 IP 17.188.136.186.5223 > 192.168.0.4.56150: tcp 53
    12:20:31.101247 IP 192.168.0.4.56150 > 17.188.136.186.5223: tcp 0

    87.98.162.88 is the Transmission port check server if that helps. Let me know if I should do some different type of packet capture, sorry I'm not very experienced with that.

    Firewall, the device isn't actually listening on the port you think its listening on.. You have the wrong IP for your forward.. Sending your traffic to wrong machine, etc..

    I'm letting UPNP handle that, and it's showing in UPNP status as mapped to the correct LAN IP
    192.168.0.4, but yes I guess it could be something there.

    Pfsense is out of the equation once you see it send the traffic to the IP you say to send it to..

    Just because some app uses UPnP to forward traffic to it, doesn't mean the OS firewall allows that.. Or 3rd party security software/firewall on the device.

    Sure, but presumably if there was firewall type issue on the Mac Mini then I wouldn't be seeing successful Test Port from pfSense.


  • Rebel Alliance Global Moderator

    @occamsrazor said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    12:20:26.325823 IP 87.98.162.88.52390 > 192.168.0.4.10004: tcp 0
    12:20:26.326102 IP 192.168.0.4.10004 > 87.98.162.88.52390: tcp 0

    Ok there is answer to 10004.. But what was the answer maybe it was Reset.. The first there should be Syn to open the connection.. But what did he answer?

    Do the sniff on pfsense, diag, packet capture -- put in your IP 192.168.0.4 and the port using lan as capture interface - and then open the sniff in wireshark.. You should see the syn and syn,ack answer.. If sending syn,ack - and you see that on pfsense lan, but you do not see it go out the wan then you have a problem. If you see it go out pfsense wan then you go problem upstream.



  • @johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    Do the sniff on pfsense, diag, packet capture -- put in your IP 192.168.0.4 and the port using lan as capture interface - and then open the sniff in wireshark.. You should see the syn and syn,ack answer.. If sending syn,ack - and you see that on pfsense lan, but you do not see it go out the wan then you have a problem. If you see it go out pfsense wan then you go problem upstream.

    Here is the capture on the LAN for Mac Mini host 192.168.0.4 and port 10004:

    0_1535190789033_Wireshark LAN capture.jpg

    When I do the same capture on WAN it is empty.

    I then tried on the Macbook and got this:

    0_1535191539195_Wireshark MacbookLAN capture.jpg

    I don't see the Transmission port check server there, and now the Macbook is having the same issue and is showing port closed like the Mac Mini did.

    EDIT: Now the Macbook is working again and this is what the LAN capture looks like:

    0_1535191915269_Screen Shot 2018-08-25 at 13.10.06.jpg


  • Rebel Alliance Global Moderator

    Your first image there shows the client sending RST!!! So no the port is not going to show open because the client basically said in a nutshell - F OFF!!

    You can not actually validate UDP ports with sites like can you see me org..



  • @johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    Thanks again for your help in nailing down the cause, much appreciated...

    Your first image there shows the client sending RST!!! So no the port is not going to show open because the client basically said in a nutshell - F OFF!!

    Do you have any idea why it might be sending this "RST"? Is it likely the OS sending it or Transmission? Any ideas on how one might fix the problem?

    You can not actually validate UDP ports with sites like can you see me org..

    I think the problematic packet and the RST are a TCP packet... no?


  • Rebel Alliance Global Moderator

    The RST is tcp yes... But you can not actually know if something is working or not with canyouseeme if its UDP. P2P is going to normally be tcp and udp..

    As to why your client is answer back with RST, no idea - but that is not anything to do with with pfsense.. You would have to get with mac os users or forum support your transmission client, etc.

    What OS you running on this mac mini? Is apple or a linux OS? If linux could be of some help, but I don't use OS X so not going to be much help trying to figure out what is sending RST... What I can tell you is that is BAD practice.. Especially to a public IP.. sending RST to IP that is on the same local network is ok - but answering RST to some ip that is not local is bad.. Firewalls don't normally do that, because its bad idea and if under attack for some sort of dos attack your just going to be hurting yourself sending a RST..

    Is it sending a icmp redirection as well? Sniff for the IP on the pfsense lan and do your test again.. Any as the protocol, ipv4



  • @johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    As to why your client is answer back with RST, no idea - but that is not anything to do with with pfsense.. You would have to get with mac os users or forum support your transmission client, etc.

    Thanks. Ugh, I think this is going to be hard to nail down. I'm running Mac OSX 10.12.

    Is it sending a icmp redirection as well? Sniff for the IP on the pfsense lan and do your test again.. Any as the protocol, ipv4

    I don't think so. Here is an all ports capture of the Mac Mini.

    0_1535264354125_MacMini all ports.jpg

    I have tried complete uninstall and reinstall of Transmission but it didn't help. I am thinking it must be something in the OS itself, rather than Transmission, because the same issue happens when I try to get remote access to Plex media server....

    0_1535264429260_Plex RST.jpg

    PS - Ignore the change from 192.168.0.4 to .2 - that was me changing the IP for testing, it's the same machine.

    I've posted for advice on the r/osx subreddit here: https://www.reddit.com/r/osx/comments/9adpuk/osx_networking_issue_port_forwards_failing_rst/


  • Rebel Alliance Global Moderator

    I would have to guess your firewall is still running.. But in a bad configuration - what OS are you running?

    Did actually validate listening on those ports with say a netstat -an.. This command works on any os really.. window, linux, os

    Many OSes if hit directly with no firewall will send RST when port is not listing..

    0_1535280265901_window10RST.png

    Im not listening on 444, if I forward 444 to my windows machine it sends back a RST..



  • @johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:

    I would have to guess your firewall is still running.. But in a bad configuration - what OS are you running?

    Did actually validate listening on those ports with say a netstat -an.. This command works on any os really.. window, linux, os x

    I did a netstat and traceroute, and posted the results in my post on Reddit here (I didn't want to clog up this forum any more seeing as it's clearly an OSX issue not pfSense): https://www.reddit.com/r/osx/comments/9adpuk/osx_networking_issue_port_forwards_failing_rst/e4urk4v/

    I tried two other things: Using a different user account on the Mac Mini = same problem. Booting the mac Mini from a clean OSX install on external drive = NO problem. This clearly indicates to me it's an issue with the OSX install... I think when I have time I'll have to nuke the whole OS and start all over again, but that's going to take some serious time so I'll have to put that plan on hold.

    Appreciate all the help - at least I don't need to spend any more time investigating pfSense, switches, etc.


  • Rebel Alliance Global Moderator

    Those are netstat -r, just showing your routes... Not what your machine is actually listening on.. For example

    See here listening on 3389 (remote desktop) on my windows machine
    0_1535281836569_listen3389.png

    actually validate your machine is listening on the port you think it is! tcp will send RST when not listening.. But firewall normally default to "stealth" mode..



  • I was unable to do netstat with the "find" addition like in your screenshot, maybe syntax is different in OSX. But when I do "netstat -an" and then search for 10002 in the long list of output, I see these items (Transmission port is now set to 10002):

    Proto Recv-Q Send-Q  Local Address          Foreign Address        (state) 
    
    tcp6       0      0  *.10002                *.*                    LISTEN     
    tcp4       0      0  *.10002                *.*                    LISTEN
    
    tcp4       0      0  192.168.0.2.10002      185.230.125.35.37657   TIME_WAIT  
    tcp4       0      0  192.168.0.2.10002      87.67.39.175.56083     TIME_WAIT  
    tcp4       0      0  192.168.0.2.10002      84.104.165.198.63194   TIME_WAIT  
    tcp4       0      0  192.168.0.2.10002      82.173.50.104.57184    TIME_WAIT 
    
    udp4       0      0  *.10002                *.*    
    
    

  • Rebel Alliance Global Moderator

    Well yeah its different in windows ;)

    So something is listening on 10002, And looks like your in a time_wait connection with 4 other IPs..

    Time wait means that hey this connection should be close but will leave the socket open until the time out..

    You sure you don't have any sort of ACL or something in the software that says who can talk to it? Do you see the same thing for your plex?

    Normally!!! If you your in a time_wait state and you get more traffic on that same conversation setup ie same IP and same port as source the server (listening side) would for sure send back a RST.. Telling you hey this conversation is closed..

    If the socket was in in time_wait and it got an RST then I believe it should close right then.. You seeing sockets in time_wait is another wrench in the issue..

    But again - this has zero to do with pfsense, or connectivity at all.. And something with the OS or application on the box.. Pfsense did its job it sent the SYN through.. It can not help the that client sends back RST..



  • Ok I worked out the OSX netstat syntax to focus on particular ports. Here is for Transmission and then for the Plex port:

    mediamac:~ ben$ netstat -an |grep .10002
    tcp4       0      0  192.168.0.2.10002      94.245.58.211.60686    SYN_RCVD   
    tcp4       0      0  192.168.0.2.10002      178.82.144.61.60173    ESTABLISHED
    tcp6       0      0  *.10002                *.*                    LISTEN     
    tcp4       0      0  *.10002                *.*                    LISTEN     
    tcp4       0      0  192.168.0.2.10002      84.104.165.198.58890   TIME_WAIT  
    udp4       0      0  *.10002                              
    mediamac:~ ben$ netstat -an | grep .32400
    tcp4       0      0  127.0.0.1.32400        127.0.0.1.51153        ESTABLISHED
    tcp4       0      0  127.0.0.1.51153        127.0.0.1.32400        ESTABLISHED
    tcp4       0      0  127.0.0.1.32400        127.0.0.1.51096        ESTABLISHED
    tcp4       0      0  127.0.0.1.51096        127.0.0.1.32400        ESTABLISHED
    tcp4       0      0  127.0.0.1.32400        127.0.0.1.49318        ESTABLISHED
    tcp4       0      0  127.0.0.1.49318        127.0.0.1.32400        ESTABLISHED
    tcp46      0      0  *.32400                *.*                    LISTEN
    

    So it seems same for Plex. And agreed nothing to do with pfSense....


  • Rebel Alliance Global Moderator

    Looks like in your 10002 your in syn_rcvd state from that 1 IP.. And do have an established connection with 178.82.x.x and in a time wait for another..

    So your going to have to work out whatever is wrong if your client or ACL... Possible your running a block list in transmission, you know keep the "spies" out... isn't there large black lists of the bad guys to try and keep out of the swarm?

    As to plex.. Maybe something common with firewall running that you think you turned off, or 3rd party security software you running on the box?



  • SOLVED.....

    On the offchance it helps someone else, in the end I fixed it by...

    1. Deleting the OSX firewall preference .plist located at /Library/Preferences/com.apple.alf.plist
    2. Cleaning system & user caches
    3. Restarting
    4. Enabling the firewall
    5. Doing this all a couple of times...

    What's strange is this whole time the firewall was OFF in System Preferences, so I've no idea why this should affect things. In addition, I had previously during testing deleted this file but seems you have to do it and restart, maybe more than once, to really clear it.

    Thanks for all the patient help in narrowing down the problem, really much appreciated....


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy