Port-forwarding, UPNP, NAT-PMP issues I can't resolve...
-
Hi,
I have a Port-Forwarding issue I just can't seem to resolve, whereby port-forwards (both manual and UPNP/NAT-PMP) work fine to one LAN machine but not another, and I can't for the life of me work out why.
Background: Home network 192.168.0.x all flat, no VLANs.
Bridged modem > pfsense router doing PPPoE for WAN > Netgear GS110TP switch > SFP > Netgear GS110TP switch > Ubiquiti access point and wired devices.I have two Macs on this network: Macbook 192.168.0.16 and Mac Mini 192.168.0.4. I am testing Transmission bittorrent client on both, and only the Macbook works. For convenient remembering I have set the ports to 10016 and 10004. For troubleshooting I have disabled the Mac firewalls completely and disabled Little Snitch. On pfSense I have disabled both PFBlocker and Suricata.
My pfSense Firewall rules:
My NAT Port Forward settings:
My NAT Outbound setting:
UPNP status shows the two mappings operational:
Firewall logging shows the packets to both machines being passed:
Test Port for both machines from pFsense is successful:
Macbook Transmission shows Port Open but Mac Mini Transmission shows port closed:
Testing from CanYouSeeMe to my public IP address confirms this:
I should add that I have also tried disabling UPNP and doing manual port forwards following the Netgate guide but it produced the same situation. I just find it bizarre that one machine works and one doesn't. When I try another application Plex that uses UPNP/NAT-PMP on the Mac Mini it faces the same issue.
Can anyone guide me on how I might troubleshoot this further? I guess it's possible the problem is not in pfSense but in switch or the Mac itself, but given the probematic Mac Mini shows the port open when tested from pfsense or other LAN devices, it seems the problem is more likely to be in pfSense. Many thanks....
-
Firewall on the mac mini?
Here is the thing if pfsense forwards the traffic - simple enough you see that in the rules. And can validate with simple packet capture.. And you do not get an answer then its the device you forwarding to.
Firewall, the device isn't actually listening on the port you think its listening on.. You have the wrong IP for your forward.. Sending your traffic to wrong machine, etc..
Pfsense is out of the equation once you see it send the traffic to the IP you say to send it to..
Just because some app uses UPnP to forward traffic to it, doesn't mean the OS firewall allows that.. Or 3rd party security software/firewall on the device.
-
@johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
Thanks, appreciate the help...
Firewall on the mac mini?
Is disabled completely. And Test Port to the Mac Mini shows that port as open
Here is the thing if pfsense forwards the traffic - simple enough you see that in the rules. And can validate with simple packet capture.. And you do not get an answer then its the device you forwarding to.
Not sure I understand but the firewall log is showing passing that traffic. Here is a packet capture on the LAN interface for the Mac Mini IP as host, when doing the port check in Transmission:
12:20:18.050294 IP 192.168.0.4.1900 > 239.255.255.250.1900: UDP, length 98
12:20:18.104690 IP 192.168.0.1.1900 > 192.168.0.4.1900: UDP, length 365
12:20:18.215560 IP 192.168.0.4.1900 > 239.255.255.250.1900: UDP, length 98
12:20:18.215609 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
12:20:18.269606 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
12:20:18.269633 IP 192.168.0.1.1900 > 192.168.0.4.1900: UDP, length 365
12:20:18.346785 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
12:20:18.400678 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
12:20:18.615051 IP 192.168.0.4.58549 > 239.255.255.250.1900: UDP, length 98
12:20:18.615303 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
12:20:18.669252 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
12:20:18.669276 IP 192.168.0.1.1900 > 192.168.0.4.58549: UDP, length 365
12:20:18.825153 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
12:20:18.879325 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
12:20:19.046467 IP 192.168.0.4.49501 > 239.255.255.250.1900: UDP, length 98
12:20:19.100646 IP 192.168.0.1.1900 > 192.168.0.4.49501: UDP, length 365
12:20:19.929573 IP 192.168.0.4.138 > 192.168.0.255.138: UDP, length 198
12:20:19.929830 IP 192.168.0.4.137 > 192.168.0.255.137: UDP, length 50
12:20:21.193944 IP 192.168.0.4.56160 > 217.146.21.134.5938: tcp 24
12:20:21.323048 IP 192.168.0.4.49453 > 170.79.112.26.45609: UDP, length 97
12:20:21.411679 IP 217.146.21.134.5938 > 192.168.0.4.56160: tcp 0
12:20:21.676011 IP 170.79.112.26 > 192.168.0.4: ICMP 170.79.112.26 udp port 45609 unreachable, length 133
12:20:21.933476 IP 192.168.0.4.137 > 192.168.0.255.137: UDP, length 50
12:20:22.797022 IP 139.162.54.192.443 > 192.168.0.4.64457: tcp 34
12:20:22.797383 IP 192.168.0.4.64457 > 139.162.54.192.443: tcp 0
12:20:26.162510 IP 192.168.0.4.50216 > 87.98.162.88.443: tcp 257
12:20:26.325823 IP 87.98.162.88.52390 > 192.168.0.4.10004: tcp 0
12:20:26.326102 IP 192.168.0.4.10004 > 87.98.162.88.52390: tcp 0
12:20:26.339765 IP 87.98.162.88.443 > 192.168.0.4.50216: tcp 0
12:20:26.507901 IP 87.98.162.88.443 > 192.168.0.4.50216: tcp 269
12:20:26.508269 IP 192.168.0.4.50216 > 87.98.162.88.443: tcp 0
12:20:30.679581 IP 192.168.0.4.56150 > 17.188.136.186.5223: tcp 101
12:20:31.101039 IP 17.188.136.186.5223 > 192.168.0.4.56150: tcp 53
12:20:31.101247 IP 192.168.0.4.56150 > 17.188.136.186.5223: tcp 087.98.162.88 is the Transmission port check server if that helps. Let me know if I should do some different type of packet capture, sorry I'm not very experienced with that.
Firewall, the device isn't actually listening on the port you think its listening on.. You have the wrong IP for your forward.. Sending your traffic to wrong machine, etc..
I'm letting UPNP handle that, and it's showing in UPNP status as mapped to the correct LAN IP
192.168.0.4, but yes I guess it could be something there.Pfsense is out of the equation once you see it send the traffic to the IP you say to send it to..
Just because some app uses UPnP to forward traffic to it, doesn't mean the OS firewall allows that.. Or 3rd party security software/firewall on the device.
Sure, but presumably if there was firewall type issue on the Mac Mini then I wouldn't be seeing successful Test Port from pfSense.
-
@occamsrazor said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
12:20:26.325823 IP 87.98.162.88.52390 > 192.168.0.4.10004: tcp 0
12:20:26.326102 IP 192.168.0.4.10004 > 87.98.162.88.52390: tcp 0Ok there is answer to 10004.. But what was the answer maybe it was Reset.. The first there should be Syn to open the connection.. But what did he answer?
Do the sniff on pfsense, diag, packet capture -- put in your IP 192.168.0.4 and the port using lan as capture interface - and then open the sniff in wireshark.. You should see the syn and syn,ack answer.. If sending syn,ack - and you see that on pfsense lan, but you do not see it go out the wan then you have a problem. If you see it go out pfsense wan then you go problem upstream.
-
@johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
Do the sniff on pfsense, diag, packet capture -- put in your IP 192.168.0.4 and the port using lan as capture interface - and then open the sniff in wireshark.. You should see the syn and syn,ack answer.. If sending syn,ack - and you see that on pfsense lan, but you do not see it go out the wan then you have a problem. If you see it go out pfsense wan then you go problem upstream.
Here is the capture on the LAN for Mac Mini host 192.168.0.4 and port 10004:
When I do the same capture on WAN it is empty.
I then tried on the Macbook and got this:
I don't see the Transmission port check server there, and now the Macbook is having the same issue and is showing port closed like the Mac Mini did.
EDIT: Now the Macbook is working again and this is what the LAN capture looks like:
-
Your first image there shows the client sending RST!!! So no the port is not going to show open because the client basically said in a nutshell - F OFF!!
You can not actually validate UDP ports with sites like can you see me org..
-
@johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
Thanks again for your help in nailing down the cause, much appreciated...
Your first image there shows the client sending RST!!! So no the port is not going to show open because the client basically said in a nutshell - F OFF!!
Do you have any idea why it might be sending this "RST"? Is it likely the OS sending it or Transmission? Any ideas on how one might fix the problem?
You can not actually validate UDP ports with sites like can you see me org..
I think the problematic packet and the RST are a TCP packet... no?
-
The RST is tcp yes... But you can not actually know if something is working or not with canyouseeme if its UDP. P2P is going to normally be tcp and udp..
As to why your client is answer back with RST, no idea - but that is not anything to do with with pfsense.. You would have to get with mac os users or forum support your transmission client, etc.
What OS you running on this mac mini? Is apple or a linux OS? If linux could be of some help, but I don't use OS X so not going to be much help trying to figure out what is sending RST... What I can tell you is that is BAD practice.. Especially to a public IP.. sending RST to IP that is on the same local network is ok - but answering RST to some ip that is not local is bad.. Firewalls don't normally do that, because its bad idea and if under attack for some sort of dos attack your just going to be hurting yourself sending a RST..
Is it sending a icmp redirection as well? Sniff for the IP on the pfsense lan and do your test again.. Any as the protocol, ipv4
-
@johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
As to why your client is answer back with RST, no idea - but that is not anything to do with with pfsense.. You would have to get with mac os users or forum support your transmission client, etc.
Thanks. Ugh, I think this is going to be hard to nail down. I'm running Mac OSX 10.12.
Is it sending a icmp redirection as well? Sniff for the IP on the pfsense lan and do your test again.. Any as the protocol, ipv4
I don't think so. Here is an all ports capture of the Mac Mini.
I have tried complete uninstall and reinstall of Transmission but it didn't help. I am thinking it must be something in the OS itself, rather than Transmission, because the same issue happens when I try to get remote access to Plex media server....
PS - Ignore the change from 192.168.0.4 to .2 - that was me changing the IP for testing, it's the same machine.
I've posted for advice on the r/osx subreddit here: https://www.reddit.com/r/osx/comments/9adpuk/osx_networking_issue_port_forwards_failing_rst/
-
I would have to guess your firewall is still running.. But in a bad configuration - what OS are you running?
Did actually validate listening on those ports with say a netstat -an.. This command works on any os really.. window, linux, os
Many OSes if hit directly with no firewall will send RST when port is not listing..
Im not listening on 444, if I forward 444 to my windows machine it sends back a RST..
-
@johnpoz said in Port-forwarding, UPNP, NAT-PMP issues I can't resolve...:
I would have to guess your firewall is still running.. But in a bad configuration - what OS are you running?
Did actually validate listening on those ports with say a netstat -an.. This command works on any os really.. window, linux, os x
I did a netstat and traceroute, and posted the results in my post on Reddit here (I didn't want to clog up this forum any more seeing as it's clearly an OSX issue not pfSense): https://www.reddit.com/r/osx/comments/9adpuk/osx_networking_issue_port_forwards_failing_rst/e4urk4v/
I tried two other things: Using a different user account on the Mac Mini = same problem. Booting the mac Mini from a clean OSX install on external drive = NO problem. This clearly indicates to me it's an issue with the OSX install... I think when I have time I'll have to nuke the whole OS and start all over again, but that's going to take some serious time so I'll have to put that plan on hold.
Appreciate all the help - at least I don't need to spend any more time investigating pfSense, switches, etc.
-
Those are netstat -r, just showing your routes... Not what your machine is actually listening on.. For example
See here listening on 3389 (remote desktop) on my windows machine
actually validate your machine is listening on the port you think it is! tcp will send RST when not listening.. But firewall normally default to "stealth" mode..
-
I was unable to do netstat with the "find" addition like in your screenshot, maybe syntax is different in OSX. But when I do "netstat -an" and then search for 10002 in the long list of output, I see these items (Transmission port is now set to 10002):
Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp6 0 0 *.10002 *.* LISTEN tcp4 0 0 *.10002 *.* LISTEN tcp4 0 0 192.168.0.2.10002 185.230.125.35.37657 TIME_WAIT tcp4 0 0 192.168.0.2.10002 87.67.39.175.56083 TIME_WAIT tcp4 0 0 192.168.0.2.10002 84.104.165.198.63194 TIME_WAIT tcp4 0 0 192.168.0.2.10002 82.173.50.104.57184 TIME_WAIT udp4 0 0 *.10002 *.* -
Well yeah its different in windows ;)
So something is listening on 10002, And looks like your in a time_wait connection with 4 other IPs..
Time wait means that hey this connection should be close but will leave the socket open until the time out..
You sure you don't have any sort of ACL or something in the software that says who can talk to it? Do you see the same thing for your plex?
Normally!!! If you your in a time_wait state and you get more traffic on that same conversation setup ie same IP and same port as source the server (listening side) would for sure send back a RST.. Telling you hey this conversation is closed..
If the socket was in in time_wait and it got an RST then I believe it should close right then.. You seeing sockets in time_wait is another wrench in the issue..
But again - this has zero to do with pfsense, or connectivity at all.. And something with the OS or application on the box.. Pfsense did its job it sent the SYN through.. It can not help the that client sends back RST..
-
Ok I worked out the OSX netstat syntax to focus on particular ports. Here is for Transmission and then for the Plex port:
mediamac:~ ben$ netstat -an |grep .10002 tcp4 0 0 192.168.0.2.10002 94.245.58.211.60686 SYN_RCVD tcp4 0 0 192.168.0.2.10002 178.82.144.61.60173 ESTABLISHED tcp6 0 0 *.10002 *.* LISTEN tcp4 0 0 *.10002 *.* LISTEN tcp4 0 0 192.168.0.2.10002 84.104.165.198.58890 TIME_WAIT udp4 0 0 *.10002 mediamac:~ ben$ netstat -an | grep .32400 tcp4 0 0 127.0.0.1.32400 127.0.0.1.51153 ESTABLISHED tcp4 0 0 127.0.0.1.51153 127.0.0.1.32400 ESTABLISHED tcp4 0 0 127.0.0.1.32400 127.0.0.1.51096 ESTABLISHED tcp4 0 0 127.0.0.1.51096 127.0.0.1.32400 ESTABLISHED tcp4 0 0 127.0.0.1.32400 127.0.0.1.49318 ESTABLISHED tcp4 0 0 127.0.0.1.49318 127.0.0.1.32400 ESTABLISHED tcp46 0 0 *.32400 *.* LISTENSo it seems same for Plex. And agreed nothing to do with pfSense....
-
Looks like in your 10002 your in syn_rcvd state from that 1 IP.. And do have an established connection with 178.82.x.x and in a time wait for another..
So your going to have to work out whatever is wrong if your client or ACL... Possible your running a block list in transmission, you know keep the "spies" out... isn't there large black lists of the bad guys to try and keep out of the swarm?
As to plex.. Maybe something common with firewall running that you think you turned off, or 3rd party security software you running on the box?
-
SOLVED.....
On the offchance it helps someone else, in the end I fixed it by...
- Deleting the OSX firewall preference .plist located at /Library/Preferences/com.apple.alf.plist
- Cleaning system & user caches
- Restarting
- Enabling the firewall
- Doing this all a couple of times...
What's strange is this whole time the firewall was OFF in System Preferences, so I've no idea why this should affect things. In addition, I had previously during testing deleted this file but seems you have to do it and restart, maybe more than once, to really clear it.
Thanks for all the patient help in narrowing down the problem, really much appreciated....
