Test Request: UPnP Fix for Multiple Consoles playing the same game / static port outbound NAT

Marc05

@vmac That's exactly what you're supposed to get. The only way to get open NAT would be for the consoles to have a public IP themselves.

vMAC

@marc05 If a proper UPnP was working then they would both get Open NAT. When I just had one PS5, it received an Open NAT with no issues. I'm not following why moderate is correct or would be considered "working."

Marc05

@vmac said in Test Request: UPnP Fix for Multiple Consoles playing the same game / static port outbound NAT:

@marc05 If a proper UPnP was working then they would both get Open NAT. When I just had one PS5, it received an Open NAT with no issues. I'm not following why moderate is correct or would be considered "working."

See here: https://portforward.com/nat-types/

winger46146

See here: https://portforward.com/nat-types/

From your link and UPnP doesn't work.

NAT Type Open
NAT Type Open is the goal when setting up an Xbox. In order to get NAT Type Open on your Xbox you need to do one of the following:

Forward the Xbox Live ports in your router to your Xbox, or
Setup a DMZ in your router pointing at your Xbox, or
Have a fully UPnP compliant network. Usually, this means having a router that supports UPnP and is enabled.
Any of the above methods should net you a NAT Type Open on your Xbox. Our site recommends a port forward as the best option to get NAT Type Open. DMZ is the easiest choice, but it's a bit overkill and best reserved as a testing tool for network problems. UPnP is a really dangerous protocol and it allows any piece of malware on your network to forward a port to any device inside your home without you knowing it. We keep UPnP turned off on our routers for increased security.

NAT Type Open means the following:

Your Xbox may or may not be behind a router.
If your Xbox is behind a router, then your router is aware of your Xbox and is forwarding incoming packets on predefined ports to your Xbox, usually 3074.
Your Xbox is able to receive incoming packets from the internet including connection requests from other players.
You are able to be the host of multiplayer lobbies.
You should have no limits on chat, video.
If you have NAT Type Open on your Xbox then you are done. There is nothing more for you to do.

iculookn

PFSENSE Team, What if we raise some funding so you can buy a few XBOX's and PS5's so you can test and fix all the gaming challenges including this upnp issue and make PFSENSE the perfect "Gaming Router".

Marc05

@winger46146 You missed the context that this was for PS5, not Xbox.

vMAC

@marc05 said in Test Request: UPnP Fix for Multiple Consoles playing the same game / static port outbound NAT:

https://portforward.com/nat-types/

That again confirms that this is not working. My connections are being limited in a moderate NAT. As I stated before when using UPnP with just one system connected I get an Open NAT, not Type 1 NAT, but Type 2 Open NAT.

When using the same config that you list above, I get a Type 3 Moderate NAT. While that is marginally better than a Type 3 Strict NAT, neither are ideal or show a "solution" to the issue that has brought many to this thread.

Marc05

@vmac I don't know what "Type 2 Open NAT" is. Every platform seems to define its own terms so it's difficult to say what should or shouldn't be. I'm not saying there isn't an issue as I've run into it before, but it's difficult to diagnose.

Having multiple consoles with static port outbound rules will inevitably lead to port conflicts. Depending on how those conflicts are treated (unclear), this can lead to different NAT Type results.

For example, this was posted previously (which is ultimately what you want):

nat quick on em0 inet proto udp from 192.168.1.31 port = 3074 to any keep state label "DemonwarePortMapping" rtable 0 -> 24.255.xxx.xxx port 3074
nat quick on em0 inet proto udp from 192.168.1.30 port = 3074 to any keep state label "DemonwarePortMapping" rtable 0 -> 24.255.xxx.xxx port 3108
[...]
rdr pass quick on em0 inet proto udp from any to any port = 3074 keep state label "DemonwarePortMapping" rtable 0 -> 192.168.1.31 port 3074
rdr pass quick on em0 inet proto udp from any to any port = 3108 keep state label "DemonwarePortMapping" rtable 0 -> 192.168.1.30 port 3074

Does this mean that static ports were used for both 192.168.1.31 and 192.168.1.30, and the consoles automatically try opening a different port (in this case 3108) through UPnP? If so, will every game always do this, or is it up to each game to decide on the implementation?

Jon8RFC

@jimp I re-read everything on redmine and see that you mentioned not having two consoles for testing...
The issue is reproducible on a Windows 10 PC with Call of Duty Warzone, with a PS5 console playing Warzone on the same network.

I assume it's reproducible with two PCs as well. It's a free game, so you can download this on multiple PCs or consoles for testing since you're familiar with what's going on with the networking side of things, unlike me:
https://www.callofduty.com/content/atvi/callofduty/warzone/web/en/download.html

Thanks for everything you and everyone are doing to get this resolved!

aniel

i just dont understand why or how they cant fix this, given that many household usually have 1 or more gamer in the same house. this issue have been reported for many years now.

DonZalmrol

@aniel While I like this also fixed for myself (and others), you do understand that PFSense is a business oriented firewall and not for "regular" households. So might take longer (or might never be implemented) to be fixed, all depends on the Devs and the demand for it.

encrypt1d

Thanks to everyone in this thread. With the help of these posts I was able to get CoD WWII at least working with 2 PCs, using the non-uPNP method @hansaya posted (I have 2 open NATs, and they can play together on a custom game, with other friends joining from the internet). Sadly that technique doesn't work for Black OPS III. I'm on Pfsense 2.5.2, miniupnpd 2.2.1.
Using just uPNP with no manual outbound nats/port forwards works on only one PC (Open NAT, but the second cannot establish any connection at all).
So I am available to help test fixes as they become available as well. Looks like the thread ended with the last dev snapshot still not working. Is it still being looked at by the devs?

encrypt1d

I spent more time testing this weekend, and narrowed things down a bit for the second PC in a call of duty WWII setup, where UDP 3074 seems to be the port of contention. (Reminder, first PC connects fine, gets Open NAT inside the game, second PC cannot connect at all).

I started by pcapping all the upnp control traffic. It starts off by asking for 3074, which is already taken by the first PC. The server responds correctly with an error code 718, and then the DemonwarePortMapping service on the PC asks for a new random port, usually in the 31xx range. The upnp server responds successfully, adds the mapping and I can see it in all 3 places - the pfsense gui, the output of the pfanchordrill and on the windows PC in the settings of the DemonwarPortMapping entries on the upnp router (in the network section of file explorer). This cycle repeats forever at this point, and the client will not connect. Each time it asks for another port, and obtains it successfully. The list of ports in the uPNP GUI fills up indefinitely.

So then I opened 2 ssh sessions to the firewall, did a pcap on the LAN and WAN at the same time, then merged them in wireshark. This allows me to see the packet as it left the host, and the packet as it leaves the wan interface, with timecodes to correlate everything.

Interestingly enough, despite the correct NAT/RDR pair in place, absolutely no packets leave on the wan. They come in from the client with src 3074/dest 3074, but never go out on the internet with the NATed src port. In fact nothing comes out. I have no firewall blocks at all. That explains why WWII just keeps asking for new ones as each attempt times out.

So what would cause the firewall not to NAT and transmit outbound on the WAN, despite the correct NAT/RDR paring in place?

Is there a debug log to check that might show why it isn't NATing and forwarding? There's nothing in routing.log, where miniupnpd writes its entries.

Hoping this sheds some light on the true nature of the issue.

encrypt1d

Did a bit more troubleshooting, this time using pflog which is a bit easier to use in real time. X.X.X.X is my public ip masked out. 192.168.7.2 is the 1st PC that gets Open NAT. 192.168.7.3 is the second PC that will not connect at all. igb0 is my WAN interface.

Here are the nat/rdr pairs in place according to the pfanchordrill:

nat log quick on igb0 inet proto udp from 192.168.7.2 port = 3074 to any keep state label "DemonwarePortMapping" rtable 0 -> X.X.X.X port 3074
nat log quick on igb0 inet proto udp from 192.168.7.3 port = 3074 to any keep state label "DemonwarePortMapping" rtable 0 -> X.X.X.X port 3182

rdr pass log quick on igb0 inet proto udp from any to any port = 3074 keep state label "DemonwarePortMapping" rtable 0 -> 192.168.7.2 port 3074
rdr pass log quick on igb0 inet proto udp from any to any port = 3182 keep state label "DemonwarePortMapping" rtable 0 -> 192.168.7.3 port 3074

Looks perfect so far.

Here's what pflog shows happening with the UDP packets for the first PC, which gets Open NAT in game:

18:20:06.661452 rule 732/0(match): pass in on lan: 192.168.7.2.3074 > 185.34.107.128.3074: UDP, length 3
18:20:06.661471 rule 116/0(match): pass out on igb0: X.X.X.X.3074 > 185.34.107.128.3074: UDP, length 3
18:20:06.885506 rule 0/0(match): rdr in on igb0: 185.34.107.129.3075 > 192.168.7.2.3074: UDP, length 15

Again, the above is exactly what it should be.

Now PC 2 tries to join the game, and here's the issue:

18:31:45.791654 rule 732/0(match): pass in on lan: 192.168.7.3.3074 > 185.34.107.128.3074: UDP, length 3
18:31:45.791668 rule 115/0(match): pass out on igb0: 192.168.7.3.3074 > 185.34.107.128.3074: UDP, length 3

Note that the NAT failed to assign the Firewall interface IP to the Source IP. It straight up routed the packet as-is (with an RFC 1918 private IP). It also did not replace the source port with 3182 as it should have. (EDIT) ~~Of course the upstream router will get that packet and toss it.~~ The pfsense will never actually route this packet as it is in the wrong subnet for that interface. That would be why in my previous tests, I didn't see them in the packet capture. (END EDIT) I am puzzled as to how that could happen. The second PC is falling right through the NAT rules, despite a manual catch all outbound NAT I have defined. That rule obviously works or I would have no functional internet traffic at all.

I have to say, this looks like a PF issue to me, not miniupnpd.

If there is anyone out there still listening, it would be great if you could try this and see what you get.

To use pflog, you can type this on an ssh shell:

service pflog onestart
tcpdump -eni pflog0 | grep 3074

Likewise, to stop pflog use:

service pflog onestop

iculookn

@encrypt1d Great work. We are listening, hopefully the PFSENSE team is listening and can review your work.

jimp

That in itself isn't a pf issue really. It's what happens when there is a port conflict.

There can only be one NAT state for <wan addr>:<source port> -> <destination addr>:<destination port>. If there is already a state for that and something else comes along and tries to start another connection that would end up using the same external source port and destination. It can't create a state for the second one because it already exists, but the firewall rules have passed the traffic, so it leaves without NAT.

What that does tell me is that in your case they are both matching some other outbound NAT rule that is set to use static port, not the rules you printed above, but it does give me an idea.

Try applying this change, then either reboot or do a filter reload then reset states:

diff --git a/src/etc/inc/filter.inc b/src/etc/inc/filter.inc
index d36d6df2e2..5a7c21bc2a 100644
--- a/src/etc/inc/filter.inc
+++ b/src/etc/inc/filter.inc
@@ -2091,6 +2091,8 @@ function filter_nat_rules_generate() {
 
        $natrules = "no nat proto carp\n";
        $natrules .= "no rdr proto carp\n";
+       $natrules .= "binat-anchor \"miniupnpd\"\n";
+       $natrules .= "nat-anchor \"miniupnpd\"\n";
        $natrules .= "nat-anchor \"natearly/*\"\n";
 
        $natrules .= "nat-anchor \"natrules/*\"\n\n";

encrypt1d

@jimp
Very promising! I just tested, and I think that change will work for everyone, except me. I can see the packets getting translated now. So that might be the fix for everyone else, but sadly I cannot completely verify.

I hadn't brought this up yet to avoid mixing too many issues. I am unfortunately behind a ISP that gives me a 10.x.x.x address on my firewall. I have read all about the changes made a while back in miniupnpd that broke double NAT. My ISP gives me a dedicated Public IP, and forwards everything fully with no translations other than the public IP to my 10.x WAN IP. (paid extra for that).

What I see now is it using the public IP instead of my WAN interface IP. I have the ext_ip=x.x.x.x line in my miniupnpnd config file, without which it will never even add nat/rdr pairs. I tried using STUN, but then miniupnp says it is impossible to forward under that config via a debug message.

So I think I have had 2 issues all along. The first being those missing lines from the filter.inc file, and now I've moved to the 10.x address on my WAN interface.

Thought I was almost there. Is there a way to get miniupnp to use 10.x addresses for the NAT rules? From what I've read, it seems like no, so I might just be SOL. If there is an easy answer to that, I'd appreciate it. If I need to open another discussion over at miniupnp I'll follow that.

Their default config has an option to revert the IGD behaviour:
https://github.com/miniupnp/miniupnp/blob/master/miniupnpd/miniupnpd.conf

force_igd_desc_v1=yes

Alas my version says this option is unrecognized. Not sure if it would even help.

I encourage everyone else behind a 'normal' ISP to give @jimp's change a try.

vi /etc/inc/filter.inc
*Make the changes
/etc/rc.filter_configure

Then from the GUI, reset the states under the diagnostics menu.

jimp

You can install the System Patches package and then create an entry for the diff I posted to apply the fix without manually editing files.

That said, being behind double NAT will be a problem for anyone in the same situation. Do they maintain static port on your outbound NAT even? Seems like most CGNAT type places won't do that. Some time ago, miniupnpd decided that it wouldn't run with private addresses. I'm still not sure why they won't give us a way to override that, as it also makes testing it internally more difficult.

encrypt1d

@jimp

Yes, my ISP doesn't mess with any ports, just the IPs. With the workaround (manual rules) that @hansaya posted, I can get 2 open NATS.

I will try and follow up with the miniupnp folks and post a link here for those that are interested when I have it. Thanks for your help!

encrypt1d

I am trying my luck on a pull request that a user was working on that was aimed at fixing this double NAT issue, I think.

https://github.com/miniupnp/miniupnp/pull/565