Port forward to UDP 10000 is NOT working

johnpoz

here

Took all of 30 seconds to test

https://www.ipvoid.com/udp-port-scan/

do that for your interface on pfsense that is your wan. Mine was igb1, yours is I don't know.

raviktiwari

Ok here you go...

Just realised it is showing wrong ip... the server ip should be: 192.168.14.22, not sure why it is showing: 192.168.14.18

I have already posted the ss for ipvoid couple of time.

Hope this helps.

Many Thanks,
Rav

johnpoz

And that has zero to do with port 10000 traffic hitting your WAN!

tcpdump -i igb1 udp port 10000

where igb1 would be replaced with whatever your wan interface is.

edit: You have NOT posted anything showing port 10k hitting your wan.. Not here..

edit2: On pfsense do a ifconfig, which interfaces show your wan IP??

Find yours - then do the tcpdump I showed filtering on 10k, and then use that website I gave to generate traffic to 10k to your wan IP... If you do not see traffic, pfsense can not forward it.

raviktiwari

Thanks @johnpoz

So I ran ifconfig in pfsense -> Command Prompt and got the wan port to be bge0 - this is correct because I can see it on my laptop, which is working as a pfsense for me.

I then entered the command tcpdump -i bge0 udp port 10000 on my ssh screen and it gives me error: No such device exists. PFA the screenshot.

I then tried the same command within pfsense-> Command Prompt and here also there is no good news - nothing happens. PFA the screenshot.

Just saw that the command finally failed with an error:
504 Gateway Time-out
nginx

Now pls don't say there is no traffic flowing and so pfsense cannot do anything. Because it is working on another test server. There is 2 test server behind this test pfsense (192.168.14.20 and 192.168.14.22) both were not working and then with help of you guys a few days ago it started working on its own and then after formating both the test server - both test servers were not listening to UDP10K and then after pfsense restore, server with IP ending in 22 is working fine but 22 is not able to listen. So we know WAN port (which is same for 20 and 22) is working fine...

And pls don't say that 2 servers cannot listen to same UDP port thru same pfsense within the same network - it was working until the day before yesterday.

As lastly when you say "you have NOT posted anything showing port 10k hitting your wan.. Not here.." Do you mean this screenshot?

If yes, I have already posted it 2 times and this is 3rd time... :-) and if this is not what you are looking for, my apologies, can you psl share the link once more, so that I can take the screenshot for you.

I hope it makes sense now... if not, I am happy to speak to you on phone, WhatsApp, Teamviewer or my screen sharing tool - that I am trying to test for further development. I am in the UK, so it's quite late now, but I won't be able to sleep until this is fixed - so timing is not an issue for me. Happy to work with you as per your convenience.

Many Thanks,
Rav

raviktiwari

Hi @johnpoz, just did some testing and now 192.168.14.22 which was able to listen on UDP port 10K has stopped listening - AGAIN.

This flakiness is a real frustration for me. Can it because I am using an old laptop as pfsense router? I don't think that should be an issue because every other port forwarding and rules are working fine (yes all of them are TCP), only this is UDP port which is troubling me from more than a month now.

Many Thanks,
Rav

raviktiwari

Not sure, but I am thinking aloud, can it be because of the sequence in which port has been opened and rules created? And for my simplicity and clarity, I have used separator in NAT and RULE screen so that all the rules related to one server are together and then the next step of rules are put together.

So right now it is working for IP ending with 20 but not 22 and last time it was working for 22 but not 20... And when it worked for both, I was not using any separator...

I was wondering if there a way that I can write parallel rules or should I remove separator - not sure if that will make any difference.

So I just removed the separator and now both IP (20 and 22) is NOT listening on UDP 10K port. :-(
So put the separator back but still no change - still both IPs are NOT listening... I also restarted servers and pfsense - will port forwarding stop working at every restart? If yes, then that is a real serious issue for me... This is a test server and is supposed to be switched off at the end of the day, so does that mean that every morning I will have to deal with this struggle?

Thx: Rav

raviktiwari

I think I have cracked it - from pattern point of view.

As both servers (...20 and ...22) are connected to the same pfsense and on the same subnet, whichever system gets UDP port 10K, it sticks to it and then next system struggles to listen or pfsense struggles to divert traffic to that system - it's like that channel is busy (stuck to the first server).

Here is the scenario, how it is behaving:
Server 20 was listening and 22 was NOT listening...
I deleted the NAT for 20 and checked 22, it started listening...
Then I created new NAT for 20 by cloning 22 but it still did not listen.
So this time I deleted the NAT for 22 and checked 20, it started listening... And again when I created/cloned new NAT for 22 it is not listening.

The funny and irritating thing is until the day before yesterday both servers were listening to UDP port 10K at the same time - so why it cannot do the same now.

Any idea/suggestion?

Many Thanks, Rav

Gertjan

pfSEnse can only port forward to one LAN type device.
If you set up a NAT rule on pfSense , the port - or range of ports - from an ANY type of WAN device, are retransmitted to a LAN device, identified by a LAN IP.
If you have a second rule using the same ports, that want to transmit to a second device, that won't work - never get used : the first rule will match / win.

Its like an incoming phone call on an old PABX : the call can be passed to 'administration' or 'sales' : never both.

@raviktiwari said in Port forward to UDP 10000 is NOT working:

So I ran ifconfig in pfsense -> Command Prompt

That's a gadget.
Serious issues needs the console (activate the SSH so you can do so from anywhere) access. Works well for the last 50 years or so.
Easy to copy paste, etc. I even tend to say that everything that can be done by the SSH access, should be done using the SSH access. True, for pfSense, it's quiet are. Basic setup is done using the GUI.

[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ifconfig
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 6c:b3:11:50:c6:c6
        hwaddr 6c:b3:11:50:c6:c6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 6c:b3:11:50:c6:c7
        hwaddr 6c:b3:11:50:c6:c7
        inet6 fe80::6eb3:11ff:fe50:c6c7%em1 prefixlen 64 scopeid 0x2
        inet6 2001:470:1f13:5c0:2::1 prefixlen 64
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1b:21:32:da:42
        hwaddr 00:1b:21:32:da:42
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1480
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1b:21:32:da:43
        hwaddr 00:1b:21:32:da:43
        inet6 fe80::21b:21ff:fe32:da43%em3 prefixlen 64 scopeid 0x4
        inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2008<VLAN_MTU,WOL_MAGIC>
        ether 00:12:3f:b3:58:75
        hwaddr 00:12:3f:b3:58:75
        inet6 fe80::212:3fff:feb3:5875%fxp0 prefixlen 64 scopeid 0x5
        inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
enc0: flags=0<> metric 0 mtu 1536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: enc
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::6eb3:11ff:fe50:c6c6%ovpns1 prefixlen 64 scopeid 0xb
        inet6 2001:470:ccea:3::1 prefixlen 64
        inet 192.168.3.1 --> 192.168.3.2 netmask 0xffffff00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun openvpn
        Opened by PID 88169
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1472
        options=80000<LINKSTATE>
        tunnel inet 192.168.10.2 --> 216.66.84.42
        inet6 2001:470:1f12:5c0::2 --> 2001:470:1f12:5c0::1 prefixlen 128
        inet6 fe80::6eb3:11ff:fe50:c6c6%gif0 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: gif

With this info :

i know now that the device (driver) name "em3" is my WAN interface.

Btw : it is possible tu put, for example, two identical web servers behind a router, both on there own LAN type network segment (or the same LAN) - both using port 443 - and use both web servers, seen from the 'outside' as one.
That's called "load balancing". New, incoming connections are randomly, or on round robin, dispatches over 1 of the 2 web servers.
I never used such a setup. neither if it works with UDP.

Why does your setup works ones in while ?
It's a question of timing.
When he first "NAT to 10000 UDP" rule is introduced in the firewall, connections are accepted and send to the LAN IP indicated in this rule. As soon as traffic comes in from from that WAN based device, and NATted to that first "10000" LAN server, firewall states are created. A state contains the source and destination IP, (source ?) and dominations port, etc.
Now, when the second NAT rule is added (manually in the GUI), it might be placed before the first NAT rule (before not visually in the GUI, I'm talking the actual rules in the firewall, as NAT rules have no 'order').
Another WAN based device comes in using destination port "10000" and hits this second rule, traffic is send to the second LAN based port 10000 server.
The initial WAN based port 10000 is still able to send traffic to the first LAN based 10000 server .... as traffic is handled by the internal states .... the new second NAT rule isn't tested anymore, neither the first.

As soon as the states time out, a question of seconds or minutes, all traffic from any WAN based device (the entire Internet' will hit the first available NAT rule, the second one never get used an ymore.

Btw : if you change the ANY part of the WAN source IP to an identified IP, then you can select which NAT rule you want to apply for what WAN device. WAN IP1 will be natted to LAN server 1, WAN IP2 to LAN server2. WAN IP1 and IP2 can also be entire networks sections.

raviktiwari

Thanks @Gertjan

I tried testings it few times and it seems like what you are saying might be true... as soon as the 1st system on the network is able to establish the connection, 2nd one can no longer connect to the same port (on the same network).

The only workaround is to shut down the 1st server so that all the connections are released and then restart the 2nd server so that it can establish the new connection. And then if you bring up the 1st server, that will not be able to connect to the same port.

Theoretically, it makes sense, but I am very disappointed with this limitation and my brain is challenging me...

Imagine in a corporate environment out of 50 servers, if 5 of them need to have access to the same port on same network (maybe for same or even different applications), then how is that going to work?

Many Thanks,
Rav

johnpoz

@raviktiwari said in Port forward to UDP 10000 is NOT working:

Imagine in a corporate environment out of 50 servers, if 5 of them need to have access to the same port on same network (maybe for same or even different applications), then how is that going to work?

You understand in the corp world they would have more than 1 public IP right??

You can not listen on the same IP on the same port - this is not a problem be it home network or corp network or Google or MS or even Amazon - this is how tcp works... You can not have 2 applications listening on the same port at the same time on the same IP..

You can have more IPs.. Now they can all listen on port X..

if you want 2 servers on rfc1918 space behind your network to both use port 10k to the public then you need 2 public IPs.. Period. Use a different port on the public side for 1 of them.

raviktiwari

Thanks @johnpoz yes it does make sense and that is why I am also using 3 IPs now... 1 for Prod, 2nd for Test and 3rd for staging environment.

I am thinking and smiling... imagine a new engineer joins a corp and he has to do some installation/configuration and ultimately open a port - and if that port is already open on the same IP then he can tear all his hairs he is not going to get the connection. :-)

We can argue over documentation but we all know how much documentation is done in real-world and is accessible at the right time and even if we get it, we wish it was at least meaningful. :-)

@johnpoz I am going to pick your brain on another issue, which is bothering me for last 4-5 days... but there is already an open thread and you are active there, so speak to you soon.

Many thanks once again for all your help, support, patience and guidance.

Highly appreciate it mate.
Rav

johnpoz

@raviktiwari said in Port forward to UDP 10000 is NOT working:

and if that port is already open on the same IP then he can tear all his hairs he is not going to get the connection. :-)

You clearly don't get how an enterprise works at all ;) That would never in a million years happen.. If engineer wanted port X open to the public for his service... There would be security audits and reviews, there would be change control and reviews and cabs.. There would be multiple teams involved.. It sure wouldn't be the guy running the server opening the port on the firewall to the public ;) And then wondering why it doesn't work.. hehehehe

Been trying to get a simple internal routing change done for a customer for weeks now.. It's mind boggling the meetings and paperwork for a simple /31 route..

And this doesn't need to involve the security team at all.. But it does require 3 different teams to get the route to happen.. Not counting the quality control team checking all the paperwork ;)

Now in some small ma and pop shop you might have 1 guy playing cowboy with firewall rules and server open to the public, etc. etc.. But that in no way shape or form can happen in an actual enterprise ;)

raviktiwari

Looks like you are in the US... then fair enough... :-)

You have no idea how it works in UK mate... here we are very good in talking and documenting the process (not the technology) and we do delay CAB approval (but purely for personal ego purposes) and we do TALK about lessons learnt but that's it.

I wrote a long story but then I deleted it - we can always talk about it when you are drunk again with your friends - just let me know so that I can also get started along with you guys... :-)

Cheers,
Rav

johnpoz

Not sure what company you work for - We have offices in the UK mate... Same enterprise wide polices ;) Doesn't matter in your in Uganda or Berlin, New York or whereever.. The policies and procedures are global..

If someone opened a port on public firewall, shoot for that matter internal without approval.. They would be fired period..

raviktiwari