Port forward to UDP 10000 is NOT working
-
If your client is behind a different pfsense than where your server is then no you wouldn't use nat reflection.
Again where have you shown that this traffic actually hits pfsense wan?
Last I recall you were not seeing this traffic when you did a sniff.. I gave you a site that will send UDP traffic.. Use it.. sniff on your wan.. Do you see it?? If not then no its never ever ever ever going to work!! Period!
-
Sorry if I have not been clear enough but the setup is pretty simple.
I have 2 network - Test and Prod... both of them have different IP, both came from same ISP broadband modem but one goes in Prod pfSense and another goes in Test pfsense and then from there are managed as a different network. I have attached a diagram - hope it makes it easy to understand.
Now I want port 10K on UDP to be open on 2 Test Servers and 1 Prod Servers. To ensure it is working, I have been told to use those commands that I shared above. So I make Test server listen and then I send the message using echo to Prod server, which does not reach, but the other way around is working and so my application is working on Prod but not on Test. I need it to work on Test so that I can do further development and testing.
So this is my 1st issue - port 10K on UDP not working in Test Environment.
Yes, I used your tool (ipvoid) and it shows WAN port is listening on UDP 10K - I have attached the screenshot in my previous message. But yes I am not able to see any traffic on within pfsense because that box is not displaying anything - especially on production pfsense (because it is connected on Prod environment). However, when I do the same thing on the Test environment using test pfsense I can see the traffic flowing when I sniff it.
This is my 2nd issue - why I am unable to see sniffing details in the production environment.
And the 3rd issue is:
Why few of my websites hosted in test environment are NOT accessible from within the Test environment but it can be accessed without any issue on Prod as well as anywhere else in the world.
Hope this clarifies the situation and thanks once again to be there to support, I sincerely appreciate it.
Many Thanks,
Rav -
Step 1... Sniff on wan of pfsense that holds the IP your trying to send data too... Go to say the site I linked to and send UDP traffic. If you do not not see it in the sniff.. Pfsense can not forward anything.. Period!!
Untill you show us this sniff, there is nothing anyone can help you with..
This is my 2nd issue - why I am unable to see sniffing details in the production environment.
Because the data isn't get there is the most logical answer to that..
-
Hey @johnpoz I have no doubt whatsoever that you know more than me and so I dont want to sound argumentative or disrespectful, but pls try to understand the situation.
Going by your requirements:
1.> ipvoid says packet is flowing without any issue - PFA screenshot.
I hope this answers your 1st question. While my 1st issue is still at hand why my application is not able to use UDP10K - why the port is not working?
Coming to 2nd question...
Here the issue completely different - we know all data is flowing fine, I can also get messages using those commands and I can also access my application. So technically there is no issue at all and we should not be discussing about it... but the problem here is, when I sniff and use pfsense inbuilt tool "packet capture", I cannot see anything in the box below.And this is not because there is no traffic... we know everything is working fine...We also know this is not a laptop or browser issue because when I connect the same laptop and use the same browser in test environment, the same box shows all the details - for both Test and Prod public IP.
This means ( atleast in my brain - that there is some configuration that I have done within pfsense for Prod environment that is not letting that box fill up with relevant information/traffic).
And this is my 2nd issue.
For the time being, let's leave the 2nd issue we can discuss it once we have sorted these 2 open issues.
Many Thanks,
Rav -
This post is deleted! -
You need to show a sniff! where the traffic actually gets there..
If you can not see it, either can pfsense.. Do a simple tcpdump on pfsense for port udp 1000.. if your having issues with the gui.
-
Hmm.... So where and how do I do that? Which tool do I use?
-
Already went over this.. Dude I really want to help you... But this is getting more frustrating for me, than I really want to deal with... Troubleshooting port forwarding is 2 minute thing.
-
I agree @johnpoz it is more frustrating for me... because it is working in 2 out of 3 place... so the issue is why it is not working on 3rd place
And then the 2nd issue is: why I am not able to see the packet capture box pop up with traffic details.
Apologies if I am not getting you, but you asked me to use ipvpoid and pfsense packet capture tool and I have used both and shared screenshot for both. Not sure what else you need for me... I want to work with you, take your instruction and get it resolved. But unless I am clear with what yoiu want me to do, how can we close the feedback loop?
Many Thanks,
Rav -
here
Took all of 30 seconds to test
https://www.ipvoid.com/udp-port-scan/
do that for your interface on pfsense that is your wan. Mine was igb1, yours is I don't know.
-
Ok here you go...
Just realised it is showing wrong ip... the server ip should be: 192.168.14.22, not sure why it is showing: 192.168.14.18
I have already posted the ss for ipvoid couple of time.
Hope this helps.
Many Thanks,
Rav -
And that has zero to do with port 10000 traffic hitting your WAN!
tcpdump -i igb1 udp port 10000
where igb1 would be replaced with whatever your wan interface is.
edit: You have NOT posted anything showing port 10k hitting your wan.. Not here..
edit2: On pfsense do a ifconfig, which interfaces show your wan IP??
Find yours - then do the tcpdump I showed filtering on 10k, and then use that website I gave to generate traffic to 10k to your wan IP... If you do not see traffic, pfsense can not forward it.
-
Thanks @johnpoz
So I ran ifconfig in pfsense -> Command Prompt and got the wan port to be bge0 - this is correct because I can see it on my laptop, which is working as a pfsense for me.
I then entered the command tcpdump -i bge0 udp port 10000 on my ssh screen and it gives me error: No such device exists. PFA the screenshot.
I then tried the same command within pfsense-> Command Prompt and here also there is no good news - nothing happens. PFA the screenshot.
Just saw that the command finally failed with an error:
504 Gateway Time-out
nginxNow pls don't say there is no traffic flowing and so pfsense cannot do anything. Because it is working on another test server. There is 2 test server behind this test pfsense (192.168.14.20 and 192.168.14.22) both were not working and then with help of you guys a few days ago it started working on its own and then after formating both the test server - both test servers were not listening to UDP10K and then after pfsense restore, server with IP ending in 22 is working fine but 22 is not able to listen. So we know WAN port (which is same for 20 and 22) is working fine...
And pls don't say that 2 servers cannot listen to same UDP port thru same pfsense within the same network - it was working until the day before yesterday.
As lastly when you say "you have NOT posted anything showing port 10k hitting your wan.. Not here.." Do you mean this screenshot?
If yes, I have already posted it 2 times and this is 3rd time... :-) and if this is not what you are looking for, my apologies, can you psl share the link once more, so that I can take the screenshot for you.
I hope it makes sense now... if not, I am happy to speak to you on phone, WhatsApp, Teamviewer or my screen sharing tool - that I am trying to test for further development. I am in the UK, so it's quite late now, but I won't be able to sleep until this is fixed - so timing is not an issue for me. Happy to work with you as per your convenience.
Many Thanks,
Rav -
Hi @johnpoz, just did some testing and now 192.168.14.22 which was able to listen on UDP port 10K has stopped listening - AGAIN.
This flakiness is a real frustration for me. Can it because I am using an old laptop as pfsense router? I don't think that should be an issue because every other port forwarding and rules are working fine (yes all of them are TCP), only this is UDP port which is troubling me from more than a month now.
Many Thanks,
Rav -
Not sure, but I am thinking aloud, can it be because of the sequence in which port has been opened and rules created? And for my simplicity and clarity, I have used separator in NAT and RULE screen so that all the rules related to one server are together and then the next step of rules are put together.
So right now it is working for IP ending with 20 but not 22 and last time it was working for 22 but not 20... And when it worked for both, I was not using any separator...
I was wondering if there a way that I can write parallel rules or should I remove separator - not sure if that will make any difference.
So I just removed the separator and now both IP (20 and 22) is NOT listening on UDP 10K port. :-(
So put the separator back but still no change - still both IPs are NOT listening... I also restarted servers and pfsense - will port forwarding stop working at every restart? If yes, then that is a real serious issue for me... This is a test server and is supposed to be switched off at the end of the day, so does that mean that every morning I will have to deal with this struggle?Thx: Rav
-
I think I have cracked it - from pattern point of view.
As both servers (...20 and ...22) are connected to the same pfsense and on the same subnet, whichever system gets UDP port 10K, it sticks to it and then next system struggles to listen or pfsense struggles to divert traffic to that system - it's like that channel is busy (stuck to the first server).
Here is the scenario, how it is behaving:
Server 20 was listening and 22 was NOT listening...
I deleted the NAT for 20 and checked 22, it started listening...
Then I created new NAT for 20 by cloning 22 but it still did not listen.
So this time I deleted the NAT for 22 and checked 20, it started listening... And again when I created/cloned new NAT for 22 it is not listening.The funny and irritating thing is until the day before yesterday both servers were listening to UDP port 10K at the same time - so why it cannot do the same now.
Any idea/suggestion?
Many Thanks, Rav
-
pfSEnse can only port forward to one LAN type device.
If you set up a NAT rule on pfSense , the port - or range of ports - from an ANY type of WAN device, are retransmitted to a LAN device, identified by a LAN IP.
If you have a second rule using the same ports, that want to transmit to a second device, that won't work - never get used : the first rule will match / win.Its like an incoming phone call on an old PABX : the call can be passed to 'administration' or 'sales' : never both.
@raviktiwari said in Port forward to UDP 10000 is NOT working:
So I ran ifconfig in pfsense -> Command Prompt
That's a gadget.
Serious issues needs the console (activate the SSH so you can do so from anywhere) access. Works well for the last 50 years or so.
Easy to copy paste, etc. I even tend to say that everything that can be done by the SSH access, should be done using the SSH access. True, for pfSense, it's quiet are. Basic setup is done using the GUI.[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ifconfig em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 6c:b3:11:50:c6:c6 hwaddr 6c:b3:11:50:c6:c6 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 6c:b3:11:50:c6:c7 hwaddr 6c:b3:11:50:c6:c7 inet6 fe80::6eb3:11ff:fe50:c6c7%em1 prefixlen 64 scopeid 0x2 inet6 2001:470:1f13:5c0:2::1 prefixlen 64 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:1b:21:32:da:42 hwaddr 00:1b:21:32:da:42 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1480 options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:1b:21:32:da:43 hwaddr 00:1b:21:32:da:43 inet6 fe80::21b:21ff:fe32:da43%em3 prefixlen 64 scopeid 0x4 inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=2008<VLAN_MTU,WOL_MAGIC> ether 00:12:3f:b3:58:75 hwaddr 00:12:3f:b3:58:75 inet6 fe80::212:3fff:feb3:5875%fxp0 prefixlen 64 scopeid 0x5 inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo enc0: flags=0<> metric 0 mtu 1536 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: enc pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync pflog0: flags=100<PROMISC> metric 0 mtu 33160 groups: pflog ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet6 fe80::6eb3:11ff:fe50:c6c6%ovpns1 prefixlen 64 scopeid 0xb inet6 2001:470:ccea:3::1 prefixlen 64 inet 192.168.3.1 --> 192.168.3.2 netmask 0xffffff00 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: tun openvpn Opened by PID 88169 gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1472 options=80000<LINKSTATE> tunnel inet 192.168.10.2 --> 216.66.84.42 inet6 2001:470:1f12:5c0::2 --> 2001:470:1f12:5c0::1 prefixlen 128 inet6 fe80::6eb3:11ff:fe50:c6c6%gif0 prefixlen 64 scopeid 0xa nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: gifWith this info :
i know now that the device (driver) name "em3" is my WAN interface.
Btw : it is possible tu put, for example, two identical web servers behind a router, both on there own LAN type network segment (or the same LAN) - both using port 443 - and use both web servers, seen from the 'outside' as one.
That's called "load balancing". New, incoming connections are randomly, or on round robin, dispatches over 1 of the 2 web servers.
I never used such a setup. neither if it works with UDP.Why does your setup works ones in while ?
It's a question of timing.
When he first "NAT to 10000 UDP" rule is introduced in the firewall, connections are accepted and send to the LAN IP indicated in this rule. As soon as traffic comes in from from that WAN based device, and NATted to that first "10000" LAN server, firewall states are created. A state contains the source and destination IP, (source ?) and dominations port, etc.
Now, when the second NAT rule is added (manually in the GUI), it might be placed before the first NAT rule (before not visually in the GUI, I'm talking the actual rules in the firewall, as NAT rules have no 'order').
Another WAN based device comes in using destination port "10000" and hits this second rule, traffic is send to the second LAN based port 10000 server.
The initial WAN based port 10000 is still able to send traffic to the first LAN based 10000 server .... as traffic is handled by the internal states .... the new second NAT rule isn't tested anymore, neither the first.As soon as the states time out, a question of seconds or minutes, all traffic from any WAN based device (the entire Internet' will hit the first available NAT rule, the second one never get used an ymore.
Btw : if you change the ANY part of the WAN source IP to an identified IP, then you can select which NAT rule you want to apply for what WAN device. WAN IP1 will be natted to LAN server 1, WAN IP2 to LAN server2. WAN IP1 and IP2 can also be entire networks sections.
-
Thanks @Gertjan
I tried testings it few times and it seems like what you are saying might be true... as soon as the 1st system on the network is able to establish the connection, 2nd one can no longer connect to the same port (on the same network).
The only workaround is to shut down the 1st server so that all the connections are released and then restart the 2nd server so that it can establish the new connection. And then if you bring up the 1st server, that will not be able to connect to the same port.
Theoretically, it makes sense, but I am very disappointed with this limitation and my brain is challenging me...
Imagine in a corporate environment out of 50 servers, if 5 of them need to have access to the same port on same network (maybe for same or even different applications), then how is that going to work?
Many Thanks,
Rav -
@raviktiwari said in Port forward to UDP 10000 is NOT working:
Imagine in a corporate environment out of 50 servers, if 5 of them need to have access to the same port on same network (maybe for same or even different applications), then how is that going to work?
You understand in the corp world they would have more than 1 public IP right??
You can not listen on the same IP on the same port - this is not a problem be it home network or corp network or Google or MS or even Amazon - this is how tcp works... You can not have 2 applications listening on the same port at the same time on the same IP..
You can have more IPs.. Now they can all listen on port X..
if you want 2 servers on rfc1918 space behind your network to both use port 10k to the public then you need 2 public IPs.. Period. Use a different port on the public side for 1 of them.
-
Thanks @johnpoz yes it does make sense and that is why I am also using 3 IPs now... 1 for Prod, 2nd for Test and 3rd for staging environment.
I am thinking and smiling... imagine a new engineer joins a corp and he has to do some installation/configuration and ultimately open a port - and if that port is already open on the same IP then he can tear all his hairs he is not going to get the connection. :-)
We can argue over documentation but we all know how much documentation is done in real-world and is accessible at the right time and even if we get it, we wish it was at least meaningful. :-)
@johnpoz I am going to pick your brain on another issue, which is bothering me for last 4-5 days... but there is already an open thread and you are active there, so speak to you soon.
Many thanks once again for all your help, support, patience and guidance.
Highly appreciate it mate.
Rav
