Port forward to UDP 10000 is NOT working



  • Hi,

    I have got a weird situation - I have done everything that needs to be done but "port forward to UDP 10000 is not working". Obviously I am messing up something somewhere, but the frustrating thing is:

    I have 2 environments (Test and Prod) and when everything worked in Test, I decided to move on Prod but got stuck with this issue for 2 weeks now.

    Just so you know, I am trying to install/use Jitsi which is an opensource video conf and they need port 10000/UDP to be open/forwarded. I have checked, rechecked and rechecked at least 100 times, but configurations (Test and Prod) are exactly same including the firewall port (ufw) on the server (I have separate Test and Prod Server as well)

    To test port forward/open is working, Jitsi folks asked me to run following commands:

    sudo service jitsi-videobridge2 stop
    nc -l 10000 -u

    And then from another machine (should have Internet access):
    echo "123" | nc -u my.public.ip.address 10000

    This should display 123 on the first machine.

    This is working on Test Server... and I get 123 on Test Server... however, when I run the same sequence of commands on Prod and echo it from Test Server - I don't get anything.

    The irritating thing is: I did get the response a few weeks back but I have no idea what all I have changed since then and now it is not working.

    Any idea, any suggestion, what I might be doing wrong. This is now on the critical path for me as this is now impacting production server/services.

    So any help would be highly appreciated.

    Many Thanks,
    Rav


  • LAYER 8 Moderator

    Without showing your NAT and Rule entries, no one can but guess, what is going on in your setup! So please add some information and screens to your post :)

    We are running multiple Jitsi and Jitsi Dev installations ourselves and have no problem forwarding necessary ports at all so it has to be something with your ruleset or configuration.



  • Thanks for getting back to me @JeGr and I completely agree with you for the lack of screenshot.

    So here we go (Blue one is Test Config and Red is Prod.)

    Test.JPG Prod.JPG

    Test-10000.JPG Prod-10000.JPG

    Hope this helps.

    As usual, any help will be highly appreciated.

    Many Thanks,
    Rav



  • Any idea, what I am doing wrong?

    Any hint or help to check/ensure port 10000 (UDP) is actually open or port forwarded, will be highly appreciated.

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    Troubleshooting nat should take you all of 2 minutes tops...

    Its really simple.. Does the traffic hit your wan IP? Sniff - do you see traffic hit your wan IP on port 10k? If you don't its not pfsense.

    If it does - then sniff on the interface your trying to send it to.. say your lan.. Do you see pfsense send the traffic on 10k? If so then its not pfsense.

    Example - I don't even have anything running on 192.168.9.100 on 10k, but I will setup a port forward rule..

    You need to test from a location online - say https://www.ipvoid.com/udp-port-scan/

    So created a port forward for 10k..
    portforward.jpg

    Now sniffing on wan for port 10k, and then sending traffic to 10k..

    My wan sees it
    wanseesit.jpg

    Now sniffing on my lan where this 192.168.9.100 device sits.. Sending traffic again..
    lansent.jpg

    Pfsense did exactly what you told it to do.. It sent the traffic on to the client - if the client is not listening, or does not answer there is nothing pfsense can do about it.. If the traffic never hits your pfsense wan - pfsense can not send it on.. If you spend more than 2 minutes trying to figure out what step you missed or where the problem is with port forwarding..To be honest you shoudln't be doing port forwarding because you do not understand it at a basic level and your just clicking shit hoping something works.

    /rant.. Sorry been drinking with buddies via virtual happy hour.. But its like every other hour we see the same exact thing about port forwarding.. Where everything you need to try and figure it out is right here
    https://docs.netgate.com/pfsense/en/latest/book/nat/troubleshooting.html



  • Generally, I get angry with such posts, but this one brought a smile on my face - great sense of humour. :-)

    You are right, when you know your stuff, it is a 2 min job but when you don't even 2 weeks is less (as in my case). I always thought networking is not my cup of tea as my core strength is Sys Admin, but then I was able to do a lot of stuff and got 2 environments up and running pretty well which gave me a false hope that I can do this sh!t... but clearly I am still not there - not yet.

    I have been drinking as well and so I don't think this is the right time to touch the Prod environment, so I will give it a go tomorrow morning.

    However, just so you know I tried packet capturing but I was unable to open the output file, it asks for some weird file type association and I am not able to open it at all. But never mind, I will try it again and report back to you SIR. Need to go back to my drinks - it's not nice to keep someone waiting... :-)

    Thanks for getting back to me and give some pointers - appreciate it mate.

    Thx: Rav



  • @raviktiwari said in Port forward to UDP 10000 is NOT working:

    I have been drinking as well and so I don't think this is the right time to touch the Prod environment, so I will give it a go tomorrow morning.

    That's a prerequisite in my shop... :)

    But seriously.. pfsense port forwarding works just fine and is fairly easy to implement. So when someone comes along everyone here will and should get very matter of factly arrogant and corrective.. (to a point). There is no- it doesn't work on my implementation even if it does on every body else's. That said..

    Remember-- Specifically WIndows but maybe others.. built in firewalls treat any "out of subnet address" as "public" and will block anything incoming. Yes if 192.168.2.1 is not in your subnet then it is public and untrusted. even if you think that firewall is off it is often not.

    /diag_packet_capture.php is your friend.. Get to know it well. :)



  • Alright then, @chpalmer provoked me and I took a peek on Production systems....

    ipvoid shows all good on WAN port... so WAN port is listening and pfsense is doing what it has been asked to do (PFA the SS).

    Ipvoid.JPG

    However, when I go to packet capture (WAN or LAN), I am unable to see any details in the box below and download has a .cap extension, which I am not sure how to open - does not open in notepad. PFA the SS for that as well.

    Packet Capture.JPG

    I am trying to install Wireshark assuming the downloaded files can be viewed in wireshark... but any idea how I can get the captured data displayed in the box below?

    Thx: Rav



  • It should show up easily in the box below.

    Im using GRC.com to try some connections inbound and doing a packet capture but that resource stops at 1055. But my screen capture will show you what to expect.



  • But it's not... I think I might have some browser issue... some extension, java whatever (despite the fact that I even changed browser).

    I am saying this because I know my test server along with test pfsense has UDP port 10K working... so I just tried capturing traffic for that.... but I can't see anything.

    Do I need to leave the capturing on for few mins, like 2-5 mins before I can actually see any captured packets?

    Thx: Rav



  • packcap.jpg

    If you see nothing I would bet your ISP is blocking that port.



  • How is that possible?

    Just tried port 80 as well as 443 (TCP) and we all know it IS working, so why there is nothing for these port in the box below - as you can see it (in your ss).

    Do I need to capture traffic from another Ip address.. as in my public IP is: 1.2.3.4 and both me and my server is connected to this same issue (via different interface at pfsenses) so can I do packet capture from this laptop or should I use another laptop whose public ip is: 5.6.7.8?

    Many Thx: Rav
    Many Thanks,
    Rav



  • Assuming your IP address is indeed public.. Not 100.xx.xx.xx or any of the others... Your ISP can block stuff. Especially if you are a residential customer.. Look at your TOS. They probably say no servers.

    Don't feel shy to ask them.



  • Yes I truly have a public IP: 62...* and this is a commercial ISP. When I signed the contract I gave them MY ToS saying I will be hosting servers. They even support me with my issues, for example in changing my reverse DNS and so on.

    Reason I am arguing is that when I give command on Test Server:
    nc -l 10000 -u

    And then from Prod machine (should have Internet access):
    echo "123" | nc -u my.public.ip.address 10000

    I do get "123" displayed on my test server... so that means I can forward the port and it is working, listening and displaying everything.

    So why it is not working only in pfsense packet capture box? And worst-case scenario, what will ISP block? A port right? Even if they block UDP port 10K, they cannot block port 80 and 443 on TCP... but I cannot see anything even for those ports?

    Many Thanks,
    Rav



  • Well.. commercial so you should be open.

    Ive never had issues with the packet capture page on both Fireflop and Chrome.. But that is the limit of my testing..



  • @raviktiwari
    are you trying from a network behind the router itself? if yes do you have some form of NAT reflection enabled?
    system > advanced ? firewall & nat > Network Address Translation section.



  • @raviktiwari
    All comes down to the way you define your rules: The wildcard to wildcard (all to all) rules should go into the floating rules as they should be applied to every network in your domain...

    Hope it helps



  • A part of the test pfSense :

    8841fdd4-c9fb-4aa0-9bdd-b880700f79fd-image.png

    The "10000" rule send over 1.16 Giga bytes.
    The second rule shows a whopping 3 Gb being handled by this rule.

    All this traffic didn't made it into the "Jitsi" process, or it was on the LAN .... probably it hit the "wall" between it ; the OS firewall of the "Jitsi" server.

    Byw : I'm not sure, but this does not look "ok" to me :
    You have two identical rules that have a port "10000" as a source address..
    What about change the range for the second rule : 10001 - 20000 ?



  • Thanks everyone... I don't know how and why ut it seems to be working... trust me, I did not make any changes other than the testing that @johnpoz and @chpalmer asked me to do yesterday... yes, I did restart the server as there were some bionic security and kernel upgrade that was pushed on the server and a restart was needed at the end of the update.

    I am going to take a backup of the configuration and keep it safe somewhere as this configuration issue has troubled be quite a few times and for a long time. So Thanks to everyone once again.

    However, the other issue, which actually delayed the troubleshooting of this issue (Unable to see the packet capture in the box below) is still something that I would like to get resolved.

    Many Thanks,
    Rav



  • Hi @taz3146
    Yes I am trying from a network behind the router itself?

    I checked system > advanced ? firewall & nat > Network Address Translation section and it was set to disabled, because I thought I will never need it. However, I have now changed it to: Nat+proxy.

    And I am still unable to see any packet captured details in the box below.
    P.S: UDP port 10K has already started working without making this NAT reflection changes. Do you still want me to make this change - I mean for the data capture box to display me the content?

    @Gertjan, that is in the plan, I wanted to get 10K work first and then change the other one from 10K1-20K now that it is working, I plan to make that changes.

    Thanks for your time, help and support.

    Thx: Rav


  • LAYER 8 Global Moderator

    If you can not see traffic in diag.. Yeah you have an issue there..

    As to firewall&nat set to disabled? What did you change?

    NAT Reflection mode for port forwards?

    You have zero use for that unless your actually doing the abomination that is nat reflection.. Which your port forwards would have zero to do with..

    As to your sniffing - you sure your sniffing on the correct interface?

    So from ssh, just do a tcpdump... Do you see flow of traffic?

    tcpdump.jpg



  • Hey @johnpoz thanks again for your time and support.

    I agree...I don't need to enable Nat reflection in advance but because someone suggested it to try so I gave it a go, which made no difference, so I can now revert it back to disabled.

    I understand it gives me some benefit to run everything from behind the same firewall but I never had any requirement or issues anyways, so I probably don't need it anyways.

    BTW, what do you mean by "If you can not see traffic in diag.." What is diag? Is it Packet Capture under Diagnostic mode? I know everything is working but the packet capture information is not being displayed in the box below. And that is what I want to get resolved.

    I think I am sniffing right interface, but I am more than happy to be corrected. So how do I ensure that I am sniffing the correct interface? Can you give me some commands of path to menu which I need to click to ensure I am doing it right?

    And yes, from ssh, doing a tcpdump on my network card gives loads of traffic flow - so much so that I had to stop it as it seemed it is never going to stop. :-)

    As usual amny help/support/pointers is highly appreciated.

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    Well what settings did you do on the diag, packet capture menu.. You have to make sure you select the correct interface for stuff to show up.. I have never seen such an issue..

    I would validate by doing a sniff for what your wanting to see via tcpdump say your 10k port on your "wan" interface that your selection in the diag, packet capture.. Do you see it there, and then setting the same filtering on the gui packet capture you not seeing anything come up??

    Is this traffic coming in via a vlan? Or a PPPoE connection? That might give you some issues with not seeing traffic in the packet capture screen



  • @johnpoz apologies for the delayed response...

    As far as settings are concerned, I have not made any changes and it is pretty standard stuff... you can see the SS for more details and if you see something silly, please do not hesitate to correct me.

    This test was done from a laptop behind the production server trying to capture UDP port 10K on Test server as well as on Prod Server (both of them have got different public IP). I even tried capturing the LAN port (changed the interface to LAN and Host address to 192.168 IP) - but same result, no result in the box below. It says packet capture is running, then I stop it and there is nothing in the box, even when I click on View capture, there is nothing.

    Packet Capture.JPG

    However, I have just realised an interesting thing:
    Just did a test on WAN and Opt1 interface, with respective IP for TCP port 443 and now I can see information in the box below.
    Also, it works only for the IP that I am behind.. so from a laptop behind Prod IP, I can not capture packets (or see any information) for Test IP address - which probably makes sense.

    For connectivity point of view, my WAN connection is via cable broadband (Virgin Media Fibre Optic), which terminates on ISP Router (now running as Modem because I have taken 3 static IP addresses). From there it comes into my pfSense and then from pSense LAN and Opt1 port goes to office network and server network respectively.
    issues with not seeing traffic in the packet capture screen

    So now the question is: why UDP ports are not being captured or displayed?

    As always, any help/support/suggestion/pointers are highly appreciated.

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    @raviktiwari said in Port forward to UDP 10000 is NOT working:

    So now the question is: why UDP ports are not being captured or displayed?

    Because pfsense is not seeing them.. If it sees tcp, it would see UDP as well..

    If your not getting them to your WAN... Then it can not forward them.. Quite possible they are blocked UPSTREAM of pfsense... If your not seeing them via your capture, then pfsense is not seeing them.



  • It makes Sense now : the pfSEnse setup may be identical, the upstream router isn't.
    It's this ISP (?) router that has to forward the UDP ports also.



  • @johnpoz but pfSense is forwarding now and I am able to get the job done... if you look at my previous post, I said, "It is working now.... Now the forwarding is not an issue... Issue is: why I am not getting any details in the box below for UDP Port 10K?

    Not sure if this is a tricky or stupid situation... Because when you did the same thing, you could see the data being populated in the box below (as per your ss)... so why I cannot?

    Maybe pfSense does not like me... :-(

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    Dude pfsense has no clue who you are or who I am - its some software that runs on some hardware - its sees traffic or it doesn't... If you say its working then clearly is seeing traffic that it can forward per your settings.

    My point is if your not seeing that traffic on your wan, then its not there...If your saying its working but your not seeing it then your looking in the wrong place.



  • Hi @johnpoz I agree with you that pfsense does not care whether it is you or me doing the testing. But we have an issue here... right? So looks like pfsense DOES care who is doing the task. :-)

    If I am looking at the wrong place, I want to be corrected. I have shared the screenshot, pls let me know what else I can do.

    And to top it up, I am back at square one with the port forwarding as well.

    I had to format my test server, build it up with same IP address, did not touch the pfsense at all... and now again UDP port 10K is not working... I can see the traffic moving fine at WAN level, but then from there, it does not reach the Test server.

    On Test Server I type:
    sudo service jitsi-videobridge2 stop
    nc -l 10000 -u

    And then from Prod machine (which has got Internet access):
    echo "123" | nc -u my.public.ip.address 10000

    This should display 123 on the Test machine. I had an issue when I started this thread and it finally worked... but now again it is not working. This is still working on Prod Server... and I get 123 on Test Server...

    As I said, I have not made any changes on the pfsense (neither on Test nor on Prod), so why it has died again?

    Any idea/suggestion?
    UDP-WAN.JPG

    Many Thanks,
    Rav



  • Ok... So I think I have found the root cause of the issue, but I still don't know the resolution.

    I connected my laptop on Test network and using firefox and chrome, I was able to capture packets and everything was displayed in the box below (I still don't know why and how it is capturing, whatever it is capturing, but that is for some other day).

    However, when I connected the same laptop on Prod network, using the same browser the box below goes blank. So this means it is not the laptop or browser setting... it is something with the network or pfsense configuration that is blocking the data to be displayed/popped up in the box below.

    Does that make sense? Does that ring any bell to anyone, as in what changes should I make in the Prod environment, so that when I do a packet capture, I can see what I am supposed to see.

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    @raviktiwari said in Port forward to UDP 10000 is NOT working:

    echo "123" | nc -u my.public.ip.address 10000

    Where is this machine - your wanting to use NAT reflection for this to work?



  • I am stopping the service on Test Server and then listening on UDP port 10000 - using following commands:
    sudo service jitsi-videobridge2 stop
    nc -l 10000 -u

    After this, the cursor is waiting to listen and then on Prod Server, I enter the command
    echo "123" | nc -u my.public.ip.address 10000

    This shows 123 on the Test server.

    Both these Servers are on 2 different public IP behind 2 different pfSense.

    I think I need NAT reflection but not sure where to make the appropriate changes.

    I am also having another issue with NAT Reflection - few of the websites that I have a built-in Test environment, I cannot access it from within Test environment. However, it is accessible from the Prod environment and the outside world.

    Again this means that I need NAT reflection but not sure where to make the appropriate changes. :-(

    Thx: Rav


  • LAYER 8 Global Moderator

    If your client is behind a different pfsense than where your server is then no you wouldn't use nat reflection.

    Again where have you shown that this traffic actually hits pfsense wan?

    Last I recall you were not seeing this traffic when you did a sniff.. I gave you a site that will send UDP traffic.. Use it.. sniff on your wan.. Do you see it?? If not then no its never ever ever ever going to work!! Period!



  • Sorry if I have not been clear enough but the setup is pretty simple.

    I have 2 network - Test and Prod... both of them have different IP, both came from same ISP broadband modem but one goes in Prod pfSense and another goes in Test pfsense and then from there are managed as a different network. I have attached a diagram - hope it makes it easy to understand.

    Netwprk Diagram.JPG

    Now I want port 10K on UDP to be open on 2 Test Servers and 1 Prod Servers. To ensure it is working, I have been told to use those commands that I shared above. So I make Test server listen and then I send the message using echo to Prod server, which does not reach, but the other way around is working and so my application is working on Prod but not on Test. I need it to work on Test so that I can do further development and testing.

    So this is my 1st issue - port 10K on UDP not working in Test Environment.

    Yes, I used your tool (ipvoid) and it shows WAN port is listening on UDP 10K - I have attached the screenshot in my previous message. But yes I am not able to see any traffic on within pfsense because that box is not displaying anything - especially on production pfsense (because it is connected on Prod environment). However, when I do the same thing on the Test environment using test pfsense I can see the traffic flowing when I sniff it.

    This is my 2nd issue - why I am unable to see sniffing details in the production environment.

    And the 3rd issue is:

    Why few of my websites hosted in test environment are NOT accessible from within the Test environment but it can be accessed without any issue on Prod as well as anywhere else in the world.

    Hope this clarifies the situation and thanks once again to be there to support, I sincerely appreciate it.

    Many Thanks,
    Rav


  • LAYER 8 Global Moderator

    Step 1... Sniff on wan of pfsense that holds the IP your trying to send data too... Go to say the site I linked to and send UDP traffic. If you do not not see it in the sniff.. Pfsense can not forward anything.. Period!!

    Untill you show us this sniff, there is nothing anyone can help you with..

    This is my 2nd issue - why I am unable to see sniffing details in the production environment.

    Because the data isn't get there is the most logical answer to that..



  • Hey @johnpoz I have no doubt whatsoever that you know more than me and so I dont want to sound argumentative or disrespectful, but pls try to understand the situation.

    Going by your requirements:
    1.> ipvoid says packet is flowing without any issue - PFA screenshot.

    Ipvoid.JPG

    I hope this answers your 1st question. While my 1st issue is still at hand why my application is not able to use UDP10K - why the port is not working?

    Coming to 2nd question...
    Here the issue completely different - we know all data is flowing fine, I can also get messages using those commands and I can also access my application. So technically there is no issue at all and we should not be discussing about it... but the problem here is, when I sniff and use pfsense inbuilt tool "packet capture", I cannot see anything in the box below.

    And this is not because there is no traffic... we know everything is working fine...We also know this is not a laptop or browser issue because when I connect the same laptop and use the same browser in test environment, the same box shows all the details - for both Test and Prod public IP.

    This means ( atleast in my brain - that there is some configuration that I have done within pfsense for Prod environment that is not letting that box fill up with relevant information/traffic).

    And this is my 2nd issue.

    For the time being, let's leave the 2nd issue we can discuss it once we have sorted these 2 open issues.

    Many Thanks,
    Rav



  • This post is deleted!

  • LAYER 8 Global Moderator

    You need to show a sniff! where the traffic actually gets there..

    If you can not see it, either can pfsense.. Do a simple tcpdump on pfsense for port udp 1000.. if your having issues with the gui.



  • Hmm.... So where and how do I do that? Which tool do I use?


  • LAYER 8 Global Moderator

    Already went over this.. Dude I really want to help you... But this is getting more frustrating for me, than I really want to deal with... Troubleshooting port forwarding is 2 minute thing.


Log in to reply