Thanks for the reply CMB.

Have some new information to add to this. Here's our setup: We have 2 identical SuperMicro servers, one in production, one as a backup. No CARP, as we have another issue/bug we a troubleshooting there as well. Currently, the production box is set to "Round Robin" for NAT Translation, and the backup box is set to "Round Robin with Sticky", and both are running fine.

We changed the production box from Round Robin, to Round Robin with Sticky. The server was fine for about 30 seconds, and then locked up the exact same way we saw before. All interfaces stayed up, and everything looks fine, but the box is not administrate-able and no traffic was actually passing. We cut to the backup (which was running fine with Sticky), and we saw the exact same thing. Everything worked fine for about 30 seconds, and then poof, box explodes. We had to reboot the server, disconnect all traffic-bearing interfaces (so it wouldn't immediately lock up again), and revert the config to get things back up and running.

So, it seems changing the config wasn't a problem until you start having traffic use the new NAT translations options. Has anyone seen anything like this before? Is this a software bug, or does it seem more like a hardware incompatibility?

For those who are curious. We are trying the Sticky option due to possible issues with client devices have multiple sessions that NAT to multiple IP public addresses. When we statically set those clients to a single NAT IP address, those problems clear up. So, we were hoping that the Sticky option for NAT might alleviate these issues wide-scale.

Thanks!

Hello everyone.
I faced today what I see as a quite similar issue… and I found a kind of workaround.

I am going to describe my scenario, the issue i faced and the workaround.

I have a multilan and multiwan installation with 2.1-RELEASE.
A lan and a wan are dedicated to VoIP (SIP) traffic. For those not familiar with SIP, it's just traffic on udp/5060 (in this case).
I had a working installation, then I made changes and I decided to reconfigure the firewall rules from scratch.
The SIP server is in the VoIPLAN with an ip address IPPBX.
For outgoing traffic i created a firewall rule on the VoIPLAN interface to send all the udp traffic originated by the IPPBX ip address through the VoIPWAN gateway/connection (VoIPWAN is the dedicated wan with a bunch of public IPs on it and a router that interconnects with the voip provider).
Then i created a Port forwarding rule to allow incoming traffic from VOIPPROVIDER ip address to be natted towards IPPBX (considering port udp/5060). Firewall rule was automatically created.
IPPBX sends periodically an udp packet to VOIPPROVIDER ip and receive an analogue udp packet in reply (sip ping), all on the udp/5060.
That was working fine.

Incoming traffic originated from VOIPPROVIDER towards VoIPWAN ip was unfortunately not forwarded.
Actually i tried to craft some udp packet from another public IP and sent them to VoIPWAN ip: that traffic was forwarded so i started to think that was something related to the originating IP.

Then i figured it out then in this case there were two NAT session conflicting: one dynamic (created by my IPPBX with the periodic sip ping) and one static (created by the port forwarding rule)… in this case, the dynamic one was the first being created. Previously that was the second one: infact, in the first setup of the system, the sip ping was not active but i activated when the firewall was already configured.

So I took these steps in sequence:

stopped the sip ping

cleared all nat entries, in states page, related to IPPBX ip and udp/5060

edited and resaved PF rule and reloaded rules

reactivated the sip ping

Now all seems working… I will confirm tomorrow when i will be in office ;)

If someone wants to investigate this issue, i looked at the states' entries in the not-working and in the working situation and they were slightly different.
Not working:

VOIPPROVIDER:5060 <- IPPBX:5060

IPPBX:5060 -> VoIPWAN: <randomport>-> VOIPPROVIDER:5060</randomport>

Working:

VOIPPROVIDER:5060 -> IPPBX:5060

IPPBX:5060 <- VoIPWAN:5060 <- VOIPPROVIDER:5060

I suppose the "single arrow" mapping is the dynamic one. Maybe since it was already present, it forced the static NAT to use a random port and that made mismatch the portforwarding rule? (this is probably like throwing dice ;D ).

Bye,
SB

Running a vpn server inside the network is at best a problematic setup.

OpenVPN servers behind firewalls can work with a port forwarding and a static route so there is no rocket science involved.

One scenario (that requires OpenVPN server(s) BEHIND pfSense) is when there are multiple OpenVPN servers behind the firewall/pfSense. E.g. for penetration/version or testing and/or high availability.

It would really help me (and Christoph) if there is some pfSense configuration/setting available who supports this configuration.

Thanks, regards,

Beau

Sorry… i forgot to mention that the other pfsense box is on another part of the map. The only connection it had is through the wan(10.180.10.254:10001 which my pfsense box is 10.20.20.254:10000 ). I can access/ping/ssh them using this 10.180.10.254 ip as long im in the 10.20.20.x network. As in illustration, it would be like this

my current set up
.ADSL --->my pfsense box (10.20.20.254:10000) = cannot connect/nat/portfowward to 10.180.10.254:10001

what im trying to achieve is

.ADSL--->my pfsense box (10.20.20.254:10000)--->my other pfsense box (10.180.10.254:10001)

And as why im trying to avoid vpn is, on my pfsense box, im still using 2.1.5 which is the only version (i hope its not) didnt broke sarg report and squid3 in transparent mode and its vpn server seems broken (constantly restarting vpn service due to error which also will bring squid3 and squidguard to a halt).

Why would you not just listen on 443 the standard ssl port… Then you could access it externally and internally with just https://fqdn

What purpose does listening on 5001 vs 443 serve when your forwarding 443 from the public side??

Nat reflection should really be be avoided whenever possible..  And really don't see as needed here if you just listen on the standard port, or just forwarded 5001 to 5001 and used the url https://fqdn:5001 both external and internal

Agreed, nat reflection is not something you want to do.. Why hairpin the traffic if you don't have too?

After getting the NVR set up on a different IP and starting my search for the unknown device, when I did the MAC lookup and it said IOGEAR, I knew exactly what it was. The NVR is in a secure cabinet a good distance away from the wired part of my network so I use an IOGEAR Ethernet to WLAN bridge which for some reason decided not to stay on its assigned IP of .252 and went rouge on .246 when it was connected to the NVR. When I changed the NVR IP it went back to its .252 and has stayed there since and everything is running great!

I am glad I know what the problem was so if it does happen again, I should be able to get it fixed.

One more quick question: Would creating Static ARP Entries for my Static Mappings have anything to do with this issue occurring?

Thanks for the answer!

I didn't get the chance to test pinging with the LAN address on pfSense, but the firewall rules on the server side were already configured to pass traffic from LAN to Remote.

However, I did solve the problem by realising I shouldn't fill in the "Remote IPv4 Network" field with the remote subnet in the openvpn configuration. Instead, I only need to have outbound NAT set up, and it started working!

Really appreciate your help anyways!

Good catch.. Yeah source should be ANY!!  doubt ftp is coming from port 21 to 21

What are your client going to be using passive or active??  As stated pfsense has no helper/proxy for ftp any more.  Dropped in the 2.2.x I believe..  So if you want to run a passive ftp server behind pfsense you have to forward the passive ports it will use..  And you need to make sure the ftp server actually hands out your public and not its private IP.

This is clickity clickity setup in filezilla server - but it still has to be done.

You need to understand how ftp works to be honest if your going to forward it and not have some helper fix it up for you..  dd-wrt has a helper to auto forward the passive ports and adjust the IP if it sees the server sending out private one.

http://www.slacksite.com/other/ftp.html

If you want some friendly advice - ftp should of been dead years ago… You should move to sftp as it is way more secure and don't have to worry about the control data channel active passive nonsense.

any update on this?

We had something similar go on when using the secondary gateway as master. Would work for a short amount of time and then end up failing.

I am leaning towards an ARP cache issue on the FIOS modem/router combo that we pass through

Port alias called PBX_Ports containing all of the port numbers needed for SIP, RTP, and other control ports. (usually 5060 and 10000:20000, but varies from provider to provider and PBX implementation)

How is this unclear?  They give you the standard defaults that a lot of VoIP phones use, and then say to check with your provider.  We can't tell you which ports they use if they aren't standard, so if the standard ports don't work for you then you will need to contact your provider and ask them.  They probably have a FAQ about it on their website.

Correct, the NAT translations aren't logged.

^ Split DNS +1

So first you need to validate that 443 is actually hitting your wan IP.. Its quite possible its blocked upstream.

2nd validate that it actually gets sent to your local machine… This is 5 seconds of sniffing on pfsense interfaces with diag, packet capture.

You sure machine your forwarding too doesn't have firewall blocking it?  Have you gone through the troubleshooting doc?

https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

You don't have captive portal setup on the interface your server is on?

Hello Everyone,

Sorry for the confusion. Let me explain my goal clearly.

I have two WAN links terminated at my Router. Router has the internal network 10.50.0.0/16 and I can distribute the load on WANs based on IP address(Which I need to do for different departments )

I want to place a PF Sense in my Internal network and want to create other 3 networks (of different department) behind it.

Now I would like to do NAT for each network's traffic so every network will get the appropriate WAN and the respective IP address.

Please let me know if you still have any confusion or questions.

Thank you