NAT Issue on Virtual IP on WAN2

zilnar

Hello, I'm having some issues with NAT/Port forwarding to a secondary WAN Virtual IP address. It will be hard to explain, but we'll see what we can figure out.
Couple notes.

Ideally I Will have a Round ribbon type connection, an app will point to service.domain.com, I'll have the DNS setup with round ribbon and IP monitoring, so traffic can come through either my primary or secondary ISP Connection and if one goes down my DNS will compensate, If there is a better way of doing that, I'd be happy to try.
pfSense running in Hyper-V 3 Physical NICs one for Each ISP and LAN
I Have 2 ISPs, both with a /29 subnet (5 available IP Addresses)
I have Both interfaces setup and Working fine
Failover works fine
NATing from any of the 5 IP addresses on the Primary WAN work with no issues.
NATing from the secondary WAN Configured IP address works with no issues.
[NATing from any of the Secondary WAN Virtual IP addresses does not work.
The Target Server is on a VLAN that pfSense Routes
This happens on any Service/Target Server I try to Put on the secondary WAN Virtual IPs
I used to have this working on a sonicwall, but I got a new Secondary connection at the same time I replaced the Sonicwall with pfSense.

When I Setup a NAT Rule for a Virtual IP on the Secondary WAN, it Partially works, I can use the App for a very short bit (less than a minute) then after that, I get very spotty access, I Would say 90% of the traffic does not work.

it almost seems like there is something Messing up the timing of these packets.
When I run a Packet Capture I get a ton of these errors:
Expert Info (Warning/Sequence): Connection reset (RST)

I've tried many things to try to work this out,

I've recreated NAT Rules
Rebooted everything :)
Talked to my ISP to verify Static IP is correct and setup correctly on their end (don't know that I trust their verification)
Tried just about every option in the Advanced NAT Settings, both in the NAT Rules and system-wide Settings.
1-to-1 NAT, but I don't usually use these, so I may not have had it setup properly, and I don't think it would be helpful if it did work.
Setting Different Gateways rules to try to make sure Traffic was flowing back out on the proper ISP.
many different random items I've found in my search.
swapped the Virtual and Interface IP, this Worked, but only for the current interface IP, the Virtual IP's Still did not work.
reset connections, Arp , and pretty much any other table like that I could
I Believe Early on I removed the Similar NAT rules from the Primary connection.

I haven't tried

re-setup the SonicWall, that is my next test if we can't figure it out here.
Swapping the Primary and Secondary WAN connections.

viragomann

@zilnar
Did you specify the proper /29 network mask for the virtual IPs?

Ensure that there is a pass rule on WAN2 interface tab matching the forwarded traffic.
No floating pass rule or one on an interface group must match the traffic!
Enable logging and specify unique rule names, then check the log to get sure.

zilnar

@viragomann
Thanks for the quick reply!

yes, the proper /29 Mask is set, I did try both 32 and 29.
Yes Pass rules are there and as far as I can tell are working.
I landed on the floating rules after I posted this, I did have about 7 Floating rules, I have Worked to disable all of them, but I haven't had a chance to Test externally yet, I'll let you know if that helps
I guess i haven't dove too far into the logging, if the floating rules don't help, I'll do that!

zilnar

@viragomann
So I have the same Issues After removing the Floating Firewall rules.....

I have enabled logging on all of my WAN1 and WAN2 and created a Deny all Rule at the end for logging. ultimately, I wasn't able to figure out anymore from that, any time it works properly I see the Allow Action happens under the proper Rule, but any time it doesn't work, I do not get anything in the logs, I think the Packets are being reset before it hits the firewall.

going back to the ISP setup, I have a rfc 1483 via dhcp connection, so They do have some equipment before the pfSense Router. I guess I don't really understand how that works, I usually try to set up as a transparent bridge, but I Don't think I can do that with this ISP, I have disabled all firewalls, and I have verified that the IPs do work.... but not to the point of port forwarding since I would need to take everything offline and set up a bunch of stuff to test.

Not sure what my next step is, maybe I'll try the Sonicwall, just to see if it does anything different, but that is going to take some time to get setup when I can take down our production system...

johnpoz

@zilnar said in NAT Issue on Virtual IP on WAN2:

Connection reset (RST)

Yeah that is a device involved in the conversation saying I am done - I don't want to talk to you anymore.. This is not a port forward problem.

Not sure what your wireshark is suppose to show us that some sort of port forwarding isn't working when clearly those are devicees talking to each other.. and then you see a client send to 443 a fin,ack, and then both server 443 and client 50207 sending RST..

How does that say that pfsense is not sending traffic through from client to server??

Then same sort of issue with the 50209 conversation..

pfsense is not going to send RSTs etc.. So where is this nat or port forward problem??

zilnar

Update, I Was never able to get this working properly, but Now that the 2.7.0 update has been released, once I updated, everything is working as expected. not sure if it was some sort of Hyper-V Driver issue, or some other bug that was fixed in this release.... just glad I can utilize my secondary internet connection better now. thanks for all the help!