Pfsense not reaching wanGW from lan

[shh]

Hi

I have a test setup where my pfsense cluster works flawlessly, I have pulled network cables, and power cables alike to test.
But when i move to production enviroment, where i have the exact same ip adresses - only difference is that instead of the webserver, i have my hosting providers Cisco switch, no traffic from lan reaches the cisco switch.

TEST enviroment
Laptop <-LAN-> Pfsense <-WAN-> Webserver

Laptop setup:
LAN ip 10.0.0.20 (NAT 1:1 x.x.x.20)
local firewall disabled

Pfsense1 BACKUP
LAN ip 10.0.0.2
LAN CARP ip 10.0.0.1
WAN ip x.x.x.2
WAN CARP ip x.x.x.4
Default Gateway x.x.x.1

Pfsense2 MASTER
LAN ip 10.0.0.3
LAN CARP ip 10.0.0.1
WAN ip x.x.x.3
WAN CARP ip x.x.x.4
Default gateway x.x.x.1

Webserver
LAN ip x.x.x.1
did: route add x.x.x.20 mask 255.255.255.255 x.x.x.4

With this test setup Laptop can view webpages on the Webserver, and Webserver can view webpages on the Laptop No flaws what so ever.
When laptop telnets x.x.x.1 on port 80 i can see in netstat and iis logs both, that i present myself as the correct nat x.x.x.20
i have enabled ICMP so i can allso ping from laptop to webserver and webserver to laptop.

Once i move the pfsense cluster to my production enviroment, things change.

Instead of webserver i now have a cisco switch with a vlan to my two pfsense boxes.

My MASTER node is able to ping x.x.x.1(cisco switch) from wan interface
The cisco switch can ping all wan adresses on the pfsense cluster .2 .3 and .4

when trying to ping from LAN side of the pfsense again a server with ip 10.0.0.20 that is natted to x.x.x.20 i can see the icmp passed in the firewall log. But the cisco switch never recieves the traffic
the lan side server can ping x.x.x.2 x.x.x.3 and x.x.x.4 (pfsense wan side ips) but somehow the traffic is not passed on to the cisco switch.

Added info, from wan ip in diagnostics -> ping i can ping 8.8.8.8 - traffic is passed nicely.

on the cisco switch i do not see other ip adresses on the vlan contacting it than x.x.x.2 x.x.x.3 and x.x.x.4
When i fail back to my old firewall (non pfsense) the cisco switch is instantly populated with around 125 ipadresses. (all my natted addresses)

I am at a loss as to what i should look for here. Keep in mind that i never so much as changed an ip adress. Since my test enviroment ip wise replicates the exact setup, with public ips and everything.

i tried testing again back in the test enviroment, and spoof the mac address on my webserver which acts as the wanGW to see if it was somehow the ARP messing me about. It isnt, everything works.

[stephenw10]

Do you have VIPs on the WAN to NAT to?

If you're using NAT there should be no need to add routes on the WAN side server (or switch) as the x.x.x.20 address is in the same subnet.

Steve

[shh]

@stephenw10:

Do you have VIPs on the WAN to NAT to?

If you're using NAT there should be no need to add routes on the WAN side server (or switch) as the x.x.x.20 address is in the same subnet.

Steve

Thank you.

The wan webserver is a windows machine. It has x.x.x.0 routed on-link shown by route print

so when it recieves packages from x.x.x.20 it never goes to its gateway, thats why i have that route in the test setup as i have it switched in a non managed switch. On the production network the switch has an ip addr directly connected to the firewall.

So it feels like the pfsense is not even sending its natted packages to the cisco vlan in the production enviroment, im gonna try and put up a build with no CARP to see if its my outbound natrules somehow messing things, they work in test though.

I control all the x.x.x.x ips so im not sure about your VIP question, i dont think i should have to make a VIP for every nat manually, it should be taken care off by the outbound nat rules right?

[stephenw10]

If you don't have VIPs then there will be nothing to respond to ARP at those IPs.
Run a packet capture on the WAN to see what is actually happening there.

I'm not clear on the route you added. I would not expect it to go to it's gateway as it's in the same subnet as you say. That should be OK though since the IP is 1:1 NAT'd so the external address that the web-server sees is in that subnet.

Steve

[shh]

The route is only in my test enviroment, because the Default gateway for the pfsense box is a windows machine, and that windows machine needs to know how to reply on x.x.x.y network. If i dont add a route to x.x.x.4 it will try to reply on-link.

That windows machine is not part of the production enviroment, in production, im connectet to a cisco switch vlan.

When setting VIP i only make a 1:1 nat rule, and have outbound to automaticly do the rest, in test i can see from lan to wan that the correct nat is tranlated and sent to the webserver on wan side.

i have a /28 range, how in pfsense would i set all my nat VIPs, other than in 1:1 nat ?
/Shh

[stephenw10]

Hmm, well I'm stil not sure why you need the route. If you are NATing then the traffic will all appear to be coming from the same subnet as the Windows box so it can reply directly without needing a route.
Anyway given that's only your test environment that's an aside right now.

You can add a range of VIPs if you use the Proxy ARP type and a matching range of 1:1 NAT entries by specifying the subnet size.

Steve

[shh]

Rumor has it that the problem was solved yesterday, and we will test it in production, at the start of next week.
Solved with adding VIPs as you said Stephen, i will write a status and follow up on this when its in production, with the resolution incase someone else misconfigures the same way i have and stumples on this thread later.

Thank you for the help :)

/shh