Internal pfSense + external pfsense + WAN Router = limited internet

[dbennett]

Greetings,

I have a situation and it's become frustrating as we have tried 2 times now to upgrade our internet with our ISP with no success.  Here are the basics.

Current setup High level starting from the WAN:
CISCO Switch (Old ISP edge) -> Gig Switch (not configured) -> pfSense WAN port (considered External) -> DMZ Switch (not configured) -> pfSense DMZ port (considered internal) -> pfSense Internal LAN port -> My computer.

Future setup high level starting from the WAN
CISCO Switch (New ISP edge) -> New ISP supplied router -> Gig Switch (not configured) -> pfSense WAN port (considered External) -> DMZ Switch (not configured) -> pfSense DMZ port (considered internal) -> pfSense Internal LAN port -> My computer.

Current configuration details and status:
Configuration:

• We have three public subnets that are currently active and in use

• Our pfSense External connects to a CISCO ISP Switch

• pfSense External is NATing "1:1" select public IPs

• pfSense Internal is NATing "Outbound" my (computer's) subnet.

• Traffic that leaves my subnet uses the pfsense internal DMZ port NAT ip and WAN traffic meant for my subnet uses the pfSense internal DMZ public ip.

Status: We have no issues getting to and from the internet.

Future configuration details and previous results:
Configuration:

• Same as current configuration but with one difference.

• We have three public subnets that are currently active and in use

• Our pfSense External connects to a ISP supplied router

• pfSense External is NATing "1:1" select public IPs

• pfSense Internal is NATing "Outbound"; Settings [DMZ / ANY / my subnet / ANY / Interface address]

• Traffic that leaves my subnet uses the pfsense internal DMZ port NAT ip and WAN traffic meant for my subnet uses the pfSense internal DMZ public ip.

Result: My subnet is unable to get to the internet.

• Traffic from our DMZ and other sub networks internally (also NATed by pfSense external) are able to get to the internet and browse without issues.

• Other subnets that are NATed are behind load balancers that also NAT to other subnets and they can assess the internet just fine. (Ex; pfsense (public -> NAT) -> DMZ -> Load balancers (NAT / Private IPs)

Testing / information we do have:

• ISP says configuration is the same on the new device compared to the old device.

• Gateway IP's haven't changed

• Network settings haven't changed.

• Internal settings haven't.

• Ping from pfSense Internal to 'google.com' failed.

• Ping from pfSense Internal to 8.8.8.8 failed.

• pfsense internal DMZ (LAN) selected Default Gateway of pfSense External's DMZ (WAN) port

• Packet capture on pfSense internal show's the ping leaving but not returning.

I have the configured ISP router and using an identical configuration as my pfSense External box I am able to validate that ALL the public subnets gateways are accessible.  I was even able to plug into the ISP port side, find the next IP on that port and was able to communicate through to it.  BUT…  I was NOT able to add a test pfsense Internal.  So I only can confirm that the ISP router 'appears' to be functioning correctly.

I hope this is enough information.

My thoughts?  ...  That adding a router between the CISCO switch and my pfSense External firewall is stripping away the information needed to for the WAN traffic to return to my subnet.

Please!  Any thoughts or ideas would be appreciated.

Thank you ahead of time for time spent helping me.

dbennett

[dbennett]

Good morning to all!

Well!  Thanks to all that read my issue.

And this isn't an issue any more.  Turns out the last two attempts by the ISP failed because of them.  We finally were assigned an engineer who knew what he was doing and did it right.

As for my post, if anyone can give me some advice on how I could have done it differently so that anyone would have tried to help me out sooner, I would appreciate it for future posts.

dbennett