Thanks for you Time johnpoz !

First - i extendet my Drawing:

So are you natting with pfsense or not??

At the Moment yes.

Really need to understand how you have pfsense configured here..

Actually, at the moment this isnt working at all and i just tested around.
No NAT Rules Set. i Just test it with Down-nat (no outbound - its automatic)
Also test with ANY-ANY Rules on WAN side.

But let my describe this Scenario from the beginning -

The Internet Uplink is a 100Mbit directional radio from Mountain to Mountain. (i live in Austria)
The Cisco Network / Switches provice several Networks for each individual Customer on this Mountain. (Hotels, ski lift Stations, flatlets, and so on)
So its Come in on One Point and is distributed via Fiber (from 100 to 500 meters) to differnt locations.

Cisco Switches are 48Port.
Each Customer has 4 Ports on these Switches with their Own VLAN provided.

Mine is - 10.3.17.0/24
They Router/Gateway for my Network is 10.3.17.254.
So Basically it isnt a Transit Network - i Just Used it as one.
I will call this 10.3.17.0/24 Network for now "Cisco Network".

Cause you can easly "hack" (and by hack i mean just plug a network cable into the Switch - cause they are easly to access) and my Customer said he dont Trust the other Customers who use the provided Network/Switches - i build the Pfsense Network for my Customer with his own Network 10.3.16.0/24 and the DMZ 10.13.16.0/24 for the Guest-Network.
Classic WAN-LAN Network.

As mentioned there is also a OpenVPN Tunnel to a Branch Office. (the Branch office has a Public IP - the Main Office hasnt …)

Until one week everything worked just fine.
Now my Customer bought a Fancy KNX System from an Electrican and control several things. (Light, Heating, ..)
No Problem in my Own Network.
The KNX Server is is 10.3.16.21
The Server also Use Multicast for some controls.

My Customer was so Happy with this System he decided he want to control more !
And the More are Other Locations in this Cisco Network. (wich are conected via Fibre)

And the Struggle begans - cause i didnt designed the Network for that Purpose.
The Customer already had Clients in the Cisco Network - but Only for Internet Usagage - not to communicate with Clients in the secured 10.3.16.0/24 Network.

Use the 10.3.17.1 (Pfsense WAN) as my Gateway was just a plain stupid idea and i just realized it while i was starting this topic here.
I ran into this Idea cause if i set 10.3.17.1 as Gateway on my Laptop i could ping 10.3.16.1 without a problem - and thought - easy cheezy i got this.

Another Idea is - To Hook up my LAN Side with the Cisco Switch - but i think this will also bring trouble since VLAN is designed just for one Broadcast adresse.
I dont know how the Cisco Switche are configured.
I cant test it - but its not clean.

If i cant get this to work there is a Worst Case Solution for the Problem -
There are Several free Pairs on the Fibre Cables.
So i can build my own new Network wich is in the 10.3.16.0/24 Network (with new Switches on each location)

but the best way would be to get this working with the pfsense.
Cause it gave me headache thinking about to build another Network since there is already one.

Hope this helps you understand my "Scenario"
Tryed my best.

Look at the rules that are automatically generated on LAN. Duplicate them on OPT1 (Changing all instances of LAN to OPT1 of course).

The installer creates a rule for you on LAN but when you add another local interface you have to create the rule yourself.

NAT should be taken care of automatically.

Yep.  It's a real problem.

Only solution I have been able to come up with is to setup an OpenVPN tunnel bound to a gateway group.  This creates a potential re-design at the head end depending on what the phones actually have to connect to.

It works a treat.  States can remain as the tunnel never changes.  Only how the tunnel routes out.

If you need that VPN to terminate on multiple head end circuits (no BGP), you can set additional destinations in the advanced options window of the VPN.

I have a compromise working…  I wouldn't say this is solved, so please help if you can suggest a better way?

It looks as though the ADSL modems we have don't support VLAN, one of them does support bridging PPPoE though.
As an experiment I've defined an additional PPPoE interface (OPT1) which gives pfsense an inteface for NAT rules to be set up against. OPT1(primary) and WAN(second), and it works!

The following tweak is now allowing inbound NAT from both ADSL routers to the same SSH server

- same physical configuration as above
  - updated adsl1 modem to run PPPoE bridging, so its public IP is now on the pfsense router
  - WAN configuration as follows:(interfaces->assign)
    + PPPs set up a new PPPoE connection with my ADSL/ISP login details  (on adsl1/WAN)
    + created a new Interface OPT1 which usese that PPPoE connection
    + WAN interface is now configured for the secondary IP (10.10.2.2/24) (I deleted the Virtual IP)

- Routing->Gateways:
    + GW_WAN on PPPoE (default gateway)
    + GW2: on WAN set to have gateway 10.10.2.1 (also had to go back to the OPT1 interface and set its gateway to GW2)

primaryDSL  +-------+ (10.10.1.1)       +----> | adsl1 +------+----------+(other 10.10.1.0/24 devices             +-------+      |                now have no gateway with PPPoE moved to pfsense)                             |   secondIP  +-------+ 10.10.2.1       +----> | adsl2 +------+             +-------+      |                             | PPPoE:PrimaryIP/WAN:10.10.2.2                     +------+------+                     | pfsense|NAT |                     +-------------+                             |192.168.1.0/24                             |                             |  +------------+                             +--+ ssh Server |(192.168.1.10)                               +------------+

Its not ideal, as we use the 10.1.1.0/24 subnet for other things, and with PPPoE on pfsense they don't have a public gateway any more… so it replaces one problem with another.  (But it is a configuration that I can work with if there's no better method)

I guess I'm coming back to my original question, Is there a way to Assign NAT rules to a second IP on the same physical WAN network port? This would work better in our situation if its possible...!?

Using an L3 switch and pfsense is easy.  You setup the switch and VLANs for all local traffic and routing of local traffic.  You then add an extra VLAN I call a router VLAN which will connect to pfsense.  I use an /24 mask but the only 2 devices in this network are pfsense and the switch IP for the router VLAN.  Create a default route on the switch which points to pfsense.  Pfsense is just used for internet traffic.  You need to create routes on pfsense to point to all VLANs on the L3 switch which of course will point to the switch's router VLAN IP address.  You then need to create firewall rules to allow for all the network VLANs on the L3 switch.  I think that covers it.  pfsense will also provide NTP to your L3 switch.

This is going to be a temporary arrangement while we transition to the new IP address and network - the new 60Mb/10Mb network will cost us about $100/month whereas the existing 3Mb/300kb network is costing nearly $300/month.  So we're just in it for the money.

At a high level, you would configure policy based routing to route traffic sourced from 192.168.1.x out WAN 2's gateway.

There are several things happening here:

Assuming you did not make a typo on the LAN subnet in your diagram, you will need to re-do your subnets slightly and make some changes:

Your transit network (listed as 10.0.100.0/16) is entirely too large.  Also, as currently configured, your transit network is actually 10.0.0.0/16 right now.  Narrow it down to at least a /24 (you can go as narrow as /30).

How dug in are you on VLAN100?  The path of least resistance with the least amount of changes would be simply changing the subnet of VLAN100.  Or you can keep VLAN100 and modify your transit network.  Another option if you want to keep your IP scheme consistent, is to remove VLAN100 and create a different VLAN (e.g. VLAN90 - 10.0.90.0/24)

Even though you stated that you created a LAN gateway of 10.0.100.2,  your screen shot shows that you actually created a LAN gateway of 10.0.100.1, which is incorrect, but moot anyway because it wasn't going to work regardless because of your transit network being too wide.  Your LAN gateway IP should be the routed port on your switch.

Once you have your subnets figured out, verify that there is a default route on your switch pointing back to PFsense

Finally, add static routes for the rest of your subnets

At this point, assuming you have your DHCP server handing out the VLAN IP as the default gateway for each VLAN, all should be working.  This is exactly how I have my network configured.

Hi

Yes, both DSL are PPPoE

Thanks

Don't know. Draw a diagram and it should be pretty obvious what you need to route where and what you need to pass where.

www.gliffy.com

Good idea Will take your advice :)

I have a multi-WAN setup and the graphs are fine - you could try simply unplugging one WAN connection at a time and see what happens assuming that you have pfSense configured to switch defaults when one goes off-line.  Does the system up the UP WAN and ignore the DOWN WAN each time?  Depending on what happens, that might tell you where to start looking.

My guess is that the graphs are telling you that you have a configuration issue - multi-wan can be complex.

Kapara, that is the dilemma. I have a pretty good understanding of the current configuration because it is a bit simpler without the redundancy. I need to take it a step at a time. I've already read of the problems with static routes and multi-wan gateway groups.. I will have another go at it in the near future. Like I said I almost had it 100% after making the new firewall rules to replace the static rout functionality. I am a little confused though why I could communicate with my remote 192.168.104.x network afterwards but not immediately the remote VoIP subnet without adding additional firewall rules even though those networks were listed in the openVPN remote network config. Any thoughts?

:)
glad you got i sorted

Agree that it would be awesome if this was fixed. There  isn't much point in running multi-wan on pfsense with or without squid right now as there are issues with apinger, and pfsense doesn't recovery well after links are restored.

It doesn't work on sophos either when running squid which is why I switched pfsense.

My best guess from that description is maybe the rule doesn't match on the subsequent attempt. Like if it's to a service that's distributed across multiple IPs and your rule only matches one or some of those. What's it using to upload?