@laurens-DS said in HA Setup:

The problem was I had WAN2 set up but nothing stuck in yet because I don't have a 2nd provider right now

That is not the classic HA from the documentation. What you're want to do is HA with Multi-WAN.
Have a read through Netgate doc: High Availability Configuration Example with Multi-WAN.

@viragomann
Hello,

Thank you for your help. I changed the default routing to create an additional static route for this unique IP, in order to replace the subnet route. And for accesses that require it, I create policy-based rules.

Have a very good day.

I finally got Ping working in Windows. Had to accept ANY source for Remote Address in Windows Defender Firewall for Private.Public Profile.
And I am getting sub ms response times from the Pi to Windows (~0.56ms). So the route seems to be direct without any detours.

Traceroute still fails, but that could be the ISP modem/router not allowing it.

So, it appears ~75Mb/s is the best I can expect. 5x faster than before!!!

Thanks Everyone!

P.S. ICMP also needed to be added to the Firewall Rules in pfSense on the WAN interface to allow Pings through

@patient0 Thank you

@ddbnj Awesome!

@drmarian0
Yes, the rule should work.

Ensure that the policy routing rule on pfSense is applied. Is it configured for any protocol? If it's TCP only ping will not work.
Enable logging, then try to access a public IP and check the log
after.

Or run a packet capture on OPNsense on the WG interface to verify that the upstream traffic is routed over the VPN.

@MrHedgehog With any luck, this will be fixed in the next release of Plus and CE:

Redmine: https://redmine.pfsense.org/issues/16103

Meanwhile, anyone who continues to experience this problem can manually patch /usr/local/bin/ppp-linkdown.

@viragomann said in Multi WAN and multiple gateway issue:

The proper rerouting is controlled by the reply-to tag. Did you disable it in System > Advanced > Firewall & NAT or in the rule by any chance?

Not disabled.

I didn't look closely enough when reviewing the state tables to see if WAN2 was referenced when WAN1 should have been.

Hopefully it never happens again, but I have some things to look into if I ever come across this again. Thanks for discussing it with me!

@michmoor

Yeah could be broken in many ways. I haven't used it for anything else. Thankfully I got another spare port.

@hillblock
The problem in this thread is that the VPN endpoint is not the default gateway. In this case an outbound NAT rule enables you to access the local network.
But the NAT has no impact on accessing the web GUI of pfSense, since this traffic doesn't doesn't go out on an interface.

@AlcMat
Sniff the traffic to see if the masquerading rule works properly.

If it's fine that's all you can do on pfSense. Then there might be something wrong on the Windows machine.

@Gblenn

These rules weren't added on both my pfSense box installations, I'm aware what masquerade is and does from working with iptables, however I don't see any masquerade option within pfSense, and definitely no rules were added automatically to help me allievate this situation. I'm not even sure what the automatic rule should look like but I do have a bunch of them on each installation.

@kukoarmas Thanks a bundle for this post! Putting a gateway on the WAN static IP interface fixed the issue I was having with asymmetrical traffic! The other WAN interface is DHCP, so no need to put a gateway, it gets one by itself.
I searched for days for a resolution and this post finally helped me understand what was happening and how to fix it.

@Ascar
Then the rule might be wrong anyhow, so that it doesn't match.

@ltechnology said in Dedicated Business Fiber Internet and Netgate:

Thanks for the added information.
I think what I did not communicate properly is that I don't want to put a 2nd piece of equipment (or use their provided router) as I have in the past. I have traditionally opted to purchase a Cisco router, configure and put this in place as the intermediate piece instead of giving ATT any extra money.

What I am hoping to do is integrate the functionality and routing of this Cisco piece into the front end of the Negate, eliminating the hardware.

I think that one of two options might work:

Set up the WAN port with the serial IP, and set the LAN IP (from ATT) to a virtual IP. Route the virtual IP to the LAN on configured on the Netgate.

Utilize the virtual layers now available, set up 1 virtual to handle the ATT portion, and then have it hand off to the 2nd virtual which would be my normal Netgate firewall./router configuration, using the 1st virtual as my WAN interface.

I know they are using a Fortinet Fortigate router for the provided equipment now and doing this all in that unit - so it should be workable on the Netgate as well. Strangely noone has done this - or has taken the time to post and share their efforts.

Thanks

OK I understand now.

I've never done this. I think the challenge is that the upstream device you will be directly connecting your Netgate WAN port to is in a different subnet than the static addresses you are allowed to use. You'll probably need a Netgate with an SFP port, since you'll be attaching directly to fiber (or a media converter). I can't really test this, so I can only suggest things in a "spitballing it" kind of way.

One should note that this may not be possible without getting AT&T to change some things.

First Option - Use the AT&T "WAN Information" instead of the LAN information.

Set your Netgate's WAN IP to 32.xxx.xx.224/30. Use a gateway address of 32.xxx.xx.225/30 (or possibly 226/30. I'm making an assumption about the IP of the gateway directly upstream here. It could be either one, I merely suspect it's 225/30).

Test the connection.

If AT&T does any vlan tagging or MAC address filtering / authentication, this may fail right out of the box. If this is the case, you'd need AT&T to give you a mac address that is authenticated that you could spoof on your Netgate's WAN port (not impossible you could get the address right off the AT&T Cisco router as well) and what the vlan tagging configuration is, if any.

You state you have previously purchased your own Cisco router and configured this. If that's the case, you may know the vlan tagging configuration already and know how to deal with possible mac authentication issues if they even exist.

AT&T may also expect packets to originate from you LAN block (12.xxx.xx.129/30) rather than from AT&T's own WAN block (32.xxx.xx.224/30). AT&T's equipment might reject this traffic without them changing the configuration of the upstream devices.

Downsides to this method

you may need to get AT&T to change things on their end for this to work you will need to be able to directly accept a fiber connection (SFP port should work here) you don't have a guarantee that the SFP port on the Netgate will be compatible with the SFP module connecting fiber to your WAN interface (it's likely it will be, but it's not assured. If it isn't you might need a media converter to convert fiber to ethernet/rj45 and use an ethernet port for WAN). You only get one public facing IP address instead your full block of 5.

Second Option - Configure your WAN gateway to allow non-local gateway addressing.

This is similar to your previous proposal about using 32.xxx.xx.129/30 as your gateway and using 12.xxx.xx.130/29 as your WAN interface IP address.

Configure a Gateway in System -> Routing. It's IP address is 32.xxx.xx.225/30. Before saving the Gateway, click Display Advanced. At the very bottom, check the box labeled "Use non-local gateway through interface specific route." Save and Apply Changes.

Configure WAN interface to use IP address 12.xxx.xx.130/29. Select the gateway you just created for the IPv4 upstream gateway. Because you checked the box allowing a non-local gateway, this setting will be permitted as valid whereas in a default setup it would not.

This may still fail for similar reasons as the first method, relating to mac address authentication, vlan tagging issues, and source IP address issues. This is also a very non-standard configuration. 99% of the time if the gateway for your WAN interface is in a different subnet than your WAN interface's IP, it's a misconfiguration. There are a few legit instances where it's needed, it could possibly work here.

I'm not sure which I'd attempt first for certain, I might actually lean slightly towards the first method if you are good with a single WAN IP address on your Netgate. If you need to use multiple addresses in the block, it basically pushes you into the second method.

A more ideal solution might be if AT&T makes the device that is typically upstream from their Cisco router to assign 12.xxx.xx.129/30 as the inward facing IP address (instead of 32.xxx.xx.224/30). Then you'd have your full block of 5 IP addresses and a very standard WAN configuration on your Netgate. Since AT&T doesnt do it this way, I presume there are reasons why they don't, so it may not be possible or something they'd be willing to change.

@testing123 This scenario is quite well documented, here: https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-s2s-route-internet-traffic.html
In particular, check out the outbound rules for the phone.

Your policy rule looks good, and should take care of all traffic from that device.
Are you sure that the phone actually has that IP? Phones these days randomize their MAC unless you turn that off in the phone. So it may have a different IP the next time it connects...

Then at the UDM side you must make sure the rules allow for internet access, and back. Assuming that is what you want.. basically making your phone appear as if it was located at your parents place?

No worries. I remember hitting that issue myself when I first setup pfSense with multiwan. Too long ago to mention!

Until you realise how pfSense determines what is a WAN interface and what that triggers it can easily seem like magic. 😉

@IgiveUp Netflix works without IPv6 for sure, but if your TV (Netflix) somehow get's a DNS resolution when it tries IPv6, and no traffic, then it fails. Try turning IPv6 off in the TV, and then check that it is completely off in pfsense.
System > Advanced > Networking
System > Routing Gateway (None for IPv6)
WAN and LAN interfaces (IPv6 Configuration Typ - None)
Services > IPv6 Relay disabled

@IgiveUp said in 2 wan 2 lan, send lan2 traffic out non default gateway of wan2:

@Gblenn problem is that when you create policy routing through non default gateway the traffic seems to be rerouted from nondefault gateway out the default one.

Could you show your rules perhaps? It may be easier to see what might be wrong since there is no reason it shouldn't work. Do you have an floating rules?