@Gblenn

These rules weren't added on both my pfSense box installations, I'm aware what masquerade is and does from working with iptables, however I don't see any masquerade option within pfSense, and definitely no rules were added automatically to help me allievate this situation. I'm not even sure what the automatic rule should look like but I do have a bunch of them on each installation.

@kukoarmas Thanks a bundle for this post! Putting a gateway on the WAN static IP interface fixed the issue I was having with asymmetrical traffic! The other WAN interface is DHCP, so no need to put a gateway, it gets one by itself.
I searched for days for a resolution and this post finally helped me understand what was happening and how to fix it.

@Ascar
Then the rule might be wrong anyhow, so that it doesn't match.

@ltechnology said in Dedicated Business Fiber Internet and Netgate:

Thanks for the added information.
I think what I did not communicate properly is that I don't want to put a 2nd piece of equipment (or use their provided router) as I have in the past. I have traditionally opted to purchase a Cisco router, configure and put this in place as the intermediate piece instead of giving ATT any extra money.

What I am hoping to do is integrate the functionality and routing of this Cisco piece into the front end of the Negate, eliminating the hardware.

I think that one of two options might work:

Set up the WAN port with the serial IP, and set the LAN IP (from ATT) to a virtual IP. Route the virtual IP to the LAN on configured on the Netgate.

Utilize the virtual layers now available, set up 1 virtual to handle the ATT portion, and then have it hand off to the 2nd virtual which would be my normal Netgate firewall./router configuration, using the 1st virtual as my WAN interface.

I know they are using a Fortinet Fortigate router for the provided equipment now and doing this all in that unit - so it should be workable on the Netgate as well. Strangely noone has done this - or has taken the time to post and share their efforts.

Thanks

OK I understand now.

I've never done this. I think the challenge is that the upstream device you will be directly connecting your Netgate WAN port to is in a different subnet than the static addresses you are allowed to use. You'll probably need a Netgate with an SFP port, since you'll be attaching directly to fiber (or a media converter). I can't really test this, so I can only suggest things in a "spitballing it" kind of way.

One should note that this may not be possible without getting AT&T to change some things.

First Option - Use the AT&T "WAN Information" instead of the LAN information.

Set your Netgate's WAN IP to 32.xxx.xx.224/30. Use a gateway address of 32.xxx.xx.225/30 (or possibly 226/30. I'm making an assumption about the IP of the gateway directly upstream here. It could be either one, I merely suspect it's 225/30).

Test the connection.

If AT&T does any vlan tagging or MAC address filtering / authentication, this may fail right out of the box. If this is the case, you'd need AT&T to give you a mac address that is authenticated that you could spoof on your Netgate's WAN port (not impossible you could get the address right off the AT&T Cisco router as well) and what the vlan tagging configuration is, if any.

You state you have previously purchased your own Cisco router and configured this. If that's the case, you may know the vlan tagging configuration already and know how to deal with possible mac authentication issues if they even exist.

AT&T may also expect packets to originate from you LAN block (12.xxx.xx.129/30) rather than from AT&T's own WAN block (32.xxx.xx.224/30). AT&T's equipment might reject this traffic without them changing the configuration of the upstream devices.

Downsides to this method

you may need to get AT&T to change things on their end for this to work you will need to be able to directly accept a fiber connection (SFP port should work here) you don't have a guarantee that the SFP port on the Netgate will be compatible with the SFP module connecting fiber to your WAN interface (it's likely it will be, but it's not assured. If it isn't you might need a media converter to convert fiber to ethernet/rj45 and use an ethernet port for WAN). You only get one public facing IP address instead your full block of 5.

Second Option - Configure your WAN gateway to allow non-local gateway addressing.

This is similar to your previous proposal about using 32.xxx.xx.129/30 as your gateway and using 12.xxx.xx.130/29 as your WAN interface IP address.

Configure a Gateway in System -> Routing. It's IP address is 32.xxx.xx.225/30. Before saving the Gateway, click Display Advanced. At the very bottom, check the box labeled "Use non-local gateway through interface specific route." Save and Apply Changes.

Configure WAN interface to use IP address 12.xxx.xx.130/29. Select the gateway you just created for the IPv4 upstream gateway. Because you checked the box allowing a non-local gateway, this setting will be permitted as valid whereas in a default setup it would not.

This may still fail for similar reasons as the first method, relating to mac address authentication, vlan tagging issues, and source IP address issues. This is also a very non-standard configuration. 99% of the time if the gateway for your WAN interface is in a different subnet than your WAN interface's IP, it's a misconfiguration. There are a few legit instances where it's needed, it could possibly work here.

I'm not sure which I'd attempt first for certain, I might actually lean slightly towards the first method if you are good with a single WAN IP address on your Netgate. If you need to use multiple addresses in the block, it basically pushes you into the second method.

A more ideal solution might be if AT&T makes the device that is typically upstream from their Cisco router to assign 12.xxx.xx.129/30 as the inward facing IP address (instead of 32.xxx.xx.224/30). Then you'd have your full block of 5 IP addresses and a very standard WAN configuration on your Netgate. Since AT&T doesnt do it this way, I presume there are reasons why they don't, so it may not be possible or something they'd be willing to change.

@testing123 This scenario is quite well documented, here: https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-s2s-route-internet-traffic.html
In particular, check out the outbound rules for the phone.

Your policy rule looks good, and should take care of all traffic from that device.
Are you sure that the phone actually has that IP? Phones these days randomize their MAC unless you turn that off in the phone. So it may have a different IP the next time it connects...

Then at the UDM side you must make sure the rules allow for internet access, and back. Assuming that is what you want.. basically making your phone appear as if it was located at your parents place?

No worries. I remember hitting that issue myself when I first setup pfSense with multiwan. Too long ago to mention!

Until you realise how pfSense determines what is a WAN interface and what that triggers it can easily seem like magic. 😉

@IgiveUp Netflix works without IPv6 for sure, but if your TV (Netflix) somehow get's a DNS resolution when it tries IPv6, and no traffic, then it fails. Try turning IPv6 off in the TV, and then check that it is completely off in pfsense.
System > Advanced > Networking
System > Routing Gateway (None for IPv6)
WAN and LAN interfaces (IPv6 Configuration Typ - None)
Services > IPv6 Relay disabled

@IgiveUp said in 2 wan 2 lan, send lan2 traffic out non default gateway of wan2:

@Gblenn problem is that when you create policy routing through non default gateway the traffic seems to be rerouted from nondefault gateway out the default one.

Could you show your rules perhaps? It may be easier to see what might be wrong since there is no reason it shouldn't work. Do you have an floating rules?

Hi @viragomann,

thanx for your answer.

Finally, it was a problem on the other end and my configuration was OK.

I didn't have to change anything.

@viragomann said in Double WAN and access device from the outside:

According this rule you want to allow access to 192.168.X.X in fact. So you need to state this in the destination of the pass rule as well.
However, you can also let the NAT rule add an associated pass rule.

thank you so much! this has helped me resolve my problem, i had in fact misunderstood how things were working but i've changed my NAT rule this way :

Interface : WAN-W
Protocol : TCP
Source address : * [then restrained it]
Source Ports : *
Dest. Address : WAN-W address
Dest. Ports : 8296
NAT IP : 192.168.X.X (my device lan ip address)
NAT Ports : 8296

@viragomann said in Double WAN and access device from the outside:

associated pass rule

Fact is it couldn't reach because the device's server simply wasn't on the port 80 ahaha.
I've also added this instead of trying to do it manually and i can now access my device on port 8296 with the redirection of me reaching the WAN-W:8296.

thank you again :))

@viragomann Thank you so much. I am grateful you pointed it out, this would have killed me with how much it was staring me in the face.
I had the wrong config on the Satellite. I was allowing access to its own LAN through its VPN.

Satellite config allowing for LAN access to itself
8c144893-fa55-4da5-9136-c988963e2219-image.png

HQ config allowing the correct, remote network access
1167428e-4dce-4ade-ab54-45f61836e0ba-image.png

@karan293474

And what is opnsense ?

Policy routing started behaving after I changed my WAN ipv6 configuration from "none" to "6to4 tunnel".
I have strange feeling that redirected hist will be able to escape redirection if remote hist is resolving to ipv6 address though.

@oznet Local traffic (same subnet) does not go to the firewall. If you have other networks then you would need to allow them as well.

@johnpoz said in datatransfer rate not as high as it should be:

So this was never about file copy speed, but only iperf tests?

I think it was actually about file transfers as well...
Here's from one of the earler posts..

@pleaseHELP said in datatransfer rate not as high as it should be:

I've noticed that sending data to that TrueNAS is always capped at 80 Mb/s, but getting data from it runs on about 1Gbit/s, as it also should for upload. This isn't just a TrueNAS thing.

@Gblenn said in Strange behaviour with pfSense and 4G:

There is no way any outside IP can ping your local devices, either through your ISP router or pfsense. Your mobile ISP would have no clue where to route such a request...

I agree with you, because these are non routable IP adresses.

I think there is only one explanation... that the phone is actually connected to the 172.16.10.1/24 network.
Either you have some VPN/Tailscale running which can connect it to your LAN. Or it is in fact actually on wifi although you think it's off.

Even though I didn't detect anything, this seems the most logical.

Or perhaps another explanation is that the mobile ISP is using 172.16.NN subnet for their mobile clients and you are pinging someone elses phone or tablet... Check your IP on the phone...

This hypothesis is not possible because I could ping only my devices. And I just checked again, my phone IP is on a different range : 92.184.x.y

I didn't indicate it, because I uninstalled the server in the meantime, but I did some tests with OpenVPN. Even if it seems impossible to me, it gives the impression of a persistence of the tunnel

Anyway, I just checked and everything is back to normal.

Thank you for looking into this situation !

@michmoor I have a pile of /32 routes I may try the xml file. Thnx.

I should add, if I add an entry into my computer's hosts file to directly resolve to the internal IP of the NPM server, the loading is very fast, so it's not an issue with the Nginx server or the NPM in front of it. This makes me think that maybe the solution is to put manual entries into PF's resolver, but it still bothers me that it would only happen on one set of sites. Also, I am trying to keep VLAN 10 completely isolated from the rest of the VLANs, so having to resolve to it goes against what I'm trying to accomplish.

@khb FWIW, I ran speedtest-go with the default options every 15min for a couple of days.

avg stdev. min max download 128.3 49.5 40.3 309 upload. 11.6 5.4 5.7 33

The test was executed from the netgate, using only the starlink linked interface. A few sanity checks running the starlink native (naive, not advanced) test within a minute or so of the speedtest runs varied (sometimes matching a result from 15m before or after). The variability is large, which I expected, but varies faster than I'd have guessed.