@conejero Easiest way is cat /tmp/rules.debug for the most human-friendly version. You can also pfctl -vvsr to see the raw rule set but it is broken down a lot more that way. It's generally not necessary to look there to get a picture of the rules and what they are doing.

If anyone else has this problem, I ended up creating another firewall rule under the wireguard interface. I passed anything NOT RFC1918 to the Nord Gateway then created an outbound NAT rule on the NORDVPN interface. Works Now.

@russm Great - and I for sure learned something about issues that can come about with zen, there is actually another active thread which I think is the same problem

@gwaitsi said in PureNAT / NAT + Proxy - Am i doing it right?:

seems to be working.

Then don't fix it ;)

@eliteharbinger42 first of all I'm confused - are you using two routers? If so, is the server located on the 192.168.1.0 -network?

Just remove the USG and put the server on the same network as PFsense and you'll be able to ping.

@mzabik not possible to set a time config (unless maybe you fiddle with scripts in the OS but that's unsupported and probably hard to do) but you have options for "high latency" and "packet loss" that should satisfy your needs. Try using packet loss instead of member down if that's the way it's set up now.

Otherwise the short answer is get a better WAN.

@viragomann said in Got openvpn working, but cannot make it work for one VLAN only:

@newsboost said in Got openvpn working, but cannot make it work for one VLAN only:

I don't think they send requests to public DNS servers directly. I think they send requests to pfSense, which either replies back directly - or asks Cloudflare upstream with DNS-over-HTTPS - which is also what I think I wanted in general - except maybe it would be nice to make an exception such that VPN-traffic on VLAN 10 also uses the DNS-servers from the VPN-provider. Creating this exception rule seems not so easy, at least I haven't succeeded yet. Here are my rules for VLAN 10:

Again, all you have to do for enabling this is adding a port-forwarding rule for DNS requests to a public server. Might be the VPN providers DNS or Cloudflare or whatever.

Okay I did as you described - but with one exception (and I also screwed this up the first time by letting "interface" be the OpenVPN-interface, instead it should just be VLAN-10). This is what I ended up with:

84bb308d-e310-4bd7-b9fe-802babacada8-image.png

This will accomplish what I want: Use secure Cloudflare DNS, except when using VPN - then use DNS through the VPN tunnel. But something weird happened. First, what I hoped to achieve was that I shouldn't hardcode the DNS-server, for OpenVPN it should use the DNS servers provided by the VPN-provider, that's why I checked "Pull DNS" in the OpenVPN-client config, please see (the bottom checkbox):

6921612c-f78d-429c-a506-a6e7ff094d5f-image.png

I tried to explain this several time above and posted my NAT rule as an example. So read again, please.
There is no exception needed and no further special rules.

Thank you, I read it again. As I understand it, you use plain unencrypted DNS and the same for all interfaces, meaning your ISP can spy on your DNS-requests.

The reason I've been slow at doing what you propose is because I wanted (want) an exception such that everything going through the VPN tunnel uses DNS from the VPN provider. Since the tunnel is encrypted, so will DNS-requests be, at least from the point of view of my ISP - on the VPN-side DNS will be unencrypted, but non-personal because many users will share the same VPN-server and unless you're a (or I am a) top wanted criminal, I'll have privacy and anonymity via VPN. So I wanted/want an exception: Cloudfare by default, but whatever DNS comes with the VPN on the other side of the tunnel. I think actually I've accomplished that using the NAT / Port forward I showed a screenshot of in the top of this reply/post. But I wasn't expecting what I see: As you see, I entered the Google DNS: 8.8.8.8 - so I expected that when I ran the online DNS-leak-tests, they would show Google. But that's not what happens.... With the setup above (NAT / Port forward in the top screenshot for interface VLAN10_UK_VPN) DNS leak tests show that my DNS servers are:

1 "UK Dedicated Servers Ltd" IP address + a few "Cogent Communications" IP addresses - (they're all London, UK)

I then ran a test: On my normal network (not via VPN) I ran the "ExpressVPN"-app and then ran the DNS leak tests. They revealed my DNS servers were:

1 "Heficed" IP address + a few "Clouvider Limited" IP addresses (also either London or geographically close to London)

I also wouldn't have liked to use Google DNS in the first place. I checked the firewall log for 8.8.8.8 (since it didn't show up in the DNS leak tests) - but that hasn't been blocked. I have a theory that instead it might've been redirected - or I'm just dumb and stupid. But then I thought about the great neat trick you learned me with /tmp/rules.debug - I downloaded it and filtered all lines with port 53 in it, it looks like this:

1f82401d-0d85-49da-97f1-4aea55db8d5b-image.png

It's great that on VLAN 10, it now picks up other DNS-servers than cloudflare - so I guess I've accomplished what I want. I just don't understand why it's not using google (not that I dislike it, would just be nice to understand it 😆 ). But as I've mentioned earlier: I'm already extremely happy with my setup so I can live with not understanding it all. Thanks again for your patience, I know I'm a bit slow at learning/understanding sometimes, I'm grateful for having learned a lot already 😄

@mmiller7 Hello, thank you for your answer. I think I'll do like you while waiting for an update that will fix this problem. I wish you a good day.

@manicmoose That option is responsible for killing states when a particular WAN interface's IP address changes (via DHCP, for example). The option is to kill ALL states when this happens, instead of just those on the old WAN IP. This has absolutely nothing to do with WAN failover, since in that case, the interface IP addresses don't change, just which one is being used for routing.

Another option which seems helpful is the "Flush all states when a gateway goes down" option on the System->Advanced->Miscellaneous tab. However, this is what enables the failover, but not the failback (i.e. this won't do anything when a down gateway comes up).

This issue remains, and I've been correcting it for a few years now using this script (more or less).

@falassion DHCP not renewing properly.
I would run a packetcapture on the WAN on UDP port 68 (you can leave this running long-term if you limit the protocol and port number, and change the capture size from 100 to 0) and then see what results.

You can download the file after you stop it to view in Wireshark.

@kitdavis Hey, Kit, if your backup has a smaller MTU (e.g. DSL) make sure to set MTU/MSS on the interface. I ran into this with a TMobile 4G Home Internet box which has a 1420-MTU due to their 464XLAT encapsulation, and it straight-up broke WiFi calling not having the MTU set low enough.
I thought it was pfSense due to several messages here, but when helping out a friend with pfSense vanilla everything on a Spectrum cable connection it worked perfectly so I took a closer look.
2.6.0 in both locations.

¡Good day everyone!

I got support from a AWS TAM, he helped me pointing that a new instance w/ no config or packages get this kind of strange routes. The key was that pfSense is getting dns server updates via dhcp so pfSense add this bogus route whenever it gets an interface dhcp message.

@elrick75 Personally I would try changing these settings:

Unless you are forced by the ISP to use 178.250.208.135/209.34, change to 8.8.8.8/8.8.4.4 or 208.67.222.222/208.67.220.220 Do not assign a Gateway to the DNS servers in System -> General (set to "none") Uncheck "Allow DNS server list to be overridden by DHCP/PPP on WAN" In your System -> Routing, choose an IP address that is NOT one of your DNS servers as the Monitor IP. If you are not sure what IP to use, run a traceroute and choose a nearby hop that responds to ping, or you can try my hopfinder script to auto-detect the best one to use. Make sure the firewall rule on your LAN interfaces does not specify a specific gateway (under Advanced options) or if it does, make sure it is set to use a Gateway Group)

Once you have done all of that, re-try your test...

I find why....
I had set Group Gateway as defaut gateway on OpenVPN rules too, i switch it to default (= * ) and all is working fine now.

f148e2dd-73bf-4092-9bc5-e73db3ccf6b4-image.png

I have another specific question.

Before switching to Dual Wan, I was able to connect from my home in VPN to my home :)
That is to say from the VLAN MY LAN I connect with VPN to the VLAN_DMZ, which was convenient for security reasons and to manage VLAN_DMZ.
Even if I was still depending on my ISP to access VLAN_DMZ, I didn't have to have a machine on the VLAN_DMZ at home.

Since I am in DUAL Wan, I am not able to do this anymore.
When trying to connect from my MY LAN VLAN, i receive TLS Error, OpenVPN client log :

Sun May 15 10:54:05 2022 Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication Sun May 15 10:54:05 2022 Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication Sun May 15 10:54:05 2022 MANAGEMENT: >STATE:1652604845,RESOLVE,,,,,, Sun May 15 10:54:05 2022 TCP/UDP: Preserving recently used remote address: [AF_INET]185.XXX.XXX.XXX:53XXX Sun May 15 10:54:05 2022 Socket Buffers: R=[65536->65536] S=[64512->64512] Sun May 15 10:54:05 2022 UDPv4 link local: (not bound) Sun May 15 10:54:05 2022 UDPv4 link remote: [AF_INET]185.XXX.XXX.XXX:53XXX Sun May 15 10:54:05 2022 MANAGEMENT: >STATE:1652604845,WAIT,,,,,, Sun May 15 10:55:05 2022 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity) Sun May 15 10:55:05 2022 TLS Error: TLS handshake failed] Sun May 15 10:55:05 2022 SIGUSR1[soft,tls-error] received, process restarting Sun May 15 10:55:05 2022 MANAGEMENT: >STATE:1652604905,RECONNECTING,tls-error,,,,,

Is there a specific parameter in the OpenVPN configuration file that could solve this?
Do you know if it is possible to solve this problem?

Thank you.

@steveits each 2 pfsense are connected with a patch cable and we configured the setting in pfsync Peer IP ... but still the issue occurred

@ace-1 It is easy. 😉

Capture.PNG
For instance, your rule there has a destination of any (everything) and it has a gateway set, which means, everything has to go through that gateway out to the internet, so no chance for you to connect to IoT anymore.

@reberhar I finally bit the bullet and waited until late at night and just did a reboot and that seems to have fixed it.

@viragomann I did restart pfsense, nothing changed. I'll try capturing packet when I get home from school.

@krbvroc1
Hello.
You can use option SYSTEM - ROUTING - GATEWAYS - clic ADD - clic DISPLAY ADVANCED - check "Use non-local gateway"
Regards.

@johnpoz said in Wake-on-LAN using directed broadcast:

Can you get your port 9 forwarding thing to work, if you use a directed IP and then static arp on pfsense for that specific IP?

I just managed to get it working using the aforementioned workaround. I guess that the ARP entry has to be static. Otherwise the final router would probably not forward the magic packet to the destination interface.

@johnpoz said in Wake-on-LAN using directed broadcast:

It could be also that sending to say 192.168.1.100 with all ffs as mac, might not be enough for that client to wake up? Maybe it wants to see its IP and mac in the WOL packet?

It is enough that the packet contains the right MAC address and gets spewed out of the right interface. The IP address serves just for routing the packet through the network to the final router.

@luckman212 Super, thank you. I will have to get my head around some of that, but it's the basis to get the 2nd line installed so I can even test it.

@professor your setup is a MESS.. Yeah your going to have problem with it, its a asymmetrical nightmare..

Fix it! Use a transit network - now you can just use pfsense as a router with a any any rule..

Fixed the issue: I had to enable Filter IPsec VTI and Transport on assigned interfaces, block all tunnel mode traffic under VPN -> IPsec -> Advanced Settings on fwl-01 and fwl-03

Documentation here:
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/advanced.html#advanced-ipsec-settings

@akuma1x Thank you! It's working!

This was fixed by deleting all manually added GWs and using the one assigned by the upstream DHCP as default. the DNS-Redirect rule does not seem to have any effect (enabling/disabling it does not affect connectivity).
Thank you guys for helping out ^_^