@SteveITS

Right now, I just switched back to bridge mode for the VM in Fusion. I assigned it a static dhcp address in pf and it has that iP address. It's in the same subnet as the host.

I have an entry for it's IP address in the host overrides section of DNS resolver in pf.

So now I can ping it via IP address and host name from anywhere on my internal subnets.

I installed gufw on the VM and set it to allow all incoming.

I have pf fw nat rules set up to allow requests to the VM's IP address from WAN address on 80 and 443 to pass.

Still, when I try to run ./discourse-setup it tells me '443 does not appear to be accessible using hostname'

@tinfoilmatt I think OP has done https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/opt-wan.html. Or should have.

@atlassol Your second pic has them with the same IP …? That’s not going to work on any configuration.

@kj32

User error. I found some old notes that included this observation:

"Lan gateway should be defined under System | Routing, not interface."

Removed the spurious definition under interface, and now it works again.

I am hoping this fairly ancient -5100 appliance holds on. It's been through a lot including several dead cable modems. However, physical intervention has to wait until my next in-person visit. I only have non-technical people on site (as indicated by the hard power off recovery).

The site is in an area with buried cable and a high water table which is a recipe for disaster (eats a cable modem every 18 months) and explains the starlink back-up. We use tailscale so remote access is fairly tolerant of CGNAT and our windows DC / local users can phone home and network admin can remote in. When the comcast circuit is down I lose remote admin via the fqdn/public IP which is stressful as I'm reliant on tailscale coming up using starlink.

As a first order trouble shooting step I'm stepping up a local VM as a tailscale client to give myself a back door if the netgate box becomes unreachable again and potentially to automatically attempt a pfsense reboot in the event of a sustained loss of connectivity.

I will experiment with turning off gateway monitoring after that and watch for physical events. Next time I'm on site I'm going to insert a fiber media converter between the cable modem and the netgate (or it's successor if I can get the budget for it) to remove physical plug events from cable modem reboots and risk of electrical shocks on the wan port.

@netblues And UPnP is also port forward.. just automagic. But as I said, never got it to work behind private IP using STUN. There is a feature request active to get a setting to allow UPnP to accept WAN with private IP though...

If the OP needs remote access or host services, they should be using a VPN like Tailscale, which will traverse any level of NAT, including CGNAT

@laurens-DS said in HA Setup:

The problem was I had WAN2 set up but nothing stuck in yet because I don't have a 2nd provider right now

That is not the classic HA from the documentation. What you're want to do is HA with Multi-WAN.
Have a read through Netgate doc: High Availability Configuration Example with Multi-WAN.

@siegmarb said in dpinger not reliable - ping request/replies:

Apr 8 07:07:12 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 27627us stddev 14386us loss 21%
Apr 9 11:07:14 dpinger 89446 GW_KD_DH 1.1.1.1: Clear latency 22638us stddev 53358us loss 5%
Apr 15 01:00:42 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 21509us stddev 5210us loss 22%
Apr 15 01:06:52 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 1293341us stddev 881246us loss 95%
Apr 15 01:07:07 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 243671us stddev 600909us loss 70%
Apr 15 01:07:48 dpinger 89446 GW_KD_DH 1.1.1.1: Clear latency 93734us stddev 349188us loss 5%
Apr 15 06:30:21 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 28365us stddev 7087us loss 21%
Apr 15 06:34:35 dpinger 89446 GW_KD_DH 1.1.1.1: Clear latency 28547us stddev 86447us loss 5%
Apr 16 10:44:07 dpinger 89446 GW_KD_DH 1.1.1.1: Alarm latency 28632us stddev 35032us loss 22%

dpinger not reliable - ping request/replies

You can remove the word 'not'. 😊
Test for yourself : Go here : Diagnostics > Packet Capture
and select (Capture Options) your WAN interface,
Set "View Options" to High,
Set PROTOCOL to PING, and ETHERTYPE to IPv4.
Hit the green Start.

From now on, you'll see that "ICMP echo requests" are send. It's the dpinger process that pings ^^
These "ICMP echo requests" are send to an upstream gateway (you'll see the IP in the capture also) and if all goes well, and answer "ICMP echo reply" comes back.
The duration between the moment a packet was send and the answer comes back is known as :

89ddd9ad-0612-4ee8-afa5-c61c05cdd4cb-image.png

You'll se the avarage time it took, and te variation.
The simple fact that packets did come back is n enough to mark the interface as "Online".

So, now you know dpinger is reliable ^^
Less reliable is probably your connection, as you've shown yourself : ICMP packets are (always) send, but not all come back. That said, a couple over several days ... that not that bad.
Or : maybe the gateway to where the packets where send to was very busy and missed a packet, so it didn't reply back.

Be aware that the ICMP packets have less priority as other TCP or UDP packets, so if a ICMP gets discarded, then that's not the end of the world. It can happen.
If your connection is saturated, then its normal that you see that a ICMP packet didn't make it back.

@viragomann
Hello,

Thank you for your help. I changed the default routing to create an additional static route for this unique IP, in order to replace the subnet route. And for accesses that require it, I create policy-based rules.

Have a very good day.

I finally got Ping working in Windows. Had to accept ANY source for Remote Address in Windows Defender Firewall for Private.Public Profile.
And I am getting sub ms response times from the Pi to Windows (~0.56ms). So the route seems to be direct without any detours.

Traceroute still fails, but that could be the ISP modem/router not allowing it.

So, it appears ~75Mb/s is the best I can expect. 5x faster than before!!!

Thanks Everyone!

P.S. ICMP also needed to be added to the Firewall Rules in pfSense on the WAN interface to allow Pings through

@patient0 Thank you

@ddbnj Awesome!

@drmarian0
Yes, the rule should work.

Ensure that the policy routing rule on pfSense is applied. Is it configured for any protocol? If it's TCP only ping will not work.
Enable logging, then try to access a public IP and check the log
after.

Or run a packet capture on OPNsense on the WG interface to verify that the upstream traffic is routed over the VPN.

@MrHedgehog With any luck, this will be fixed in the next release of Plus and CE:

Redmine: https://redmine.pfsense.org/issues/16103

Meanwhile, anyone who continues to experience this problem can manually patch /usr/local/bin/ppp-linkdown.

@viragomann said in Multi WAN and multiple gateway issue:

The proper rerouting is controlled by the reply-to tag. Did you disable it in System > Advanced > Firewall & NAT or in the rule by any chance?

Not disabled.

I didn't look closely enough when reviewing the state tables to see if WAN2 was referenced when WAN1 should have been.

Hopefully it never happens again, but I have some things to look into if I ever come across this again. Thanks for discussing it with me!

@michmoor

Yeah could be broken in many ways. I haven't used it for anything else. Thankfully I got another spare port.

@hillblock
The problem in this thread is that the VPN endpoint is not the default gateway. In this case an outbound NAT rule enables you to access the local network.
But the NAT has no impact on accessing the web GUI of pfSense, since this traffic doesn't doesn't go out on an interface.

@AlcMat
Sniff the traffic to see if the masquerading rule works properly.

If it's fine that's all you can do on pfSense. Then there might be something wrong on the Windows machine.