@Derelict Hi, now everything so fine and load balancing over to WAN function as desired. Meaning I get the bandwidth as sum of both WAN. Now I have to questions: If I check "sticky connections" I no longer have the sum of both bandwidth. Its rather randomly once WAN1 or WAN2. Eeven mixed for Upload/Downlaod meaning it may occur that for download it takes WAN1 and for upload WAN2 but never both. It this the expected behavior? How can I load balance the two WAN get both bandwidth added but still redirect all the traffic over VPN-Server (like mullvad, one or more sever)? Best regards Santo

@gwaitsi hmmm. so i set the default gateway to automatic instead of the gateway pool and it seems to have solved the problem. The pf box now defaults over the wan, and the policies are correctly working. so i am happy.

There may be a problem with your setup(?), as it would be quite a problem, if the known (trusted) DNS servers did not respond to the ping and would the provider's CPE restrict you from using ping ??? (this is just an idea why you can stop pinging from a known DNS server, for example, make sure the gateway IP, DNS severs, WAN IP, etc. are in your HOME_NET list / IPS/IDS) We have been using Cloudflare DNS servers (1.1.1.1 / 1.0.0.1) for many - many years for monitor IP purposes, we have never experienced the problem you outlined. Many ISP gateways really do not respond to ping, so a known DNS server is a good solution. Test the best DNS server for you, starting with: https://www.grc.com/dns/benchmark.htm Or use this and try to PING the selected DNS server from a desktop machine for a long time and analyze the values obtained: https://emcosoftware.com/ping-monitor I don't think the multiple - gateway monitor IP is the solution, it would only bring more measurement tasks and results to the system, this is irrelevant here. PS: We have had the experience that sometimes on a self-made (from internet) blocklist, 1.1.1.1 is added to the list of banned IPs, the list is periodicaly updated on the firewall and 1.1.1.1 no longer works. What did your own ISP answer this question? (FRITZ!Box vs. PING issue)

I would examine the rules on your OpenVPN tab and make them explicit otherwise traffic can get matched and sent down a different interface than you're expecting.

Post the remote access config and the site-to-site config for site 2... both located in /var/etc/openvpn

The program that builds the table may not be able to probe that interface since it's special. It also doesn't support IPv6. We're testing out a better method for 2.5.0 (iftop) and at a quick glance there it appears to see IPsec traffic and puts it in the table, at least for VTI. It doesn't work for tunneled IPsec since enc0 doesn't have an IP address on it, and iftop requires that.

@an0nymity said in Communicattion between Subnets with their own Dedicated Interface: I can communicate inter-subnet (Rule created for this) I can communicate from the subnets to the internet (Rule created for this) I cannot communicate from lan-subnet-1 to lan-subnet-2 (Rule created for this) Ok, so going down the list: communication on the SAME subnet doesn't touch the pfsense firewall, it's all done on the switch that your devices are plugged into. You don't need any rules on the firewall to handle this type of traffic. You can witness this "doesn't touch" process by watching the states and traffic on the rules you have made that you think this traffic is going thru. It should say "0/0" for states and traffic. you always have to create rules for getting subnets/networks out to the internet, so plus 1 on that, you must have done it correctly. As an aside, an easy way to do this it to mimic the default LAN rules pfsense creates by itself. on your "lan-subnet-1" to "lan-subnet-2" rule, you layout it like this. Pass all traffic on all ports for source "lan-subnet-1" NET to "lan-subnet-2" NET. Make sure this rule doesn't have any block rules above it that would stop the traffic flow. One final point, many operating systems now BLOCK, by default, traffic coming from other subnets. Even though you wrote a proper allow firewall rule, the hosst you're trying to get to might be blocking the traffic all by itself in it's own firewall settings. Windows 7, 8, and 10 are notorious for doing this. Hope that helps. Jeff

Interesting. it turns out when failover doesn't switch back to tier 1 if I reset filter rules that it will go back. So it doesn't actually matter to have wrong double default gateways when the default lan rules have the gateway set to failover group in advanced. I wonder if it's not updating the variable for my failover group to the proper interface via "route-to" to what I see in the /tmp/rules.debug or is not applying it.

If it's still doesn't work after you disable default rules, you may be running into the issue I have reported here https://forum.netgate.com/topic/153039/dmz-to-multi-wan-over-vpn If your GW is set dynamically most likely it's not available when system boots and your firewall rule will end up just allowing all traffic Check your /tmp/rules.debug it's likely to have something like pass in on { vmxXYZ } $GWWAN1_IPV4 inet from .......... If your GW is not available at the boot time the $GWWAN1_IPV4 will be empty and remain empty even after your WAN1 GW is up. So you would just allow all traffic through and will go through default system GW Easiest way to test if it's the case is to reload the firewall after system up and running without doing any other modification. If it does help, see my post for details, otherwise it's something else.

Not really, though I don't think I've seen this issue in quite a while.

@Rico thank you for your support i will read them

@viragomann said in Routing configuration issue between 3 interfaces on pfsense (New to pfsense): Check that twice to be sure. Than check it again... Your lan rules are by default any any so if you did not mess with that, then any devices on the lan would be able to talk any device on either of your 2 networks with no rules even on those interfaces. So as long as the device in the other vlans is pointing back to pfsense as its gateway.. Its most likely the devices firewall, or other security software on it that you didn't disable.. Simple test can device in nework A ping pfsense IPs you have listed there 10.1.2.1 and 10.1.3.1 from the 10.1.1.0 network.. If so simple do a sniff on pfsense say on network B interface - while you ping something network be at 10.1.2.x -- do you see the ping go out from pfsense.. If so then its not pfsense.. Here example.. My lan rules. [image: 1588973136568-lanrules.jpg] My lan is 192.168.9.0/24, pfsense IP is 192.168.9.253 Another segment of mine (dmz) is 192.168.3.0/24 where pfsense IP in that is 192.168.3.253 I can ping 192.168.3.253 from my 192.168.9.100 box. $ ping 192.168.3.253 Pinging 192.168.3.253 with 32 bytes of data: Reply from 192.168.3.253: bytes=32 time<1ms TTL=64 Reply from 192.168.3.253: bytes=32 time<1ms TTL=64 Here is sniff of that 192.168.3.253 interface only for stuff going to 192.168.3.10 while I ping that ip [image: 1588973395669-sniff.jpg] So you see the ping go out, and in my case get a response... Do you see ping request go out.. Make sure your sniffing on pfsense B interface, while you ping from A (your lan with rules that are any any).. Just to be complete - my dmz rules do not allow pinging anything in my other networks. [image: 1588973666776-dmzrules.jpg] So while something in my dmz can ping pfs IP 192.168.3.253, can not ping pfsense IP say 192.168.9.253 root@pi-hole:/home/pi# ping 192.168.3.253 PING 192.168.3.253 (192.168.3.253) 56(84) bytes of data. 64 bytes from 192.168.3.253: icmp_seq=1 ttl=64 time=0.653 ms 64 bytes from 192.168.3.253: icmp_seq=2 ttl=64 time=0.497 ms Trying to ping 192.168.9.253 just fails.. root@pi-hole:/home/pi# ping 192.168.9.253 PING 192.168.9.253 (192.168.9.253) 56(84) bytes of data. ^C --- 192.168.9.253 ping statistics --- 10 packets transmitted, 0 received, 100% packet loss, time 9350ms

No.. carriers are generally Carrier Grade NAT. Though I have seen government agencies use cameras on Verizon service that could be accessed via public IP. I need to do more research. My Cradlepoint is in bridge mode on my test router right now. My test router has an address of 100.103.169.98 I see nothing today in my firewall logs today. Yesterday after I set it up in bridge mode I saw constant pings and udp traffic from other public IP's. So seems like somwhere a firewall got switched on.. I need to check the Cradlepoint closer.. VPN. Yes you can use OpenVPN as a client behind CGN to a box running as OpenVPN server. Some MIFI's will block VPN traffic by default and have to have it switched on in the device GUI. Im not sure about your modem.

Just as a follow up, I could never get OpenBGP to work. So I switched over to FRR and used its BGP and it works perfectly. So, there ya go.

Brilliant, I was hoping it would be that simple. Thank you.

You could try setting it up manually, like so: ifconfig gre0 inet6 <LocalV6> <RemoteV6> prefixlen 128 I did that for my dual stack tunnel and it seems to work well so far. … that is, until pfSense removes the v6-address again.