Well, if you haven't set it up very strangely, the routes should already be there. But to be sure, you can check Diagnostics -> Routes. You can also try to ping a host on the other network. If you don't get something like "Destination net unreachable", chances are good that your routes are fine.

Most likely you need to add/adapt some firewall rules. Probably you have some firewall rules that restrict access between your LANs before the "allow to any" rule that allows Internet traffic, or the "allow to any" rule excludes your local networks?

I haven't completely understood your SERV setup. As far as I understand, webmail.ORGNAME.co.uk has a public IP so it is accessible from Internet? How do you map that to your private IP addresses?

By default all upstream traffic is sent to the default gateway. So for that VLAN which is meant to go out the default gateway there is nothing to do.

Assuming your default GW is on WAN1.
For the VLAN20 you have add an outbound NAT rule to masquerade the outbound traffic with the WAN2 address.
Go to Firewall > NAT >Outbound and select the hybride mode and save it.
Then add a rule:
interface: WAN2
source: VLAN20 network
translation: WAN2 address

For directing outbound traffic from VLAN20 to WAN2 gateway, you have to use policy routing.
That means you have to edit your firewall pass rules for upstream traffic on the VLAN20 tab, open the advanced options, go down to gateway and select the WAN2 GW.

Consider that if you need any access to pfSense itself like DNS, you have to specify additional pass rules for that, because the policy routing rules only allow access to the stated gateway.
The same applies to access to other local networks.

If you have inbound traffic on the non-default WAN also consider to only allow it on the WAN2 tab. Do not use floating rules or interface group rules for that!

@pm1961 said in IoT device will work on one subnet but not another...............:

I thought what goes on in my LAN, stays in my LAN?

True - but OS and Applications can do stuff differently when they think they are on a public IP vs a rfc1918..

While technically speaking sure if you want to use MS IP space internally have at it - other draw back is hope you don't actually want to go to any site using that public space.

Do yourself a favor, and use the rfc1918 space as it was intended... The other benefit of that is anyone trying to help you isn't going to be rolling their eyes.. Which was exactly the first thing I did.. Along with facepalm... you got a twofur

eyeroll-face.gif

Also @SampleX is correct what you have given there is a host address.. If you want to call out a network the octet given would be the wire, not some host on that wire... So 10.10.10.0/24 would be the correct network address.. Anything else .1-.254 would be seen as host on that network..

Also it makes it difficult for anyone trying to help you to know that those networks are actually just local and not some public space you have control over.. Since no sane person would just grab IP space out of thin air and use internally ;)

@chpalmer
The local phone company supplied the modem and set up bridge mode. I know they have some access in their end but I don’t think it’s the modems own GUI but rather a management interface. When installed they did nothing locally in terms of configuration. I will give that address a try.

So more troubleshooting this evening. Here's the current hangup - still stuck on outbound routing of packets from the /28 to the wireguard interface/gateway. Here's an illustration that I've put together based on tcpdump tracing of the ICMP packets.

alt text

Wireguard Firewall Rules:
alt text

Vlan 41 Firewall Rules:
alt text

And finally, the current output of netstat -r:

Destination Gateway Flags Netif Expire default (WAN Gateway) UGS vtnet0 44.48.41.16/28 link#7 U vtnet1.4 44.48.41.17 link#7 UHS lo0 (ISP Subnet)/22 link#1 U vtnet0 (WAN Address) link#1 UHS lo0 localhost link#3 UH lo0 192.168.0.0/24 link#9 U vtnet1.1 192.168.0.1 link#9 UHS lo0 192.168.1.0/24 link#2 U vtnet1 router link#2 UHS lo0 192.168.2.0/24 192.168.2.2 UGS ovpns1 192.168.2.1 link#10 UHS lo0 192.168.2.2 link#10 UH ovpns1 192.168.16.1 link#11 UH tunwg0 192.168.200.0/24 link#8 U vtnet1.2 192.168.200.1 link#8 UHS lo0 (ISP DNS Server 1) UHS vtnet0 (ISP DNS Server 2) UHS vtnet0

Hopefully that clears things up a bit. Or makes everything even more confusing. Sorry to keep asking what could be a bunch of dumb questions.

@viragomann said in Routing issue with two LANs and external router:

Routing works properly only if the source and the destination are on different interfaces. If they are on the same, you will get an asymmetric routing issue.

Yes, that's the classic asymmetric routing. But what deceived me is what is written in the pfSense documentation:

In asymmetric routing scenarios, there is an option that may be used to prevent legitimate traffic from being dropped. The option adds firewall rules which allow all traffic between networks defined in static routes using a more permissive set of rule options and state handling. To activate this option:

Click System > Advanced
Click the Firewall/NAT tab
Check Bypass firewall rules for traffic on the same interface
Click Save

So I was thinking that the option "Bypass firewall rules for traffic on the same interface" was enough to make everything working

Yes, this can also be done on pfSense, but it's not an optimal setup though. By doing masqerading, all pakets seem to come from the router itself instead from the origin device.

However, if you want to set that up:
Firewall > NAT > Outbound
Switch into the hybrid mode and save.
Add a new rule:
Interface: LAN
source: LAN net
destination: 192.168.0.0/24
Translation: interface address (default)
Save

That's make sense and can be a solution. I will try it but I feel confident.

Thanks for your help

Would need to assess the network layout to look at possible solutions. However, one thing that could've been done is change LAN subnet on the VPN firewall.

your rule with 'linkfailover2' is the only rule that is used in your ruleset. that is why everything is going through the 'wrong' wan

https://docs.netgate.com/pfsense/en/latest/firewall/firewall-rule-processing-order.html#short-version

@louis2 said in How to access local services “Via the Front Door” !!??:

The security is bypassed (unless you define and maintain a second rule set). Unacceptable IMHO
You have to define and maintain a second DNS tree, not a good idea as well
Sincerely,

Louis

A) your picture in the previous post seems to indicate a complicated ruleset with a bunch of floating rules.
A.1) try to simplify your setup. it's almost never a good idea to make simple stuff hard,just because you can
A.2) floating rules have their uses, but a lot of times they are used incorrectly or without warrant
B)what security are you talking about?
B.1)by maintaining a second ruleset you mean?

creating a single additional firewall-rule? that uses the same alias as your existing rule? and if changes are made to the alias, both rules are up to date ?

C) no need for a seperate dns tree, just use host or domain overrides
D) there might be better ways to accomplish whatever it is you need. But without explaining you setup in detail it will be hard for some of the experienced community members or devs to pitch in

So I got a little bit further on this:

If I use the igmp proxy package instead of pimd, with the upstream addresses being the two ISP addresses and 232.x.x.x multicast address, and dowstream address being the LAN subnet (that contains the STB) I can see the the ISP's IP connect to the 232.x.x.x multicast address on the pfSense WAN interface (after setting up the appropriate pass firewall rule). This leads me to believe that multicast is working.

However, I'm not sure how to duplicate this same behavior using pimd. It seems straightforward if all I'm doing is multicast routing between local subnets, but how would it work when the WAN interface and NAT come into play? Do I have to setup some manual NAT forwarding rules? Can I configure pimd in such a way that multicast traffic reaching WAN will reach the STB on the LAN subnet?

Thanks in advance any help you can provide, I really appreciate it.

10 bucks the real cause is your NIC and not the system load. ;-)
Realtek is just bad and combined with PPPoE it's even more worse.
Are you running the driver shipped with FreeBSD/pfSense? You could try with this one then https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release
Or try with Intel NIC if possible.

-Rico

Well pick and interface on your 3100 and 5100 to use as your transit network.. For your 3100 you will prob have to carve out one of the switch ports to use. Put it on a different vlan... The 5100 has discrete interfaces so a bit easier.

I restored a backup from March which worked. Seems like I configured something wrong and it only showed after the rebooot.

@hopkins said in Routing 169.254 Networks:

shows no blockage for packets in 169.254.0.0/16

To talk to something on another vlan that is using apipa, that device would have to have a gateway.. Or you would have to nat to it, etc.

Ok, I had it set up right then. From some reason, it doesn't come out right of the untangle so that may be my problem. But just wanted to make sure the outbound of the pfsense was setup right.

Thanks

Re-reading all the docs over the weekend and going through forum posts. The following forum posts and doc links explained the intra-VLAN connectivity issue:

Forum post
Firewall Rules and Policy Route Negotiation
By-passing Policy Route Negotiation - Inclues example local segment rule
Mulit-WAN Policy Routing
Another Multi-WAN Policy Routing page

My missing link was my failure to understand that the second you create a gateway group you enter policy routing territory (as soon as you set the gateway on a rule). My simple "allow all" one rule per VLAN then stop working for local segments as of course everything is directed out the gateway. Seems obvious now. So I need to split my rules up as explained by the forum post so that local traffic is passed without a gateway group and the gateway group rule is that last rule in the set.

I am still completely confused about why the firewall itself can't get out externally though. The IPv4 default gateway is the gateway group. This forum post says that is what is needed for the the firewall to maintain external connectivity. It also talks about a setting that flushes state when any gateway goes down. But in my scenario the Tier 1 gateway is down as the firewall comes up so I'm not sure what needs to be reset. I will try this setting anyway and see if anything changes. My understanding is that this setting will cause disconnections for both gateways if either fails but I'll worry about that if the setting fixes the current issue. I found a couple of old posts that talked about the previous feature "Default Gateway Switching" not working with PPPoE gateways. So maybe my Tier 2 gateway isn't taking over because it is a PPPoE gateway.

I will experiment further and update here after. I've posted all links used in case anyone else is struggling.

And the winner was outgoing NAT-rule on the VPN-interface :)