@johnpoz Yeah, the public IP from my /29 made sense in the old scenario, since the 7100 was doing the load balancing/failover. The VoIP system still needed a public IP to function optimally.

One other thing to consider is actually leaving the public IP, but configuring it as a WAN2 connection. Ubiquiti recommended only to use failover, not load balancing, so it may work if Spectrum goes down.

@Ducati0927
You can do this with policy routing.

Set a static mapping for the TV in the DHCP server or give it static IP.

Create an alias for private network ranges if you didn't already. Add all RFC1918 network ranges to it.

Add a firewall rule on the LAN with the source IP of the TV and at destination use the RFC1918 alias together with "invert match" checked.
Open the advanced options, go to gateway and select the desired WAN gateway.

Put this rule to the top of the rule set, so that it is probed before the common outbound NAT rule.

I finally fixed it, it seems.

A few weeks ago I put outbound NAT in as a fix for NTP that was throwing IPv6 errors into the system log on both the Starlink and T-Mobile interfaces. NTP was also failing to consistently determine time or serve it. All this despite IPv6 being blocked/unused on my pfSense.

However, the outbound NAT was for interface traffic going from This Firewall to anywhere on that interface translated to the interface address.

Looking at that today it finally dawned on me that the outbout NAT could be capturing the ARP traffic to the Starlink gateway. I narrowed the outbound NAT down to just port 123 and the llinfo errors stopped ocurring.

That said, holy cow the error messages in FreeBSD are frequently far, far away from being intuitively obvious. Hours of searching failed to even find a definition of what llinfo was, let alone what the error meant.

Amazingly, at one point I seem to have found the original code by the original author who commented that he'd be back around to figure this out later. Later appears to be many, many years later. Still waiting, I guess.

Dating myself, I really miss the formality of OpenVMS and the incredibly helpful Error Messages and Codes manual. Loose software engineering, standardization, phrasing and undefined error messages make trouble-shooting admittedly lame user errors incredibly hard.

I love the stability, flexibility and UI of pfSense, but better error messages and codes documentation would save many dumb old former CS professionals like me many, many hours of frustration along the way. That in turn might sell more products and support the future of the company better.

@pankajpomal1 and all of your network connections are only 100mbps?

Rough napkin math puts you at like 2-3mbps per camera when recording.. (if your only doing motion only)

You might be able to get by if there is not a lot of motion on all the cameras at the same time.

But you need to move this data flow away from your normal users data flow.. With 100 some cameras, I would assume you have multiple switches.. So you run into a bottleneck on the uplinks..

Can you provide a basic drawing of how everything is connected, how many switches - what is connected to what, etc. Are you just all on one flat network, ie 192.168.1.0/24 or do you have multiple networks 192.168.1/24, 192.168.2/24, 192.168.3/24 etc..

You really should be using gig for the amount of cameras and and that recording rate..

edit: in the most basic of setups using just simple physical isolation. You would use different switches for your camera network vs your user network.

basic.jpg

When you have multiple networks for connecting all of your devices you would have to worry about physical uplinks between switches, etc. But generally the idea is to separate all your camera data onto its own switches and uplinks from your user data. So depending how many switches, how many devices, and physical locations of users and cameras in relation to what switches they can connect too.

@johnpoz thanks a lot for your help,
Well colt did not give any details however from your great schema I think all what I am missing is to set the IP addresses for the Lan 2 Lan network.

Then we should be ok for the rest, Ill keep you posted.

Again thanks a lot for your help.

@Gertjan said in Possibility to use an internet link from another firewall:

Condition : add on every pfSense a ninth interface.

Unfortunately I cannot add more network interfaces as I am limited to a maximum of 8 per VM.

@bigbang

Not a problem

Thanks
Dan

@Gertjan said in PPPoE with private gateway IP conflicts with LAN:

The thing is : he has forgotten he can change it again.

I didn't forgot that :) But wanted to avoid that as it was a pain to do and I would like to have avoided that. (finished the change last night).

@Gertjan said in PPPoE with private gateway IP conflicts with LAN:

Your ISP, mine, all the others, they have the same problem : they've bought boatloads of IPv4 in the good old days. These days, thee IPv4 are worth a fortune.
They can't give every customer an IPv4 anymore. So, they map more and more clients to RFC1918 (CGNAT ?) as that will give them a solution.

The thing is, they STILL give a public IPv4 address. Their gateway is (and was) the issue..
13780b09-2103-4965-8995-a5155c52a153-image.png

You want a real IPv4 ? You'll have to $$ ^^

They have you pay for static (only for business plans). Which also will suffer from the above issue when the rollout is complete (and they use a network 10.10.10.0/x

@askon
The company's firewall.
But I suspect, this is not an option for you. So you would have to go with a static route on the WAN device which you want to access the IoT subnet from.

So I think the thing I'm missing is that I shouldn't route the prefix from the PoP's to pfSense and instead setup BGP on pfSense and announce the route that way...

Does that make sense ?

edit: don't think that makes any difference since it will still route the same prefix over both interfaces, I just can't get my head around it.

Maybe it doesn't matter if the same prefix is routed via both interfaces and I can just do static IPv6 configuration on another interface ?

@anyn12 Delete the setup and start the setup again, I mean the LoadBalance setup, have u try this?
Regards!!!

@chitchat Do u have a FailOver setup as Default gw?
Regards!!!

@mcury I see. Thanks.

You have two gateways.
Try to check states and firewall logs on them,
Check exactly on which interface the traffic appears.

@insmod Try to use computer. Connect to first switch and change swtich port to vlan 200. Next configure pppoe on your computer. Install wireshark and watch packets.

you have mishmash with vlan

correct configuration
modem0 vlan 100 access mode
modem1 vlan 200 access mode

igb0 switch port connect hybrid or general mode non tagged 100, tagged 200
interface on pfsense igb0 non tagged (for 100 vlan), igb0 tagged 200 (for vlan 200)
or
igb0 switch port connect trunk mode tagged 100,200
interface on pfsense igb0 tagged 100, igb0 tagged 200