Changed the NIC to an Intel ET adapter from a Marwell Yukon. No packetloss and so far still a stable connection.

Looks like the Marwell driver has a memory leak.

@donuts

It shouldn't. But that's why you should know what's normal, before trying to find out what's failed.

@mjimlay

The same way as you'd route over any IP interface. Go into System>Routing and go from there. You might also have to consider firewall filters.

Well, I think that there is two solutions, depending of the objective:

Objetive 1.- Local Server <---> Remote Server

Create a Phase 2 and configure:

Local Network: Address > 10.10.10.10
NAT/BINAT translation: Address > 192.168.20.1
Remote Network: Address > 20.20.20.20

Only the Host that We put in Local Network can go through the VPN to the Host that We put in Remote Network.

We also need add a Firewall Rule (Firewall > Rules > IPsec) that permit the traffic from 20.20.20.20 to 192.168.20.1

The Local Server can connect to Remote Server through the IP 20.20.20.20 and the Remote Server can connect to Local Server through the IP 192.168.20.1

Objetive 2.- Local LAN <---> Remote LAN

Create a Phase 2 and configure:

Local Network: Network > 10.10.10.0/24
NAT/BINAT translation: Network > 192.168.20.0/24
Remote Network: Network > 20.20.20.0/24

All de Hosts in the network that We put in Local Network can go through the VPN to the Hosts in the network that We put in Remote Network.

We also need add a Firewall Rule (Firewall > Rules > IPsec) that permit the traffic from 20.20.20.0/24 to 192.168.20.0/24

In this situation, the NAT is done Host to Host, that is:

10.10.10.1 > 192.168.20.1
10.10.10.2 > 192.168.20.2
10.10.10.3 > 192.168.20.3
10.10.10.4 > 192.168.20.4

And the Remote Hosts can reach the Local Hosts by the corresponding NATed IP (192.168.20.x)

I think that this is correct. If It is not correct, please, tell me.

We are thinking that all config is correct in the Remote FW.

Regards,

Ramsés

@NogBadTheBad Yes, understood - I just tweaked it a bit to confirm the root cause of the issue 😆

@Gertjan

I'm sorry. I'm not understanding what you wrote. Unbound has forwarding disabled so it should be doing it's own resolving. I'm not sure what else you would want me to detail.

Network Interfaces: LAN+All VLANS
Outgoing Network Interfaces: LAN + OPT1
DNSSEC: ON
Forwarding: OFF
DHCP Registration: ON
Static DHCP: ON
OpenVPN Clients: ON

As for the version issue. Just refreshing changes whether it says I'm on the latest or if there is a new version available. If i keep refreshing it switches back and forth. I'm assuming that is because sometimes there is response on the WAN and sometimes on the OPT1, however, I would expect it to either be correct in showing 2.4.5 or just fail. Since I've disabled OPT1 it's been correct every time I refresh.

I know there is some kind of DNS issue on the AT&T side. I'm facing 2 issues that I see:

Why DNS queries are sent out of OPT1 when the routing is still going out of WAN. Why DNS is failing and returning the wrong info over the DSL where it shows google.com at 192.168.1.254 and the Arris modem is trying to pass off the certificates. I assume that is because that is what the modem is sending.

My best guess at this moment is that the DSL modem has been reset and is intercepting all of the traffic because it's waiting for a user to log in and activate. That's the only thing I can think of that would cause it to behave in this way. (I HATE DSL). I suppose what I need to know, then, is how to limit DNS queries to go out the interface that is the current route. I don't want queries going out OPT1 when routing has the data going over the WAN.

@penguin-nut Brilliant mate - many thanks - thats the starter for 10 i needed. I'll give it a shot at some point.

@viktor_g, the only solution is?

pfSense-01:

Export the Certificate and the Key.

pfSense-02:

Import the Certificate and the Key exported. Create a new User identical to the User in the pfSense-01. To edit the new User and select to use the Certificate imported.

It's right?

The problem is the password of the new User, isn't it?

Regards,

Ramsés

What is your RADIUS server?
FreeRADIUS or AD?
Any 2FA features (like DIGIPASS)?
can you check it with simple shared secret and userpass (like '123')?

And what sort of rules would you put in this group... Keep in mind rules on a network.. Quite a few of them you would want to be specific to that network... Pinging the specific interface for example - so you clients can validate they can actually talk to the gateway.. Maybe use dns to that interface... Sure you could allow dns to this firewall sort of rule..

Or you could use this firewall alias if you don't care if client pings another IP on the firewall..

How best to do the rules would depend on your specific setup and your specific wants and needs for what is allowed and what is not allowed. Do you even know what these are as of now?

If so spell them out, and we can go through how you can do what you want with the min amount of rules. But smallest amount of rules is not always the best case..

Its very handy to put the rules on each interface so you can easy check with simple glance at that interface what that network can do.. Example... Here is a common sort of lock down that you might do for a network.

rules.jpg

I can look at that, and in seconds understand exactly what this network can an can not do..

It can ping pfsense IP on that network, it can ask it for dns, it can talk to it for ntp.. Any other port it can not talk to the firewall on.. That would be any other vlan, or the wan.. It can not talk to any rfc1918 address - ie any other network at all.. Then anywhere else - Ie the internet, it can do anything it wants.. Those rules take all of like 1 minute to setup.. And are easy and clear and easy to understand. I don't have to check floating tab and look to see if a specific rule as interface X in it.. I don't have to check an interface group..

Here is a group rule... But what interfaces are in that?

group.jpg

If I would just take the extra 15 seconds and put the rule directly on the interface - I can easy see on any network EXACTLY what the rules are for that network... I don't have to jump all over looking to see if floating rule applies or if group has interface X in it, etc..

So while these features are handy if you have 100's of interfaces.. If your managing a handful.. Its easier to just do the rules on the interfaces directly - even if you might have to duplicate a bit of work.. It will make your life easier going forward!!!

@jimp said in Why the extra NTP servers?:

Best practice is to use no less than three NTP servers, for accuracy and redundancy. With one, you have no assurance the server is accurate. With two, you can't tell which one is wrong if they don't agree. With three, you can at least have a good chance at excluding an outlier.

Yep. I have 3 stratum 1 servers and the pool. I figure that should be good enough. Also, according to what I read about multiple servers is the average is used, which results in better precision.

BTW, here's an interesting book about accurate time from the NIST:
From Sundials To Atomic Clocks

Your modules don't need to 'vendor match' on each end, but they'll need to negotiate a link speed between them. SFP+ modules are typically 10G, while SFP are 1G. You CAN plug an SFP into a SFP+ slot, but you can't do the reverse. You'll have to look at module specifics if you want a 10G fiber to negotiate down to 1G, I've never done it and I don't know how well it's supported. Since it's a Supermicro chassis, and not something proprietary like Cisco, you should be able to pretty much use any vendor module on that side; I can't speak to UniFi on what modules they accept.

So in a nutshell, you should be fine getting two 1G SFP UniFi (Ubiquiti) modules. Personally, I wouldn't pay $15+ for a 1G fiber module.

@chidmas said in Low bandwidth extremely low for local network to WAN however normal for pfsense box to WAN.:

N3050

I'm wondering if your CPU is getting choked out, it's pretty low spec with only two threads. Have you tried, and can you provide, the following:

Monitored the CPU load while doing the hard-wired speed test Provide the model of the Intel nic Do a direct connect speed test (remove all hardware, just pfsense + workstation) check your logs for any errors

@Gertjan Oh god! I did change what you suggested and it fixed my issue! I am not sure myself why i did change it but i'm defininitely learning from my mistake. Thank you so much!

@jimp, I'm sorry but I'm afraid that something is wrong because in "Firewall > NAT > Port Forward" works well, when I change the value of the Virtual IP (CARP) in "Firewall > Virtual IP" It updates dinamically in "Firewall > NAT > Port Forward" but not in "Firewall > NAT > Outbound".

Example.

Firewall > Virtual IP > VIP CARP DEDI_NIC_FO (Type: CARP) => 80.80.80.80

Firewall > NAT > Port Forward:

In the NAT Port Forward Rule We select Destination: VIP CARP DEDI_NIC_FO

01-NAT-PF.png

In the Rule appears the correct value of VIP CARP DEDI_NIC_FO.

02-NAT-PF.png

Firewall > NAT > Outbound:

In the NAT Outbound Rule We select Translation > Address: VIP CARP DEDI_NIC_FO.

03-NAT-Out.png

In the Rule appears the correct value of VIP CARP DEDI_NIC_FO.

04-NAT-Out.png

We Edit and Change Firewall > Virtual IP > VIP CARP DEDI_NIC_FO (Type: CARP) => 90.90.90.90

Firewall > NAT > Port Forward:

In the Rule appears the correct value of VIP CARP DEDI_NIC_FO. It's changed dinamically.

05-NAT-PF.png

Firewall > NAT > Outbound:

In the Rule appears the old value of VIP CARP DEDI_NIC_FO. It's not changed dinamically.

06-NAT-Out.png

I think that It's not correct.

Is this the correct way of operating or should It change dinamically too?

Regards,

Ramsés

RRD intentionally aggregates data into larger intervals as the data gets older.

The monitoring graphs are intended to provide troubleshooting information, not be a high-resolution, historical archive. For that you can query the device using something like cacti or zabbix or a plethora of others.

Setting 8 hours x 1 minute resolution is pretty comprehensive. Anything longer than 8 hours and the resolution will be reduced.