pfsense drops wan (PPPoe) when using speedtest.net

maxbits · Jul 27, 2020, 2:41 PM

Hello,

I have a pfsense configured to use PPPoe and make the connection with my ISP so that the pfsense can route my public ips. Before the pfsense I had a Cisco ASA establishing the PPPoe connection and when i accessed speedtest.net there wasn't a problem. Upon configuring the pfsense to make the call, whenever i visit speedtest.net there starts to be packet loss and the WAN goes down. Are there any potential solutions on fixing this problem? Thank you

Connection methodology:

ISP: Cosmote S.A. --> ISP's router in Bridge Mode --> Cisco ASA (no problem)
ISP: Cosmote S.A. --> ISP's router in Bridge Mode --> pfsense PPPoe (problem connection drops)
Please note that i am in Athens, Greece and any site located abroad show much lower speeds

Gertjan · Jul 27, 2020, 2:38 PM

Double check using this test : http://www.dslreports.com/speedtest

maxbits · Jul 27, 2020, 2:45 PM

I get the following: Still waiting to load page (23 seconds) so it doesn't load

netblues · Jul 27, 2020, 4:22 PM

The problem lies elsewhere

login-to-view
This is pfsense speedtest.net using pppoe with the said providers router in bridge mode.
(max sync is at 42mbits due to distance from dslam.)

Are you using defaults everywhere? Just a single wan?

stephenw10 · Jul 27, 2020, 5:10 PM

Sounds like catastrophic buffer bloat. It would be good to use a test that measures that.

If you canlt reach dslreports try the test at fast.com with 'Measure loaded latency during upload' enabled.

Are you seeing other issues reaching sites?
https://docs.netgate.com/pfsense/en/latest/routing/unable-to-access-some-websites.html

If you only have one WAN you can edit the gateway in System > Routing > Gateways and check Disable Gateway Monitoring Action.
That will still log stats for the gateway but it won;t get marked down.

Steve

maxbits · Jul 27, 2020, 6:56 PM

@netblues

I have 2 wan interfaces as a temporary solution.

The first wan interface does the PPPoe Connection with my ISP and receives the static IP.
The second wan (WAN2) inteface is also connected to the modem (Oxygen) and gets an IP from the modem's DHCP. We can ignore this interface but it is important to note that when i am speedtesting with WAN2 i do get 98 mbps while with the PPPoe Interface apart from packet losses i only get 76 mbps. My modem synchronises with my ISP at 108 mbps.
On the WAN interface the MTU is set to 1492.

maxbits · Jul 27, 2020, 6:51 PM

@stephenw10

I have tried the test at fast.com but as it is known fast.com overreports speed. I got the measurements: login-to-view

netblues · Jul 27, 2020, 7:20 PM

@maxbits Why do you need the second wan in the first place?
Are you policy routing/loadbalancing traffic?

maxbits · Jul 27, 2020, 7:49 PM

I had some servers installed and i didn't want guests to exit through the static ip and potentially get my ip blacklisted. I therefore decided to route traffic of the guest vlan throught the secondary wan which is all done on cosmote's modem and uses cli credentials with a dynamic external ip.

stephenw10 · Jul 27, 2020, 11:11 PM

Hmm, the bloat doesn't look all that bad.

But having both PPPoE and DHCP connections to the same modem, at the same time seems very odd. I'm not sure I understand what you're doing there....

Steve

maxbits · Jul 28, 2020, 3:14 AM

I would like to thank everyone for your assistance, i really appreciate your help. I decided to reset pfsense to defaults and switch off dhcp on the ISP's router. I also switched off gateway monitoring. Speed improved a lot. Can I increase it anymore? Connecting my pc directly to the router yields close to 100 on speedtest login-to-view

login-to-view

maxbits · Jul 28, 2020, 3:32 AM

Upon connecting my laptop directly on the LAN interface of the firewall and running speedtest using single server option i managed to get 96.login-to-view

netblues · Jul 28, 2020, 4:02 AM

Are you getting those bad speeds over the static address or via the oxygen router dhcp one?

Ypu are probably being hit by tripple nat.
Since your normal credentials are being used for the static, oxygen is using the generic otenet@otenet.gr. I bet it has an 100.x address on its wan and is behind cgnat. (with another 64 users on the same ip)
Then oxygen does another nat, and then pfsense does a third one.

Clearly, having two ppp connections with heavy traffic through the same vdsl ptm can lead to heavy bufferbloat.

Consider switching wan2 to pppoe too, eliminating one nat.
Unless voip is also used on oxygen, consider removing internet altogether from oxygen.
Use 195.170.0.1 as monitoring ip for the second wan.

Check speedtest when there is no traffic on both connections.
Then put limiters for the two connections, say 7mbit upload for servers and 2 for the users.

p.s. Maybe a second static, or a second vdsl is a far better idea after all.

maxbits · Jul 28, 2020, 4:23 AM

I am now getting 82 download on speedtest. I have called cosmote and since we are on a business branded package, cosmote has enabled cli credentials meaning that voip connects to the internet via its own PPPoe credentials cli of the form cli_210xxxxxxx@otenet.gr without using the default otenet@otenet.gr, therefore eliminating the potential nat problem. I have asked them and there is no other way of doing it. As stated earlier upon restoring pfsense to its default config i am getting 82 on speedtest and 99 on fast.com using my pppoe credentials.

netblues · Jul 28, 2020, 4:38 AM

@maxbits Are you sure cli_210 isn't getting cgn? Check oxygen status page

stephenw10 · Jul 28, 2020, 10:23 AM

This is a DSL connection right? Can you see the line speed the modem is sync'd at?

You seem to be getting at or very close to the speeds you were getting using DHCP now. I assume the ISP device does PPPoE when that is happening?

Steve

maxbits · Jul 28, 2020, 1:07 PM

@stephenw10 said in pfsense drops wan (PPPoe) when using speedtest.net:

I assume the I

Yes this is VDSL, the ISP device is in bridge mode and pfsense does PPPoe.