4100 ix Flow Control Help

selfjc

@nocling
I'll make sure to keep that in mind when I add Traffic Shaping back in.

Right now I have flashed the 4100 back to bare pfsense 23.01 because I was having the bandwidth dropout without Traffic Shaping.

The plan is to setup the interfaces with the segregated network IP ranges with only basic firewall from WAN to LANs. Hopefully the 4100 doesn't suffer drop outs with this arrangement. Then add back in the features I had before.

selfjc

@stephenw10 said in 4100 ix Flow Control Help:

So that was with the Limiters disabled?

Yessir. I've exhausted my capabilities of trying to find what feature caused the 4100 to drop bandwidth and opted to "start over from scratch."

selfjc

@selfjc
24 hour check in:
The 4100 hasn't dropped the bandwidth to ~20 Mbps. The speed tests maintain the full cable ISP link speeds ~500 Mbps and up to ~800 Mbps.

So far, so good.

I hope I didn't just jinx this. I'll do another check in later this week if the bandwidth drops or later in the week to report back the status.

selfjc

@selfjc
I spoke too soon. The internet speeds just dropped out again down under ~20 Mbps each direction. The full link speeds started 3pm yesterday and crashed out today around 8pm (29ish hours). A simple reboot with identical public IP from my ISP and I get back to full link speeds.

I am now running Traffic Status Totals to try and catch the amount of throughput makes the internet speeds crash.

stephenw10

Have you tried just disconnecting and reconnecting the WAN cable when it's in that situation?

Or rebooting the modem?

It's hard to think of anything that would affect the throughput of the 4100 like that. About the only thing I could imagine might be overheating causing the CPU to go in to thermal throttling.That would usually be fairly obvious from the temperature readings though. And it would affect all traffic to/from the box including LAN side. Also even at it's minimum speed you would see more than 20Mbps!

Steve

selfjc

@stephenw10 said in 4100 ix Flow Control Help:

Have you tried just disconnecting and reconnecting the WAN cable when it's in that situation?

I'll try that again. When I remember getting the ix3 interface to throttle and changing to igc3 interface for the WAN connection, the igc3 interface also was sluggish to start off until I did a Diagnostics -> Reboot -> Normal Reboot.

Or rebooting the modem?

I'll try that also when I get the 4100 to throttle the bandwidth.

It's hard to think of anything that would affect the throughput of the 4100 like that. About the only thing I could imagine might be overheating causing the CPU to go in to thermal throttling.That would usually be fairly obvious from the temperature readings though. And it would affect all traffic to/from the box including LAN side. Also even at it's minimum speed you would see more than 20Mbps!

Steve

The CPU temperature hovers around 48ºC to 50ºC when I do three concurrent speed tests through three different clients (two WiFi, one Ethernet). But if I get the bandwidth to throttle, I will check the CPU temperature first before the LAN cable and modem tries.

stephenw10

Mmm, that's no where near hot enough to start throttling.

Does it get a new public IP address when you reboot it?

One thing I might imagine is that the ISP is throttling the connection is reaction to something. And what that could be is the gateway monitoring pings over time. You might try disabling gateway monitoring as a test. Also consider your repeated testing itself may be seen as a problem.

Steve

selfjc

@stephenw10 said in 4100 ix Flow Control Help:

Mmm, that's no where near hot enough to start throttling.

Does it get a new public IP address when you reboot it?

No sir, the router/cable modem maintains the same Public IP address.

One thing I might imagine is that the ISP is throttling the connection is reaction to something. And what that could be is the gateway monitoring pings over time. You might try disabling gateway monitoring as a test.

I disabled the Gateway Monitor Action. The Gate Monitor I have left running. If the bandwidth drops out, then I stop the Monitor as well.

Also consider your repeated testing itself may be seen as a problem.

Steve

That's what I am worried about too. I originally only noticed when I was checking my Home Assistant history for the download speed "stuck" at 20Mbps for nearly a full week. I did a normal reboot and got full link speeds back. At that first time (~April 20) I didn't think much of the bandwidth dropout. Then I noticed the bandwidth dropouts started happening nearly on a schedule around 24 to 36 hours after a Normal Reboot. I suspect this time variance happens because there are days where I am not home using streaming services and not consuming as much bandwidth and over time not as much throughput.

But none of this happened before with the SG-4680 was still working (prior to March 24th, 2023 - mainboard failure - no console - no booting). I had Home Assistant running speedtest once an hour, I watched normal streaming services, etc. without the bandwidth dropping out with the SG-4680.

I also can't blame the ISP (yet) because with the same Public IP within 2 minutes for the Normal Reboot to complete and the full link speeds come back. The ISP isn't alibied out yet either. If the bandwidth drops out, the LAN cable plug-replug, reboot the modem, etc. make no difference, then I will run my network through a consumer Netgear router after trying the disabled Gateway Monitoring.

Thanks for the ideas!

To do:

Cable unplug-replug
Reboot modem
Disable Gateway Monitoring
Test through a Netgear router

selfjc

@selfjc said in 4100 ix Flow Control Help:

To do:

Cable unplug-replug

The cable unplug and count to 30 seconds and replugging in the WAN cable netted the same Public IP from my ISP. But the speedtest-cli on the Home Assistant (on igc4 interface) and on pfSense (ix3 WAN) both came back up to full link speeds.

Interestingly, the speedtest.net through the browser still has throttling while speed.measurementlab.net and waveform's bufferbloat come back up to link speeds (through igc0 LAN1 to ix3 WAN).

Does this mean the dreaded ISP throttling is to blame?

Reboot modem

Disable Gateway Monitoring

Test through a Netgear router

I didn't do these yet as the cable unplug and replug seemed to "fix" the bandwidth throttling.

stephenw10

Mmm, I would try disabling gateway monitoring as the next test then. You might be triggering some system at the ISP.
If that does stop it happening then try using a different monitoring IP and/or reducing the monitoring ping frequency.

Steve

selfjc

@stephenw10 said in 4100 ix Flow Control Help:

Mmm, I would try disabling gateway monitoring as the next test then. You might be triggering some system at the ISP.
If that does stop it happening then try using a different monitoring IP and/or reducing the monitoring ping frequency.

Steve

Aye aye. I think my ISP must have implemented an "overuse" policy on the no data cap plan that I have. I have toned down my Home Assistant's speedtest checks from hourly to only at 6am and 4pm. I think this will skirt the ISP throttling. So far I haven't gotten slowed back down.

I noticed when the ISP slowed my connection that the Waveform Bufferbloat was giving a score of B. I surmised that my connection/hardware was not bandwidth saturated but my ISP has instituted a silent throttling policy. The Terms of Use for Optimum hint that they can do that in very ambiguous terms (bolded section below).

From the Terms of Use for Residential Customers:

Optimum Internet Network Speeds. Subscriber acknowledges and agrees that actual Internet speeds that are experienced at any time will vary based on a number of factors, including the capabilities of Subscriber’s computer equipment, Internet congestion, the performance of network servers and routers, the technical properties of websites visited, environmental factors, the content and applications accessed, the condition of any lines between these two points, and any network management tools and techniques employed by Optimum.

This looks to be a witch hunt in where I found the ISP was throttling my connection. The 4100 appears to function fine.

stephenw10

Nice!

selfjc

@stephenw10

For educational purposes, what does unplugging the WAN cable from interface ix3 do inside of pfsense?

Does pfsense reset anything (e.g., firewall states, or buffers)?

Could a cron job of

/bin/ifconfig ix3 down
sleep 30
/bin/ifconfig ix3 up

do the same thing?

Gertjan

@selfjc

In a second console, SSH access, excute
tail -f "all interesting logs"
Like

tail -f /var/log/system/log /var/log/resolver.log  /var/log/gateways.log

Now,
do your

/bin/ifconfig ix3 down
sleep 30
/bin/ifconfig ix3 up

or take out the ix3 cable for a moment,
Or restart you upstream ISP router.

See what the logs tell you.

stephenw10

pfSense does quite a few more things when an interface bounces. It restarts a bunch of services, adds/removes routes etc.

selfjc

@gertjan
I did my best collect the tails of the logs but my ssh wouldn't allow access even though I was logged in as an admin account. I didn't have 'sudo' installed and 23.05 won't let me install 'sudo' because I am still at 23.01.

Anyways, here's what I collected from the logs via the GUI:
System -> General

May 23 06:19:29 	php-fpm 	48794 	/rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - x.x.x.x -> x.x.x.x - Restarting packages.
May 23 06:19:02 	php-fpm 	368 	/rc.linkup: Gateway, NONE AVAILABLE
May 23 06:19:02 	php-fpm 	368 	/rc.linkup: Gateway, NONE AVAILABLE
May 23 06:19:02 	check_reload_status 	405 	rc.newwanip starting ix3
May 23 06:19:01 	php-fpm 	368 	/rc.linkup: HOTPLUG: Configuring interface wan
May 23 06:19:01 	php-fpm 	368 	/rc.linkup: DEVD Ethernet attached event for wan
May 23 06:19:01 	php-fpm 	368 	/rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
May 23 06:19:00 	kernel 		ix3: link state changed to UP
May 23 06:19:00 	check_reload_status 	405 	Linkup starting ix3
May 23 06:19:00 	php-cgi 	8964 	servicewatchdog_cron.php: Service Watchdog detected service dpinger stopped. Restarting dpinger (Gateway Monitoring Daemon)
May 23 06:18:35 	check_reload_status 	405 	Reloading filter
May 23 06:18:22 	php-fpm 	368 	/rc.linkup: DEVD Ethernet detached event for wan
May 23 06:18:22 	php-fpm 	368 	/rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
May 23 06:18:21 	check_reload_status 	405 	Linkup starting ix3
May 23 06:18:21 	kernel 		ix3: link state changed to DOWN

System -> Gateways

May 23 06:19:08 	dpinger 	8948 	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr x.x..xx bind_addr x.x.x.x identifier "WAN_IPv4 "
May 23 06:18:23 	dpinger 	90025 	exiting on signal 15
May 23 06:18:23 	dpinger 	90025 	WAN_IPv4 x.x.x.x: sendto error: 65
May 23 06:18:22 	dpinger 	90025 	WAN_IPv4 x.x.x.x: sendto error: 50
May 23 06:18:22 	dpinger 	90025 	WAN_IPv4 x.x.x.x: sendto error: 50
May 23 06:18:21 	dpinger 	90025 	WAN_IPv4 x.x.x.x: sendto error: 50
May 23 06:18:21 	dpinger 	90025 	WAN_IPv4 x.x.x.x: sendto error: 50

The System -> Resolver/Unbound logs didn't seem informative so I left those out.

@stephenw10
Seems like pfSense could still be having strangeness if pfSense restarts most everything on a WAN ix3 interface down and up toggle.

Edit: Grammars

Gertjan

@selfjc said in 4100 ix Flow Control Help:

but my ssh wouldn't allow access even though

Check :

First :

Then :

( I took Putty as an example, any ssh client will do )

Btw : sudo ? why ? pfSense is not really a multi user device.
There is only one 'admin' acount. Same login as GUI. That's enough.

If you want to get wild : use :

![f1783846-9efb-41f0-b115-ceda8a0bc5cd-image.png](/assets/uploads/files/1684906559598-f1783846-9efb-41fare impo0-b115-ceda8a0bc5cd-image.png)

Console access will do also. SSH is just handy as you can connect from everywhere from LAN. It's normally the first thing you activate when you start working with pfSense, as it is as important as the GUI access.

Also : I've a 4100, using ix3 as a WAN interface.
When I restart my ISP router, this will trigger several LINK down LINK up event.
The thing is, pfSEnse won't stop generating itself (the ISP router is up and stable) LINK down LINK up events.
I'm using IPv4 DHCP, IPv6 DHCP, an OpenVPN server.
I've de activated the gateway action for both interface IPv6 and IPv4.
I saw that ix had flow control activated, I manged to stop that.

The issue isn't important for me, as ISP router and pfSense are UPS powered, and the ISP is quiet rock solid with the connection (fibre).
I I still have to sit down ones, and see why this happens. Maybe the 23.05 will resolve this.

selfjc

@Gertjan
I apologize, I used the wrong terminology of "access" when "permission" is more appropriate. When I login with my user account that is in the admin group via ssh to pfSense and run your:

tail -f /var/log/system/log /var/log/resolver.log  /var/log/gateways.log

I get this result:

tail: /var/log/system/log: No such file or directory
tail: /var/log/resolver.log: Permission denied
tail: /var/log/gateways.log: Permission denied

So then I fall back to my linux command line knowledge and try to "sudo" the command (I know pfSense is BSD, peace):

sudo: Command not found.

Then I try to switch user to root via "su":

su: Sorry

pfSesne won't allow me to install the "sudo" package because 23.05 has released and I'm still on 23.01. So I resorted to the GUI log views for now.

I also have a UPS powering my pfSense router and my ISP modem along with a few other key networking infrastructure parts (e.g., Unifi AP and Unifi Cloudkey).

It's very interesting that pfSense resets the packages and services on a WAN interface down. This seems to cure the ailment I have of the bandwidth dropping out through the pfSense router. So I'll keep the cron job at 1am to drop (down) the ix3 and ix2 interfaces for 30 seconds and bring them back up.

For anyone interested in investigating pfSense 23.01 dropping WAN bandwidth through the 4100, here are the services running on my instance of pfSense 23.01:
Screen Shot 2023-05-24 at 06.29.36.png

And my installed package status:
Screen Shot 2023-05-24 at 06.29.54.png

For me, after several weeks of disabling individual packages and services to no effect - I am considering the cron job of taking down the WAN interfaces and bringing those interfaces back up a "good enough solution."

For completeness, here's the corrected cron script that runs for at 1am to cure the bandwidth drop out every 24-36 hours. Make sure to set the file permissions "0755" (or something with the execute for everyone) if using the Filer package:

#!/bin/sh
/sbin/ifconfig ix3 down
/bin/sleep 30
/sbin/ifconfig ix3 up
/bin/sleep 10
/sbin/ifconfig ix2 down
/bin/sleep 30
/sbin/ifconfig ix2 up

Gertjan

@selfjc said in 4100 ix Flow Control Help:

When I login with my user account

pfSense is not like, for example FreeBSD.
There is only one access, no need to have an 'admin' collection : one guy can handle everything very well.
It's a firewall, not a mail/web/whatever server.

@selfjc said in 4100 ix Flow Control Help:

So I resorted to the GUI log views for now.

So your are not the admin ??
If you are, why making your own live harder ?
SSH access is not some gadget, you need it.

You can of course do the thing that needs to be done : forbid iser/password login : go forts SSH + Public key only. Forbid any access to LAN, use LAN just for admin activities, Everybody else on another LAN where SSH + GUI access is impossible.

stephenw10

Yes some things require the actual admin/root account, not just an account with admin privileges. Packet captures for example.