Vr interface resets under load



  • I've been noticing an issue if I put pfSense on my Alix under load and max out my internet connection download.  I'm hoping for help in narrowing down where the issue exists.  My internet connection is 25mb/10mb,  if I max out the download the issue occurs.  If I throttle the downloads to 16mbps everything is stable.

    I've got a set of system logs from when this happens

    
    Mar 22 02:42:41	php: rc.start_packages: Restarting/Starting all packages.
    Mar 22 02:42:38	kernel: pid 47245 (php), uid 0, was killed: out of swap space
    Mar 22 02:42:26	check_reload_status: Starting packages
    Mar 22 02:42:26	php: rc.newwanip: pfSense package system has detected an ip change -> 172.16.8.1 ... Restarting packages.
    Mar 22 02:42:24	check_reload_status: Reloading filter
    Mar 22 02:42:24	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.8.1) (interface: ) (real interface: ovpns2).
    Mar 22 02:42:24	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
    Mar 22 02:42:24	kernel: pid 66229 (php), uid 0, was killed: out of swap space
    Mar 22 02:42:17	kernel: pid 23299 (php), uid 0, was killed: out of swap space
    Mar 22 02:42:05	check_reload_status: rc.newwanip starting ovpns2
    Mar 22 02:42:05	kernel: ovpns2: link state changed to UP
    Mar 22 02:42:04	check_reload_status: Reloading filter
    Mar 22 02:42:04	check_reload_status: Reloading filter
    Mar 22 02:42:03	kernel: ovpns2: link state changed to DOWN
    Mar 22 02:42:03	check_reload_status: rc.newwanip starting ovpns2
    Mar 22 02:42:02	kernel: ovpns2: link state changed to UP
    Mar 22 02:41:52	php: rc.dyndns.update: phpDynDNS (ut.niagara-it.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Mar 22 02:41:52	php: rc.openvpn: OpenVPN: Resync server2 Personal and friends
    Mar 22 02:41:51	check_reload_status: Reloading filter
    Mar 22 02:41:51	kernel: ovpns2: link state changed to DOWN
    Mar 22 02:41:50	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
    Mar 22 02:41:49	kernel: pid 39945 (php), uid 0, was killed: out of swap space
    Mar 22 02:41:47	check_reload_status: Reloading filter
    Mar 22 02:41:47	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.8.1) (interface: ) (real interface: ovpns2).
    Mar 22 02:41:47	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
    Mar 22 02:41:45	php: rc.openvpn: OpenVPN: Resync server2 Personal and friends
    Mar 22 02:41:45	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP6.
    Mar 22 02:41:44	kernel: pid 46465 (php), uid 0, was killed: out of swap space
    Mar 22 02:41:41	kernel: pid 19097 (php), uid 0, was killed: out of swap space
    Mar 22 02:41:41	kernel: pid 21659 (php), uid 0, was killed: out of swap space
    Mar 22 02:41:39	kernel: pid 19932 (php), uid 0, was killed: out of swap space
    Mar 22 02:41:32	php: rc.newwanipv6: rc.newwanipv6: Failed to update wan IPv6, restarting...
    Mar 22 02:41:22	check_reload_status: Starting packages
    Mar 22 02:41:22	php: rc.newwanip: pfSense package system has detected an ip change 69.x.x.146 -> 69.x.x.146 ... Restarting packages.
    Mar 22 02:41:22	dhcp6c[17105]: client6_recvreply: status code: success
    Mar 22 02:41:22	dhcp6c[17105]: dhcp6_get_options: unknown or unexpected DHCP6 option opt_20, len 0
    Mar 22 02:41:21	dhcp6c[17105]: dhcp6_get_options: unknown or unexpected DHCP6 option opt_20, len 0
    Mar 22 02:41:20	check_reload_status: rc.newwanip starting ovpns2
    Mar 22 02:41:20	kernel: ovpns2: link state changed to UP
    Mar 22 02:41:20	php: rc.newwanip: Creating rrd update script
    Mar 22 02:41:19	kernel: ovpns2: link state changed to DOWN
    Mar 22 02:41:18	php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Mar 22 02:41:17	check_reload_status: Reloading filter
    Mar 22 02:41:17	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Mar 22 02:41:17	check_reload_status: Restarting ipsec tunnels
    Mar 22 02:41:17	check_reload_status: updating dyndns WAN_DHCP6
    Mar 22 02:41:17	dhcp6c[42903]: check_exit: exiting
    Mar 22 02:41:17	php: rc.newwanipv6: rc.newwanipv6: Failed to update wan IPv6, restarting...
    Mar 22 02:41:17	check_reload_status: Reloading filter
    Mar 22 02:41:16	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Mar 22 02:41:16	check_reload_status: Restarting ipsec tunnels
    Mar 22 02:41:16	check_reload_status: updating dyndns WAN_PPPOE
    Mar 22 02:41:14	dhcp6c[42903]: client6_recvadvert: XID mismatch
    Mar 22 02:41:14	dhcp6c[42903]: dhcp6_get_options: unknown or unexpected DHCP6 option opt_20, len 0
    Mar 22 02:41:13	dhcp6c[42903]: client6_recvadvert: XID mismatch
    Mar 22 02:41:13	dhcp6c[42903]: dhcp6_get_options: unknown or unexpected DHCP6 option opt_20, len 0
    Mar 22 02:41:13	dhcp6c[17099]: client6_init: skip opening control port
    Mar 22 02:41:13	dhcp6c[17099]: client6_init: failed initialize control message authentication
    Mar 22 02:41:13	dhcp6c[17099]: dhcp6_ctl_authinit: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
    Mar 22 02:41:11	php: rc.newwanip: ROUTING: setting default route to 206.248.154.103
    Mar 22 02:40:08	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:08	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:08	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:08	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:07	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:07	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:07	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:04	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:00	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:00	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:00	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:40:00	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:59	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:59	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:59	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:52	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:52	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:52	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:52	check_reload_status: Rewriting resolv.conf
    Mar 22 02:39:51	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (wan).
    Mar 22 02:39:50	check_reload_status: Rewriting resolv.conf
    Mar 22 02:39:46	php: rc.start_packages: Restarting/Starting all packages.
    Mar 22 02:39:45	kernel: pid 68419 (php), uid 0, was killed: out of swap space
    Mar 22 02:39:43	php: rc.start_packages: Restarting/Starting all packages.
    Mar 22 02:39:33	check_reload_status: Starting packages
    Mar 22 02:39:33	php: rc.newwanip: pfSense package system has detected an ip change -> 172.16.8.1 ... Restarting packages.
    Mar 22 02:39:33	check_reload_status: Reloading filter
    Mar 22 02:39:33	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.8.1) (interface: ) (real interface: ovpns2).
    Mar 22 02:39:33	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
    Mar 22 02:39:31	check_reload_status: Starting packages
    Mar 22 02:39:31	php: rc.newwanip: pfSense package system has detected an ip change -> 172.16.8.1 ... Restarting packages.
    Mar 22 02:39:31	check_reload_status: Reloading filter
    Mar 22 02:39:31	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.8.1) (interface: ) (real interface: ovpns2).
    Mar 22 02:39:30	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
    Mar 22 02:39:29	kernel: vr1: link state changed to UP
    Mar 22 02:39:29	check_reload_status: Linkup starting vr1
    Mar 22 02:39:24	kernel: vr1: Using force reset command.
    Mar 22 02:39:24	kernel: vr1: vr_stop: Rx shutdown error
    Mar 22 02:39:24	kernel: vr1: restarting
    Mar 22 02:39:23	check_reload_status: Linkup starting vr1
    Mar 22 02:39:23	kernel: vr1: link state changed to DOWN
    Mar 22 02:39:23	kernel: vr1: vr_miibus_statchg: Tx/Rx shutdown error -- resetting
    Mar 22 02:39:23	kernel: vr1: vr_miibus_statchg: Tx/Rx shutdown error -- resetting
    Mar 22 02:39:18	check_reload_status: rc.newwanip starting ovpns2
    Mar 22 02:39:17	kernel: ovpns2: link state changed to UP
    Mar 22 02:39:16	kernel: ovpns2: link state changed to DOWN
    Mar 22 02:39:16	check_reload_status: rc.newwanip starting ovpns2
    Mar 22 02:39:15	kernel: ovpns2: link state changed to UP
    Mar 22 02:39:15	php: rc.openvpn: OpenVPN: Resync server2 Personal and friends
    Mar 22 02:39:15	check_reload_status: Reloading filter
    Mar 22 02:39:14	php: rc.dyndns.update: phpDynDNS (ut.niagara-it.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Mar 22 02:39:14	check_reload_status: Reloading filter
    Mar 22 02:39:14	kernel: ovpns2: link state changed to DOWN
    Mar 22 02:39:13	php: rc.dyndns.update: phpDynDNS (niagara-it.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Mar 22 02:39:12	php: rc.openvpn: OpenVPN: Resync server2 Personal and friends
    Mar 22 02:39:12	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
    Mar 22 02:39:12	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP6.
    Mar 22 02:39:12	kernel: pid 28853 (php), uid 0, was killed: out of swap space
    Mar 22 02:39:00	check_reload_status: Reloading filter
    Mar 22 02:39:00	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Mar 22 02:39:00	check_reload_status: Restarting ipsec tunnels
    Mar 22 02:39:00	check_reload_status: updating dyndns WAN_DHCP6
    Mar 22 02:39:00	check_reload_status: Reloading filter
    Mar 22 02:39:00	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Mar 22 02:39:00	check_reload_status: Restarting ipsec tunnels
    Mar 22 02:39:00	check_reload_status: updating dyndns WAN_PPPOE
    Mar 22 02:38:44	check_reload_status: Linkup starting vr1
    Mar 22 02:38:44	kernel: vr1: link state changed to UP
    Mar 22 02:38:42	kernel: vr1: link state changed to DOWN
    Mar 22 02:38:42	check_reload_status: Linkup starting vr1
    
    


  • It seams that your modem crashes ("Could not find IPv4 gateway for interface (wan)") at high load.

    Maybe it helps to shut off flow control at WAN interface of pfSense by adding "hw.vr1.fc_setting=0" to your /boot/loader.conf.local. If it doesn't exist yet, create it at first.


Log in to reply