Issues with pfSense at a LAN over the weekend.
-
This last weekend me and a few friends put on one of our semi regular LAN events and chose to utilize pfSense on our VMWare ESXi server box as our router/firewall solution. This was our first time using pfSense. It was recommended to us via the ISP tech that was coming and bringing his 2 uncapped modems. So we get everything up and running and its running great. pfSense is configured to do load balancing over the two modems whenever there is high traffic or packet loss on the primary modem, and the modems are setup with their static IPs and gateways. Everything is running smooth as people start to arrive. We had around 250 people at this event.
Two hours later everyone starts to play their respective games and everything goes south. Our pfSense box just seemingly stops working correctly. Clients can only intermittently access anything and when they are down for HTTP traffic Steam and Skype still work for them. When I looked at the resources for the box everything was running well under capacity even the state table which was set at 98000. We looked at the firewall logs and saw a ton of requests being blocked.
Eventually after troubleshooting it for way too long we switched over to LAN only for games that supported it and ran everything else off the house internet on their cable modem at the venue.
My question is, for future events, what do I look for in finding the cause of the failure. It looks like pfSense was well under capacity for the box and was nowhere near state table limits, but looked like it was still rejecting packets over the firewall even with generic allow everything rules in place for every LAN interface. I still have all the logs for the system so I can look up anything from the prior weekend to try and figure out what went wrong.
-
We looked at the firewall logs and saw a ton of requests being blocked.
Got any examples of blocked traffic? Traffic originating from the LAN should not be blocked by default. If you click on a block log, you can quickly make a rule that allows that traffic.
-
It was literally all HTTP traffic from what I saw, but it was intermittent. League players couldn't connect to their games. No websites work. I couldn't even ping google public DNS at times when it was acting up.
-
Did you make note of exactly what the firewall log showed? With a lot of blocked traffic on LAN, it makes me think maybe you had an IP conflict so only parts of each session hit that system, which would lead to a lot of out of state blocked traffic.
If you had a LAN IP of 192.168.1.1 that becomes more likely, since that's such a common IP on various things.
Anything in the system log at the time?
-
Jul 11 08:02:59 check_reload_status: Restarting ipsec tunnels Jul 11 08:03:00 kernel: arprequest: cannot find matching address Jul 11 08:03:01 check_reload_status: updating dyndns opt2 Jul 11 08:03:02 php-fpm[33132]: /rc.dyndns.update: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:03:02 php-fpm[33132]: /rc.dyndns.update: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:03:02 php-fpm[33132]: /rc.dyndns.update: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:03:03 kernel: arprequest: cannot find matching address Jul 11 08:03:03 php-fpm[29916]: /interfaces.php: Removing static route for monitor 8.8.4.4 and adding a new route through 71.40.166.1 Jul 11 08:03:03 snmpd[54098]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 08:03:03 snmpd[54098]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 08:03:03 php-fpm[29916]: /interfaces.php: Removing static route for monitor 8.8.8.8 and adding a new route through 97.76.90.25 Jul 11 08:03:03 check_reload_status: Reloading filter Jul 11 08:03:03 php-fpm[29916]: /interfaces.php: Creating rrd update script Jul 11 08:03:04 php-fpm[53270]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:03:04 php-fpm[53270]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:03:04 php-fpm[53270]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:03:06 kernel: arprequest: cannot find matching address Jul 11 08:03:09 kernel: arprequest: cannot find matching address Jul 11 08:03:12 kernel: arprequest: cannot find matching address Jul 11 08:03:15 kernel: arprequest: cannot find matching address Jul 11 08:03:18 kernel: arprequest: cannot find matching address Jul 11 08:03:21 kernel: arprequest: cannot find matching address Jul 11 08:03:24 kernel: arprequest: cannot find matching address Jul 11 08:03:27 kernel: arprequest: cannot find matching address Jul 11 08:03:30 kernel: arprequest: cannot find matching address Jul 11 08:03:33 kernel: arprequest: cannot find matching address Jul 11 08:03:36 kernel: arprequest: cannot find matching address Jul 11 08:03:39 kernel: arprequest: cannot find matching address Jul 11 08:03:42 kernel: arprequest: cannot find matching address Jul 11 08:03:45 kernel: arprequest: cannot find matching address Jul 11 08:03:48 kernel: arprequest: cannot find matching address Jul 11 08:03:51 kernel: arprequest: cannot find matching address Jul 11 08:03:54 kernel: arprequest: cannot find matching address Jul 11 08:03:57 kernel: arprequest: cannot find matching address Jul 11 08:04:00 kernel: arprequest: cannot find matching address Jul 11 08:04:03 kernel: arprequest: cannot find matching address Jul 11 08:04:06 kernel: arprequest: cannot find matching address Jul 11 08:04:07 check_reload_status: Syncing firewall Jul 11 08:04:07 check_reload_status: Reloading filter Jul 11 08:04:08 php-fpm[4878]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:04:08 php-fpm[4878]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:04:08 php-fpm[4878]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:04:09 kernel: arprequest: cannot find matching address Jul 11 08:04:12 kernel: arprequest: cannot find matching address Jul 11 08:04:15 kernel: arprequest: cannot find matching address Jul 11 08:04:18 kernel: arprequest: cannot find matching address Jul 11 08:04:21 kernel: arprequest: cannot find matching address Jul 11 08:04:24 kernel: arprequest: cannot find matching address Jul 11 08:04:27 kernel: arprequest: cannot find matching address Jul 11 08:04:30 kernel: arprequest: cannot find matching address Jul 11 08:04:33 kernel: arprequest: cannot find matching address Jul 11 08:04:36 kernel: arprequest: cannot find matching address Jul 11 08:04:39 kernel: arprequest: cannot find matching address Jul 11 08:04:42 kernel: arprequest: cannot find matching address Jul 11 08:04:45 kernel: arprequest: cannot find matching address Jul 11 08:04:48 kernel: arprequest: cannot find matching address Jul 11 08:04:51 kernel: arprequest: cannot find matching address Jul 11 08:04:54 kernel: arprequest: cannot find matching address Jul 11 08:04:57 kernel: arprequest: cannot find matching address Jul 11 08:05:00 kernel: arprequest: cannot find matching address Jul 11 08:05:03 kernel: arprequest: cannot find matching address Jul 11 08:05:05 check_reload_status: Syncing firewall Jul 11 08:05:06 kernel: arprequest: cannot find matching address Jul 11 08:05:07 check_reload_status: Restarting ipsec tunnels Jul 11 08:05:09 kernel: arprequest: cannot find matching address Jul 11 08:05:09 php-fpm[4878]: /interfaces.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em0' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.8 Copyright 2004-2015 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ bad range, address 10.10.10.20 not in subnet 192.168.0.0 netmask 255.255.255.0 If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If you intend to request help from the dhcp-bugs at isc.org mailing list, please read the section on the README about submitting bug reports and requests for help. Please do not under any circumstances send requests for help directly to the authors of this software - please send them to the appropr Jul 11 08:05:09 check_reload_status: updating dyndns lan Jul 11 08:05:10 php-fpm[92476]: /rc.dyndns.update: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:05:10 php-fpm[92476]: /rc.dyndns.update: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:05:10 php-fpm[92476]: /rc.dyndns.update: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:05:11 php-fpm[4878]: /interfaces.php: Removing static route for monitor 8.8.4.4 and adding a new route through 71.40.166.1 Jul 11 08:05:11 snmpd[99965]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 08:05:11 snmpd[99965]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 08:05:11 php-fpm[4878]: /interfaces.php: Removing static route for monitor 8.8.8.8 and adding a new route through 97.76.90.25 Jul 11 08:05:11 check_reload_status: Reloading filter Jul 11 08:05:11 php-fpm[4878]: /interfaces.php: Creating rrd update script Jul 11 08:05:12 kernel: arprequest: cannot find matching address Jul 11 08:05:12 php-fpm[92476]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:05:12 php-fpm[92476]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:05:12 php-fpm[92476]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:05:15 kernel: arprequest: cannot find matching address Jul 11 08:05:18 kernel: arprequest: cannot find matching address Jul 11 08:05:21 kernel: arprequest: cannot find matching address Jul 11 08:05:24 kernel: arprequest: cannot find matching address Jul 11 08:05:27 kernel: arprequest: cannot find matching address Jul 11 08:05:30 kernel: arprequest: cannot find matching address Jul 11 08:05:33 kernel: arprequest: cannot find matching address Jul 11 08:05:36 kernel: arprequest: cannot find matching address Jul 11 08:05:39 kernel: arprequest: cannot find matching address Jul 11 08:05:42 kernel: arprequest: cannot find matching address Jul 11 08:05:45 kernel: arprequest: cannot find matching address Jul 11 08:05:49 kernel: arprequest: cannot find matching address Jul 11 08:05:52 kernel: arprequest: cannot find matching address Jul 11 08:05:55 kernel: arprequest: cannot find matching address Jul 11 08:05:57 php-fpm[92476]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.110 Jul 11 08:05:57 php-fpm[92476]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.110 Jul 11 08:05:58 kernel: arprequest: cannot find matching address Jul 11 08:06:01 kernel: arprequest: cannot find matching address Jul 11 08:06:04 kernel: arprequest: cannot find matching address Jul 11 08:06:07 kernel: arprequest: cannot find matching address Jul 11 08:06:10 kernel: arprequest: cannot find matching address Jul 11 08:06:13 kernel: arprequest: cannot find matching address Jul 11 08:06:16 kernel: arprequest: cannot find matching address Jul 11 08:06:17 check_reload_status: Syncing firewall Jul 11 08:06:19 kernel: arprequest: cannot find matching address Jul 11 08:06:22 kernel: arprequest: cannot find matching address Jul 11 08:06:25 kernel: arprequest: cannot find matching address Jul 11 08:06:28 kernel: arprequest: cannot find matching address Jul 11 08:06:31 kernel: arprequest: cannot find matching address Jul 11 08:06:34 kernel: arprequest: cannot find matching address Jul 11 08:06:37 kernel: arprequest: cannot find matching address Jul 11 08:06:40 kernel: arprequest: cannot find matching address Jul 11 08:06:43 kernel: arprequest: cannot find matching address Jul 11 08:06:46 kernel: arprequest: cannot find matching address Jul 11 08:06:46 check_reload_status: Syncing firewall Jul 11 08:06:48 check_reload_status: Restarting ipsec tunnels Jul 11 08:06:50 check_reload_status: updating dyndns opt2 Jul 11 08:06:51 php-fpm[59202]: /rc.dyndns.update: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:06:51 php-fpm[59202]: /rc.dyndns.update: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:06:51 php-fpm[59202]: /rc.dyndns.update: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:06:52 php-fpm[49805]: /interfaces.php: Removing static route for monitor 8.8.4.4 and adding a new route through 71.40.166.1 Jul 11 08:06:52 snmpd[65290]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 08:06:52 snmpd[65290]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 08:06:52 php-fpm[49805]: /interfaces.php: Removing static route for monitor 8.8.8.8 and adding a new route through 97.76.90.25 Jul 11 08:06:52 check_reload_status: Reloading filter Jul 11 08:06:52 php-fpm[49805]: /interfaces.php: Creating rrd update script Jul 11 08:06:53 php-fpm[59202]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:06:53 php-fpm[59202]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:06:53 php-fpm[59202]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:09:41 check_reload_status: Syncing firewall Jul 11 08:09:43 check_reload_status: Reloading filter Jul 11 08:09:44 php-fpm[21939]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:09:44 php-fpm[21939]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:09:44 php-fpm[21939]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:09:57 check_reload_status: Syncing firewall Jul 11 08:10:10 check_reload_status: Syncing firewall Jul 11 08:10:30 check_reload_status: Syncing firewall Jul 11 08:10:37 check_reload_status: Syncing firewall Jul 11 08:10:38 check_reload_status: Syncing firewall Jul 11 08:10:40 check_reload_status: Syncing firewall Jul 11 08:10:42 check_reload_status: Syncing firewall Jul 11 08:10:43 check_reload_status: Syncing firewall Jul 11 08:10:44 check_reload_status: Syncing firewall Jul 11 08:10:46 check_reload_status: Syncing firewall Jul 11 08:10:47 check_reload_status: Syncing firewall Jul 11 08:10:50 check_reload_status: Reloading filter Jul 11 08:10:51 php-fpm[54735]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:10:51 php-fpm[54735]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:10:51 php-fpm[54735]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:11:20 check_reload_status: Syncing firewall Jul 11 08:11:37 check_reload_status: Syncing firewall Jul 11 08:11:39 check_reload_status: Syncing firewall Jul 11 08:11:40 check_reload_status: Syncing firewall Jul 11 08:11:42 check_reload_status: Syncing firewall Jul 11 08:11:43 check_reload_status: Syncing firewall Jul 11 08:11:44 check_reload_status: Reloading filter Jul 11 08:11:45 php-fpm[74823]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:11:45 php-fpm[74823]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:11:45 php-fpm[74823]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:11:59 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:11:59 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:11:59 php-fpm[74823]: /firewall_rules_edit.php: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:12:06 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:12:06 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:12:06 php-fpm[74823]: /firewall_rules_edit.php: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:12:20 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:12:20 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:12:20 php-fpm[74823]: /firewall_rules_edit.php: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:12:20 check_reload_status: Syncing firewall Jul 11 08:12:28 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:12:28 php-fpm[74823]: /firewall_rules_edit.php: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:12:28 php-fpm[74823]: /firewall_rules_edit.php: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:12:35 check_reload_status: Reloading filter Jul 11 08:12:36 php-fpm[74823]: /rc.filter_configure_sync: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:12:36 php-fpm[74823]: /rc.filter_configure_sync: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:12:36 php-fpm[74823]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:17:27 php-fpm[97846]: /diag_ping.php: MONITOR: GW_WAN is down, omitting from routing group CptPlanet Jul 11 08:17:27 php-fpm[97846]: /diag_ping.php: MONITOR: WAN2 is down, omitting from routing group CptPlanet Jul 11 08:17:27 php-fpm[97846]: /diag_ping.php: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 08:19:14 check_reload_status: Syncing firewall Jul 11 08:19:21 check_reload_status: Syncing firewall Jul 11 08:20:34 check_reload_status: Syncing firewall Jul 11 08:20:35 dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such process. Jul 11 08:20:35 check_reload_status: Reloading filter Jul 11 08:20:36 dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such process. Jul 11 08:20:47 check_reload_status: Syncing firewall Jul 11 08:21:20 check_reload_status: Syncing firewall Jul 11 08:22:18 check_reload_status: Syncing firewall Jul 11 08:22:21 php-fpm[6411]: /system.php: NTPD is starting up. Jul 11 08:22:21 check_reload_status: Reloading filter Jul 11 08:23:14 check_reload_status: Syncing firewall Jul 11 14:29:45 kernel: arp: 192.168.1.11 moved from 74:d0:2b:7d:ae:38 to 00:ee:bd:c1:ec:b9 on em1 Jul 11 14:29:51 kernel: arp: 192.168.1.11 moved from 00:ee:bd:c1:ec:b9 to 74:d0:2b:7d:ae:38 on em1 Jul 11 14:45:24 kernel: arp: 192.168.0.2 moved from 08:62:66:4a:31:d0 to d8:cb:8a:14:b7:b6 on em0 Jul 11 14:45:31 kernel: arp: 192.168.0.2 moved from d8:cb:8a:14:b7:b6 to 08:62:66:4a:31:d0 on em0 Jul 11 15:28:48 kernel: arp: 192.168.1.114 moved from ac:7b:a1:45:f8:a8 to d8:cb:8a:10:a8:33 on em1 Jul 11 15:28:59 kernel: arp: 192.168.1.114 moved from d8:cb:8a:10:a8:33 to ac:7b:a1:45:f8:a8 on em1 Jul 11 15:29:01 kernel: arp: 192.168.1.114 moved from ac:7b:a1:45:f8:a8 to d8:cb:8a:10:a8:33 on em1 Jul 11 15:29:07 kernel: arp: 192.168.1.114 moved from d8:cb:8a:10:a8:33 to ac:7b:a1:45:f8:a8 on em1 Jul 11 16:04:50 php-fpm[87334]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.2 Jul 11 16:04:50 php-fpm[87334]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.2 Jul 11 16:43:17 snmpd[65290]: sendmsg: Can't assign requested address Jul 11 16:54:05 check_reload_status: Syncing firewall Jul 11 16:54:08 check_reload_status: Restarting ipsec tunnels Jul 11 16:54:10 check_reload_status: updating dyndns opt2 Jul 11 16:54:12 snmpd[68261]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 16:54:12 snmpd[68261]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 16:54:12 check_reload_status: Reloading filter Jul 11 16:54:12 php-fpm[58816]: /interfaces.php: Creating rrd update script Jul 11 16:54:12 php-fpm[58816]: /interfaces.php: ROUTING: setting default route to 97.76.90.25 Jul 11 16:55:16 check_reload_status: Syncing firewall Jul 11 16:55:17 check_reload_status: Restarting ipsec tunnels Jul 11 16:55:20 php-fpm[61263]: /interfaces.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em0 em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.8 Copyright 2004-2015 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ bad range, address 192.168.1.2 not in subnet 192.168.2.0 netmask 255.255.254.0 If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If you intend to request help from the dhcp-bugs at isc.org mailing list, please read the section on the README about submitting bug reports and requests for help. Please do not under any circumstances send requests for help directly to the authors of this software - please send them to the ap Jul 11 16:55:20 check_reload_status: updating dyndns opt2 Jul 11 16:55:22 snmpd[13485]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 16:55:22 snmpd[13485]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 16:55:24 check_reload_status: Reloading filter Jul 11 16:55:24 php-fpm[61263]: /interfaces.php: Creating rrd update script Jul 11 16:55:48 check_reload_status: Syncing firewall Jul 11 16:59:55 check_reload_status: Syncing firewall Jul 11 17:00:26 php-fpm[27071]: /system.php: NTPD is starting up. Jul 11 17:00:26 check_reload_status: Reloading filter Jul 11 17:02:39 check_reload_status: Syncing firewall Jul 11 17:02:48 check_reload_status: Syncing firewall Jul 11 17:02:50 check_reload_status: Reloading filter Jul 11 17:06:32 check_reload_status: Syncing firewall Jul 11 17:06:35 check_reload_status: Reloading filter Jul 11 17:06:41 check_reload_status: Syncing firewall Jul 11 17:06:43 check_reload_status: Reloading filter Jul 11 17:06:55 check_reload_status: Syncing firewall Jul 11 17:06:57 check_reload_status: Reloading filter Jul 11 17:07:08 check_reload_status: Syncing firewall Jul 11 17:07:10 check_reload_status: Reloading filter Jul 11 17:08:43 check_reload_status: Syncing firewall Jul 11 17:08:50 check_reload_status: Syncing firewall Jul 11 19:22:51 php-fpm[4776]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.2 Jul 11 19:22:51 php-fpm[4776]: /index.php: Successful login for user 'jlindelof' from: 192.168.0.2 Jul 11 19:22:58 check_reload_status: Syncing firewall Jul 11 19:23:01 check_reload_status: Syncing firewall Jul 11 19:25:43 kernel: arp: 192.168.0.19 moved from d4:3d:7e:b5:2d:b0 to d4:3d:7e:9a:e1:ad on em0 Jul 11 19:26:00 kernel: arp: 192.168.0.19 moved from d4:3d:7e:9a:e1:ad to d4:3d:7e:b5:2d:b0 on em0 Jul 11 19:36:04 snmpd[63404]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 19:36:04 snmpd[63404]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 19:36:06 php-fpm[44264]: /interfaces.php: Creating rrd update script Jul 11 19:36:06 check_reload_status: Reloading filter Jul 11 19:36:07 php-fpm[62528]: /rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: CptPlanet) Jul 11 19:36:08 php-fpm[62528]: /rc.filter_configure_sync: GATEWAYS: Group CptPlanet did not have any gateways up on tier 2! Jul 11 19:36:08 php-fpm[62528]: /rc.filter_configure_sync: The gateway: GW_WAN is invalid or unknown, not using it. Jul 11 19:36:08 php-fpm[62528]: /rc.filter_configure_sync: The gateway: WAN2 is invalid or unknown, not using it. Jul 11 20:05:46 check_reload_status: Syncing firewall Jul 11 20:06:48 check_reload_status: Restarting ipsec tunnels Jul 11 20:06:50 check_reload_status: updating dyndns wan Jul 11 20:06:52 snmpd[28513]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 20:06:52 snmpd[28513]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 20:06:54 check_reload_status: Reloading filter Jul 11 20:06:54 php-fpm[86685]: /interfaces.php: Creating rrd update script Jul 11 20:06:55 php-fpm[92754]: /rc.filter_configure_sync: The gateway: WAN2 is invalid or unknown, not using it. Jul 11 20:12:47 check_reload_status: rc.newwanip starting em2 Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: rc.newwanip: Info: starting on em2. Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.0.3) (interface: WAN[wan]) (real interface: em2). Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: The command '/sbin/route change -host 8.8.8.8 71.40.166.1' returned exit code '1', the output was 'route: writing to routing socket: No such process route: writing to routing socket: Network is unreachable change host 8.8.8.8: gateway 71.40.166.1 fib 0: Network is unreachable' Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: IP has changed, killing states on former IP 192.168.1.101. Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: The gateway: WAN2 is invalid or unknown, not using it. Jul 11 20:12:52 php-fpm[96069]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Jul 11 20:12:52 php-fpm[96069]: /rc.newwanip: Creating rrd update script Jul 11 20:12:54 snmpd[60872]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 20:12:54 snmpd[60872]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 20:12:54 php-fpm[96069]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.1.101 -> 192.168.0.3 - Restarting packages. Jul 11 20:12:54 check_reload_status: Starting packages Jul 11 20:12:55 php-fpm[35492]: /rc.start_packages: Restarting/Starting all packages.
theres the general sys logs for july 11th. nothing seems to jump out at me.
-
As I see it right you want to play with ~250 peoples over an OpenVPN or IPSec VPN connection?
Is this right?The next time it would be wise to take system you are more common with and not a newbie
that you can easily fix by your self.This is the really sad thing with VM stuff. In normal you would be ownig a gib box or powerful
hardware to really offer your friends a good throughput at all, but if now there is something
faulty in the ESXi set up or configuration will never know it. -
Jul 11 20:06:55 php-fpm[92754]: /rc.filter_configure_sync: The gateway: WAN2 is invalid or unknown, not using it. Jul 11 20:12:47 check_reload_status: rc.newwanip starting em2 Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: rc.newwanip: Info: starting on em2. Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.0.3) (interface: WAN[wan]) (real interface: em2). Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: The command '/sbin/route change -host 8.8.8.8 71.40.166.1' returned exit code '1', the output was 'route: writing to routing socket: No such process route: writing to routing socket: Network is unreachable change host 8.8.8.8: gateway 71.40.166.1 fib 0: Network is unreachable' Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: IP has changed, killing states on former IP 192.168.1.101. Jul 11 20:12:48 php-fpm[96069]: /rc.newwanip: The gateway: WAN2 is invalid or unknown, not using it. Jul 11 20:12:52 php-fpm[96069]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Jul 11 20:12:52 php-fpm[96069]: /rc.newwanip: Creating rrd update script Jul 11 20:12:54 snmpd[60872]: disk_OS_get_disks: adding device 'da0' to device list Jul 11 20:12:54 snmpd[60872]: disk_OS_get_disks: adding device 'cd0' to device list Jul 11 20:12:54 php-fpm[96069]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.1.101 -> 192.168.0.3 - Restarting packages. Jul 11 20:12:54 check_reload_status: Starting packages Jul 11 20:12:55 php-fpm[35492]: /rc.start_packages: Restarting/Starting all packages.
theres the general sys logs for july 11th. nothing seems to jump out at me.
it looks like you WAN & LAN subnets are overlapping and or having a invalid config ….
-
Definitely a broken network there. The biggest issue is you look to have had your WANs and LAN plugged into the same broadcast domain, and the box was picking up its own DHCP leases for its WANs. Your WANs must be completely separate from your LAN, and completely separate from each other.
Also looks like something was serving DHCP on 192.168.1.x and 192.168.0.x, likely separate /24 scopes from the looks of it, which is probably indicative of some other problem.
Several IP conflicts there also. Such as "arp: 192.168.0.2 moved from d8:cb:8a:14:b7:b6 to 08:62:66:4a:31:d0 on em0"
-
Looking though all the errors, I'm amazed it worked at all.
-
@KOM:
Looking though all the errors, I'm amazed it worked at all.
Indeed, I'm more surprised it ever worked at all than that it stopped working, given how broken the network was.