Internet drops for 2 mins a couple times a day
-
Internet drops for ~2 minutes and comes back multiple times daily. So far today it was down sometime between 1am and 2am, then again from 12 to 1pm. On the 26th it was down from 9pm to 10pm. I'm testing this by having my server ping once a second and logging it to a file every hour. The LAN works fine during this period, and I can log into pfsense when it's down.
The issue seems to be randomly distributed, averaging about twice a day. I have a Rogers internet connection (Canada) and their "Rocket" AC cable modem/router combo, which is in bridge mode. I changed from the PCI and onboard NICs, I don't think they are the issue.
PFSense box Hardware:
HP dc5800 SFF
Core 2 Duo E6550, 1x2GB DDR2, Q33 Express (PCIE 1.0)
Sandisk mSATA SSD
Intel onboard LAN (not assigned)
Intel dual-port Gb PCIE 2.0 x1 card
Intel PCI NIC (not assigned, not installed)DMESG output:
Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.3-RELEASE-p9 #1 5fc1b19(RELENG_2_3_2): Tue Sep 27 12:26:06 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz (2327.54-MHz K8-class CPU) Origin="GenuineIntel" Id=0x6fb Family=0x6 Model=0xf Stepping=11 Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0xe3fd <sse3,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>VT-x: (disabled in BIOS) HLT,PAUSE TSC: P-state invariant, performance statistics real memory = 2151677952 (2052 MB) avail memory = 2019880960 (1926 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <compaq bearlake=""> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random: <software, yarrow=""> initialized ioapic0: Changing APIC ID to 1 ioapic0 <version 2.0=""> irqs 0-23 on motherboard wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80620930, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806209e0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620a90, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647c30, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647ce0, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647d90, 0) error 1 netmap: loaded module kbd1 at kbdmux0 cryptosoft0: <software crypto=""> on motherboard padlock0: No ACE support. acpi0: <hpqoem slic-bpc=""> on motherboard acpi0: Power Button (fixed) cpu0: <acpi cpu=""> on acpi0 cpu1: <acpi cpu=""> on acpi0 attimer0: <at timer=""> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <at realtime="" clock=""> port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet1: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf808-0xf80b on acpi0 hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0 device_attach: hpet0 attach returned 12 pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0 pci0: <acpi pci="" bus=""> on pcib0 vgapci0: <vga-compatible display=""> port 0x2240-0x2247 mem 0xf0a00000-0xf0a7ffff,0xe0000000-0xefffffff,0xf0900000-0xf09fffff irq 16 at device 2.0 on pci0 agp0: <intel q33="" svga="" controller=""> on vgapci0 agp0: aperture size is 256M, detected 6140k stolen memory vgapci0: Boot video device pci0: <simple comms=""> at device 3.0 (no driver attached) em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k=""> port 0x2100-0x211f mem 0xf0a80000-0xf0a9ffff,0xf0aa4000-0xf0aa4fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: 00:21:5a:76:95:62 em0: netmap queues/slots: TX 1/1024, RX 1/1024 uhci0: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x2120-0x213f irq 20 at device 26.0 on pci0 usbus0 on uhci0 uhci1: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x2140-0x215f irq 21 at device 26.1 on pci0 usbus1 on uhci1 uhci2: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x2160-0x217f irq 22 at device 26.2 on pci0 usbus2 on uhci2 ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller=""> mem 0xf0aa5000-0xf0aa53ff irq 22 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 hdac0: <intel 82801i="" hda="" controller=""> mem 0xf0aa0000-0xf0aa3fff irq 21 at device 27.0 on pci0 pcib1: <acpi pci-pci="" bridge=""> irq 20 at device 28.0 on pci0 pci32: <acpi pci="" bus=""> on pcib1 igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1100-0x111f mem 0xf0800000-0xf081ffff,0xf0000000-0xf03fffff,0xf0840000-0xf0843fff irq 16 at device 0.0 on pci32 igb0: Using MSIX interrupts with 3 vectors igb0: Ethernet address: 6c:b3:11:1b:0b:7e igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: netmap queues/slots: TX 2/1024, RX 2/1024 igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1120-0x113f mem 0xf0820000-0xf083ffff,0xf0400000-0xf07fffff,0xf0844000-0xf0847fff irq 17 at device 0.1 on pci32 igb1: Using MSIX interrupts with 3 vectors igb1: Ethernet address: 6c:b3:11:1b:0b:7f igb1: Bound queue 0 to cpu 0 igb1: Bound queue 1 to cpu 1 igb1: netmap queues/slots: TX 2/1024, RX 2/1024 pcib2: <acpi pci-pci="" bridge=""> irq 21 at device 28.1 on pci0 pci48: <acpi pci="" bus=""> on pcib2 uhci3: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x2180-0x219f irq 20 at device 29.0 on pci0 usbus4 on uhci3 uhci4: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x21a0-0x21bf irq 21 at device 29.1 on pci0 usbus5 on uhci4 uhci5: <intel 82801i="" (ich9)="" usb="" controller=""> port 0x21c0-0x21df irq 22 at device 29.2 on pci0 usbus6 on uhci5 ehci1: <intel 82801i="" (ich9)="" usb="" 2.0="" controller=""> mem 0xf0aa5400-0xf0aa57ff irq 20 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib3: <acpi pci-pci="" bridge=""> at device 30.0 on pci0 pci7: <acpi pci="" bus=""> on pcib3 isab0: <pci-isa bridge=""> at device 31.0 on pci0 isa0: <isa bus=""> on isab0 atapci0: <intel ich9="" sata300="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2200-0x220f,0x2210-0x221f irq 18 at device 31.2 on pci0 ata0: <ata channel=""> at channel 0 on atapci0 ata1: <ata channel=""> at channel 1 on atapci0 atapci1: <intel ich9="" sata300="" controller=""> port 0x2258-0x225f,0x2270-0x2273,0x2260-0x2267,0x2274-0x2277,0x2220-0x222f,0x2230-0x223f irq 18 at device 31.5 on pci0 ata2: <ata channel=""> at channel 0 on atapci1 ata3: <ata channel=""> at channel 1 on atapci1 acpi_button0: <power button=""> on acpi0 atkbdc0: <keyboard controller="" (i8042)=""> port 0x60,0x64 irq 1 on acpi0 atkbd0: <at keyboard=""> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 fdc0: <floppy drive="" controller=""> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 sc0: <system console=""> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <generic isa="" vga=""> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: <enhanced speedstep="" frequency="" control=""> on cpu0 est1: <enhanced speedstep="" frequency="" control=""> on cpu1 Timecounters tick every 1.000 msec random: unblocking device. hdacc0: <analog devices="" ad1884="" hda="" codec=""> at cad 0 on hdac0 hdaa0: <analog devices="" ad1884="" audio="" function="" group=""> at nid 1 on hdacc0 pcm0: <analog devices="" ad1884="" (rear="" analog)=""> at nid 18 and 21 on hdaa0 pcm1: <analog devices="" ad1884="" (front="" analog)=""> at nid 17 and 20 on hdaa0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: <intel> at usbus0 uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus0 ugen3.1: <intel> at usbus3 uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus3 ugen2.1: <intel> at usbus2 uhub2: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus2 ugen1.1: <intel> at usbus1 uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus1 ugen5.1: <intel> at usbus5 ugen4.1: <intel> at usbus4 ugen7.1: <intel> at usbus7 uhub4: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus7 ugen6.1: <intel> at usbus6 uhub5: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus5 uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus4 uhub7: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus6 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub7: 2 ports with 2 removable, self powered uhub1: 6 ports with 6 removable, self powered uhub4: 6 ports with 6 removable, self powered ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: <sandisk ssd="" i100="" 32gb="" 11.56.00=""> ACS-2 ATA SATA 3.x device ada0: Serial Number 121900107474 ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 512bytes) ada0: 30533MB (62533296 512 byte sectors) ada0: Previously was known as ad0 SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ufsid/57a7eecf422c4a11 [rw]... igb1: link state changed to UP igb0: link state changed to UP ugen0.2: <logitech> at usbus0 ukbd0: <logitech 0="" 2="" usb="" receiver,="" class="" 0,="" rev="" 2.00="" 24.01,="" addr=""> on usbus0 kbd2 at ukbd0 uhid0: <logitech 0="" 2="" usb="" receiver,="" class="" 0,="" rev="" 2.00="" 24.01,="" addr=""> on usbus0 igb0: link state changed to DOWN igb1: link state changed to DOWN igb1: link state changed to UP igb0: link state changed to UP coretemp0: <cpu on-die="" thermal="" sensors=""> on cpu0 coretemp1: <cpu on-die="" thermal="" sensors=""> on cpu1 igb0: link state changed to DOWN igb1: link state changed to DOWN igb0: link state changed to UP igb1: link state changed to UP pflog0: promiscuous mode enabled ugen0.2: <logitech> at usbus0 (disconnected) ukbd0: at uhub0, port 2, addr 2 (disconnected) uhid0: at uhub0, port 2, addr 2 (disconnected)</logitech></cpu></cpu></logitech></logitech></logitech></sandisk></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></analog></analog></analog></analog></enhanced></enhanced></generic></system></floppy></at></keyboard></power></ata></ata></intel></ata></ata></intel></isa></pci-isa></acpi></acpi></intel></intel></intel></intel></acpi></acpi></intel(r)></intel(r)></acpi></acpi></intel></intel></intel></intel></intel></intel(r)></simple></intel></vga-compatible></acpi></acpi></high></high></at></at></acpi></acpi></hpqoem></software></version></software,></compaq></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>The only thing I see in there is that could be an issue is the intel firmware not being used. I added the flag to my loader.conf and rebooted. What logs should I be looking at to see what's screwing up? Are there any I can enable?
-
Anything in Status - System Logs - General or Gateways at the time of the problem?
-
@KOM:
Anything in Status - System Logs - General or Gateways at the time of the problem?
I have a buttload of these in general, on a handful of ports around 19000-19100, udp and tcp:
Nov 28 17:38:44 xinetd 14611 readjusting service 19006-tcpThey filled the log so I can't see any more.
Gateways:
Nov 28 17:34:00 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr {myipIP} bind_addr {myipIP} identifier "WAN_DHCP " Nov 28 17:34:01 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr {myipIP} bind_addr {myipIP} identifier "WAN_DHCP "Note that I rebooted PFSense after applying that boot flat around 5:30pm, that could invalidate the gateway alarms
-
Showing one or two single log lines isn't that helpful. The first one is an nginx status message. The second is a dpinger status. Showing full log around the time of the problem would be more helpful.
-
@KOM:
Showing one or two single log lines isn't that helpful. The first one is an nginx status message. The second is a dpinger status. Showing full log around the time of the problem would be more helpful.
That was all I could see in the log, the nginx status took up all the spots, and the dpinger status was the only thing in that log on that date. I caught it just a second ago, here is what I got:
Status Logs -> System:
Nov 29 21:31:21 check_reload_status updating dyndns WAN_DHCP Nov 29 21:31:21 check_reload_status Restarting ipsec tunnels Nov 29 21:31:21 check_reload_status Restarting OpenVPN tunnels/interfaces Nov 29 21:31:21 check_reload_status Reloading filter Nov 29 21:31:22 xinetd 14611 Starting reconfiguration Nov 29 21:31:22 xinetd 14611 Swapping defaults Nov 29 21:31:22 xinetd 14611 readjusting service 6969-udp Nov 29 21:31:22 xinetd 14611 readjusting service 19000-tcpSystem Logs -> Gateways
Nov 29 21:31:21 dpinger WAN_DHCP 99.233.16.1: Alarm latency 19857us stddev 24963us loss 22%Done of the other logs had entries within a minute or two of the outage. Is the dyndns message saying my WAN ip was changed, or is it something about the DNS of PFSense? DNS Forwarder is disabled, Dynamic DNS is not configured. DNS resolver has no overrides, Access Lists are blank, and the tabs are shown here:
General Settings: http://i.imgur.com/spJ2yih.png
Advanced Settings:
Hide Identity: Yes Hide Version: Yes Prefetch Support: No Prefetch DNS Key Support: No Harden DNSSEC Data: Yes Message Cache Size: 4MB Outgoing TCP Buffers: 10 Incoming TCP Buffers: 10 EDNS Buffer Size: 4096 Number of Queries per Thread: 512 Jostle Timeout: 200 Maximum TTL for RRsets and Messages: 86400 Minimum TTL for RRsets and Messages: 0 TTL for Host Cache Entries: 15 minutes Number of Hosts to Cache: 10000 Unwanted Reply Threshold: Disabled Log Level: 1 Disable Auto-added Access Control: No Experimental Bit 0x20 Support: NoOther settings:
Disable DNS Rebind Check: No (default)
Alternate Hostnames for DNS rebinging and HTTP Referer checks: none
RAM Disks used for /tmp and /varGoogle tells me to look at my cable modem, I will factory reset it tonight. Is it fair to blame my cable modem based on the WAN_DHCP log entry?
-
I've seen this before where the connection bounces and everything restarts. It could be your cable modem. It could be the gateway monitoring is choking on something. You can disable gateway monitoring under System - Routing - Gateways and see if it makes a difference.
-
Killing gateway monitoring doesn't fix the issue, it just prevents the stuff I was getting when it drops from showing up in the General System logs tab. I have no entries to the General System logs tab since I disabled gateway monitoring, but have still had connectivity issues. Nothing in Gateways or DNS Resolver. I get entries like this every few minutes all the time in the Routing tab:
Dec 9 21:28:22 miniupnpd 45410 remove port mapping 34494 TCP because it has expired Dec 9 21:28:32 miniupnpd 45410 HTTP Connection from 192.168.1.10 closed unexpectedly Dec 9 21:28:32 miniupnpd 45410 HTTP Connection from 192.168.1.10 closed unexpectedly Dec 9 21:33:36 miniupnpd 45410 Unknown udp packet received from 192.168.1.23:54768 Dec 9 21:35:31 miniupnpd 45410 HTTP Connection from 192.168.1.19 closed unexpectedly Dec 9 21:35:31 miniupnpd 45410 HTTP Connection from 192.168.1.19 closed unexpectedly Dec 9 21:44:16 miniupnpd 45410 HTTP Connection from 192.168.1.102 closed unexpectedly Dec 9 21:44:16 miniupnpd 45410 HTTP Connection from 192.168.1.102 closed unexpectedly Dec 9 21:48:38 miniupnpd 45410 Unknown udp packet received from 192.168.1.23:54768 Dec 9 21:58:42 miniupnpd 45410 HTTP Connection from 192.168.1.10 closed unexpectedly Dec 9 21:58:42 miniupnpd 45410 HTTP Connection from 192.168.1.10 closed unexpectedly Dec 9 22:03:40 miniupnpd 45410 Unknown udp packet received from 192.168.1.23:54768 Dec 9 22:04:32 miniupnpd 45410 remove port mapping 36335 TCP because it has expired Dec 9 22:04:32 miniupnpd 45410 remove port mapping 36335 UDP because it has expired Dec 9 22:05:31 miniupnpd 45410 remove port mapping 40676 TCP because it has expiredI'm changing my ping script to report exactly when a packet is dropped, hopefully that helps. Is there anything else I can try to fix this? I'll replace the cable modem next week.
-
Not that I can think of.
-
How's the cable and keystones?
-
Nov 29 21:31:21 dpinger WAN_DHCP 99.233.16.1: Alarm latency 19857us stddev 24963us loss 22%Dunno, but 20+% packet loss is considered a down connection with dpinger default settings. You are having real issues with your ISP, it's not something that pfSense would be inventing. So yeah, disabling GW monitoring won't help obviously, the packets will still get lost and things won't work.
