After upgrading to 2.3.4, all my various NetGate devices cannot route LAN to LAN
-
So, I thought maybe this was just a bug with a specific firewall config, but this now exists on two entirely different configurations, one of which is a brand new config done on 2.3.2 on a Netgate 2440.
I have allow any to any at the top of the ruleset for all ipv4 traffic on both LAN1 and LAN2 interfaces, no gateway is specified for either interface (which would turn it into a WANish type port), and have checked the system logs and they look fine. Rebooted a couple of times just to make sure.
In summary, no traffic between LAN interfaces will route, nothing hits the firewall rule logs, it simply is blackholed. pings show up as leaving the LAN they are sourced from and simply disappear. The routing table looks fine.
This happens on a Netgate SG-2440 and Netgate SG-4860. These are both brand new.
The LAN1 and LAN2 ports are up, and internet connectivity works.
These are pretty simple single WAN (static IP), multi LAN setups.
-
Here's the routing log for the clean 2.3.2 config that was updated to 2.3.4 and then restored from backup config:
Time Process PID Message Jul 13 14:45:48 radvd 37949 removing /var/run/radvd.pid Jul 13 14:45:48 radvd 37949 sendmsg: Permission denied Jul 13 14:45:48 radvd 37949 sending stop adverts Jul 13 14:45:48 radvd 37949 Exiting, sigterm or sigint received. Jul 13 14:45:45 radvd 37949 sendmsg: Permission denied Jul 13 14:45:34 radvd 37949 resuming normal operation Jul 13 14:45:34 radvd 37949 sendmsg: Permission denied Jul 13 14:45:34 radvd 37949 no auto-selected prefix on interface igb3, disabling advertisements Jul 13 14:45:34 radvd 37949 attempting to reread config file Jul 13 14:45:32 radvd 37949 sendmsg: Permission denied Jul 13 14:45:27 radvd 37949 sendmsg: Permission denied
And here's the system log for the clean 2.3.2 config that was updated to 2.3.4 and then restored from backup config:
Only errors on this appear to be:
ppc0: cannot reserve I/O port range
open /dev/ttyv0: No such file or directory (though due note usb console work just fine)
rtsold 5743 <sendpacket>sendmsg on igb0: Permission deniedJul 13 14:49:05 login login on ttyu1 as root Jul 13 14:48:59 check_reload_status Reloading filter Jul 13 14:48:59 check_reload_status Restarting OpenVPN tunnels/interfaces Jul 13 14:48:59 check_reload_status Restarting ipsec tunnels Jul 13 14:48:59 check_reload_status updating dyndns WANGW Jul 13 14:48:51 login login on ttyu1 as root Jul 13 14:48:51 sshlockout 81263 sshlockout/webConfigurator v3.0 starting up Jul 13 14:48:51 getty 80807 open /dev/ttyv0: No such file or directory Jul 13 14:48:51 oob-firewall.company.com nginx: 2017/07/13 14:48:51 [error] 22720#100081: send() failed (54: Connection reset by peer) Jul 13 14:48:50 kernel done. Jul 13 14:48:50 php-fpm 21805 /rc.start_packages: Restarting/Starting all packages. Jul 13 14:48:50 syslogd kernel boot file is /boot/kernel/kernel Jul 13 14:48:50 syslogd exiting on signal 15 Jul 13 14:48:49 kernel ppc0: cannot reserve I/O port range Jul 13 14:48:49 kernel ichwd0 on isa0 Jul 13 14:48:49 kernel done. Jul 13 14:48:49 php-cgi rc.bootup: Creating rrd update script Jul 13 14:48:49 rtsold 5743 <sendpacket>sendmsg on igb0: Permission denied Jul 13 14:48:45 kernel .done. Jul 13 14:48:45 rtsold 5743 <sendpacket>sendmsg on igb0: Permission denied Jul 13 14:48:45 kernel done. Jul 13 14:48:45 check_reload_status Updating all dyndns Jul 13 14:48:44 php-cgi rc.bootup: NTPD is starting up. Jul 13 14:48:44 kernel done. Jul 13 14:48:44 kernel done. Jul 13 14:48:43 check_reload_status Linkup starting igb2 Jul 13 14:48:43 kernel igb2: link state changed to UP Jul 13 14:48:43 check_reload_status Reloading filter Jul 13 14:48:43 check_reload_status Restarting OpenVPN tunnels/interfaces Jul 13 14:48:43 check_reload_status Restarting ipsec tunnels Jul 13 14:48:43 check_reload_status updating dyndns WANGW Jul 13 14:48:42 kernel igb1: link state changed to UP Jul 13 14:48:42 check_reload_status Linkup starting igb1 Jul 13 14:48:42 kernel igb0: link state changed to UP Jul 13 14:48:42 check_reload_status Linkup starting igb0 Jul 13 14:48:42 kernel igb3: link state changed to UP Jul 13 14:48:42 check_reload_status Linkup starting igb3 Jul 13 14:48:41 rtsold 5743 <sendpacket>sendmsg on igb0: Permission denied Jul 13 14:48:40 php-cgi rc.bootup: ROUTING: setting default route to 55.55.55.55 Jul 13 14:48:40 kernel .done. Jul 13 14:48:39 kernel pflog0: promiscuous mode enabled Jul 13 14:48:39 kernel done. Jul 13 14:48:39 php-cgi rc.bootup: Resyncing OpenVPN instances. Jul 13 14:48:39 sshlockout 8800 sshlockout/webConfigurator v3.0 starting up Jul 13 14:48:39 sshd 8545 Server listening on 0.0.0.0 port 22. Jul 13 14:48:39 kernel done. Jul 13 14:48:39 sshd 8545 Server listening on :: port 22. Jul 13 14:48:39 kernel done. Jul 13 14:48:38 kernel done. Jul 13 14:48:38 php-cgi rc.bootup: Accept router advertisements on interface igb0 Jul 13 14:48:38 kernel aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm> on motherboard Jul 13 14:48:38 kernel padlock0: No ACE support. Jul 13 14:48:38 kernel WARNING: /: TRIM flag on fs but disk does not support TRIM Jul 13 14:48:38 kernel Trying to mount root from ufs:/dev/ufsid/593ad505f8ef9bb7 [rw]... Jul 13 14:48:38 kernel Timecounter "TSC" frequency 1750043085 Hz quality 1000 Jul 13 14:48:38 kernel SMP: AP CPU #1 Launched! Jul 13 14:48:38 kernel da0: quirks=0x2 <no_6_byte>Jul 13 14:48:38 kernel da0: 29184MB (59768832 512 byte sectors) Jul 13 14:48:38 kernel da0: 40.000MB/s transfers Jul 13 14:48:38 kernel da0: Serial Number 000000225001 Jul 13 14:48:38 kernel da0: <generic ultra="" hs-combo="" 1.98=""> Removable Direct Access SCSI device Jul 13 14:48:38 kernel da0 at umass-sim0 bus 0 scbus4 target 0 lun 0 Jul 13 14:48:38 kernel umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr=""> on usbus0 Jul 13 14:48:38 kernel ugen0.3: <generic> at usbus0 Jul 13 14:48:38 kernel uhub1: 4 ports with 4 removable, self powered Jul 13 14:48:38 kernel uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr=""> on usbus0 Jul 13 14:48:38 kernel ugen0.2: <vendor 0x8087=""> at usbus0 Jul 13 14:48:38 kernel uhub0: 8 ports with 8 removable, self powered Jul 13 14:48:38 kernel uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus0 Jul 13 14:48:38 kernel ugen0.1: <intel> at usbus0 Jul 13 14:48:38 kernel usbus0: 480Mbps High Speed USB v2.0 Jul 13 14:48:38 kernel random: unblocking device. Jul 13 14:48:38 kernel Timecounters tick every 1.000 msec Jul 13 14:48:38 kernel est1: <enhanced speedstep="" frequency="" control=""> on cpu1 Jul 13 14:48:38 kernel est0: <enhanced speedstep="" frequency="" control=""> on cpu0 Jul 13 14:48:38 kernel uart1: console (115200,n,8,1) Jul 13 14:48:38 kernel uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0 Jul 13 14:48:38 kernel uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 on isa0 Jul 13 14:48:38 kernel ppc0: cannot reserve I/O port range Jul 13 14:48:38 kernel orm0: <isa option="" rom=""> at iomem 0xc0000-0xc0fff on isa0 Jul 13 14:48:38 kernel isa0: <isa bus=""> on isab0 Jul 13 14:48:38 kernel isab0: <pci-isa bridge=""> at device 31.0 on pci0 Jul 13 14:48:38 kernel ahcich3: <ahci channel=""> at channel 1 on ahci1 Jul 13 14:48:38 kernel ahcich2: <ahci channel=""> at channel 0 on ahci1 Jul 13 14:48:38 kernel ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported Jul 13 14:48:38 kernel ahci1: <intel avoton="" ahci="" sata="" controller=""> port 0x10d0-0x10d7,0x10e8-0x10eb,0x10d8-0x10df,0x10ec-0x10ef,0x10a0-0x10bf mem 0xdfff4800-0xdfff4fff irq 16 at device 24.0 on pci0 Jul 13 14:48:38 kernel ahcich1: <ahci channel=""> at channel 1 on ahci0 Jul 13 14:48:38 kernel ahcich0: <ahci channel=""> at channel 0 on ahci0 Jul 13 14:48:38 kernel ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported Jul 13 14:48:38 kernel ahci0: <intel avoton="" ahci="" sata="" controller=""> port 0x10c0-0x10c7,0x10e0-0x10e3,0x10c8-0x10cf,0x10e4-0x10e7,0x1080-0x109f mem 0xdfff4000-0xdfff47ff irq 23 at device 23.0 on pci0 Jul 13 14:48:38 kernel usbus0 on ehci0 Jul 13 14:48:38 kernel usbus0: EHCI version 1.0 Jul 13 14:48:38 kernel ehci0: <intel avoton="" usb="" 2.0="" controller=""> mem 0xdfff5400-0xdfff57ff irq 22 at device 22.0 on pci0 Jul 13 14:48:38 kernel igb3: netmap queues/slots: TX 2/1024, RX 2/1024 Jul 13 14:48:38 kernel igb3: Bound queue 1 to cpu 1 Jul 13 14:48:38 kernel igb3: Bound queue 0 to cpu 0 Jul 13 14:48:38 kernel igb3: Ethernet address: 00:08:a2:0b:d0:37 Jul 13 14:48:38 kernel igb3: Using MSIX interrupts with 3 vectors Jul 13 14:48:38 kernel igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1060-0x107f mem 0xdffc0000-0xdffdffff,0xdfff0000-0xdfff3fff irq 21 at device 20.3 on pci0 Jul 13 14:48:38 kernel igb2: netmap queues/slots: TX 2/1024, RX 2/1024 Jul 13 14:48:38 kernel igb2: Bound queue 1 to cpu 1 Jul 13 14:48:38 kernel igb2: Bound queue 0 to cpu 0 Jul 13 14:48:38 kernel igb2: Ethernet address: 00:08:a2:0b:d0:36 Jul 13 14:48:38 kernel igb2: Using MSIX interrupts with 3 vectors Jul 13 14:48:38 kernel igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1040-0x105f mem 0xdffa0000-0xdffbffff,0xdffec000-0xdffeffff irq 20 at device 20.2 on pci0 Jul 13 14:48:38 kernel igb1: netmap queues/slots: TX 2/1024, RX 2/1024 Jul 13 14:48:38 kernel igb1: Bound queue 1 to cpu 1 Jul 13 14:48:38 kernel igb1: Bound queue 0 to cpu 0 Jul 13 14:48:38 kernel igb1: Ethernet address: 00:08:a2:0b:d0:35 Jul 13 14:48:38 kernel igb1: Using MSIX interrupts with 3 vectors Jul 13 14:48:38 kernel igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1020-0x103f mem 0xdff80000-0xdff9ffff,0xdffe8000-0xdffebfff irq 19 at device 20.1 on pci0 Jul 13 14:48:38 kernel igb0: netmap queues/slots: TX 2/1024, RX 2/1024 Jul 13 14:48:38 kernel igb0: Bound queue 1 to cpu 1 Jul 13 14:48:38 kernel igb0: Bound queue 0 to cpu 0 Jul 13 14:48:38 kernel igb0: Ethernet address: 00:08:a2:0b:d0:34 Jul 13 14:48:38 kernel igb0: Using MSIX interrupts with 3 vectors Jul 13 14:48:38 kernel igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1000-0x101f mem 0xdff60000-0xdff7ffff,0xdffe4000-0xdffe7fff irq 18 at device 20.0 on pci0 Jul 13 14:48:38 kernel pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached) Jul 13 14:48:38 kernel pci0: <processor> at device 11.0 (no driver attached) Jul 13 14:48:38 kernel pci4: <acpi pci="" bus=""> on pcib4 Jul 13 14:48:38 kernel pcib4: <acpi pci-pci="" bridge=""> mem 0xdff20000-0xdff3ffff irq 23 at device 4.0 on pci0 Jul 13 14:48:38 kernel pci3: <acpi pci="" bus=""> on pcib3 Jul 13 14:48:38 kernel pcib3: <acpi pci-pci="" bridge=""> mem 0xdff00000-0xdff1ffff irq 20 at device 3.0 on pci0 Jul 13 14:48:38 kernel pci2: <acpi pci="" bus=""> on pcib2 Jul 13 14:48:38 kernel pcib2: <acpi pci-pci="" bridge=""> mem 0xdfee0000-0xdfefffff irq 16 at device 2.0 on pci0 Jul 13 14:48:38 kernel pci1: <acpi pci="" bus=""> on pcib1 Jul 13 14:48:38 kernel pcib1: <acpi pci-pci="" bridge=""> mem 0xdfec0000-0xdfedffff irq 16 at device 1.0 on pci0 Jul 13 14:48:38 kernel pci0: <acpi pci="" bus=""> on pcib0 Jul 13 14:48:38 kernel pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0 Jul 13 14:48:38 kernel acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 Jul 13 14:48:38 kernel Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 Jul 13 14:48:38 kernel Event timer "i8254" frequency 1193182 Hz quality 100 Jul 13 14:48:38 kernel Timecounter "i8254" frequency 1193182 Hz quality 0 Jul 13 14:48:38 kernel attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Jul 13 14:48:38 kernel Event timer "RTC" frequency 32768 Hz quality 0 Jul 13 14:48:38 kernel atrtc0: <at realtime="" clock=""> port 0x70-0x77 on acpi0 Jul 13 14:48:38 kernel device_attach: hpet0 attach returned 6 Jul 13 14:48:38 kernel hpet0: invalid period Jul 13 14:48:38 kernel cpu1: <acpi cpu=""> on acpi0 Jul 13 14:48:38 kernel cpu0: <acpi cpu=""> on acpi0 Jul 13 14:48:38 kernel device_attach: hpet0 attach returned 6 Jul 13 14:48:38 kernel hpet0: invalid period Jul 13 14:48:38 kernel acpi0: Power Button (fixed) Jul 13 14:48:38 kernel acpi0: <core coreboot=""> on motherboard Jul 13 14:48:38 kernel padlock0: No ACE support. Jul 13 14:48:38 kernel cryptosoft0: <software crypto=""> on motherboard Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (vesa, 0xffffffff81017160, 0) error 19 Jul 13 14:48:38 kernel kbd0 at kbdmux0 Jul 13 14:48:38 kernel netmap: loaded module Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647e10, 0) error 1 Jul 13 14:48:38 kernel iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647d60, 0) error 1 Jul 13 14:48:38 kernel iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647cb0, 0) error 1 Jul 13 14:48:38 kernel iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620b10, 0) error 1 Jul 13 14:48:38 kernel ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80620a60, 0) error 1 Jul 13 14:48:38 kernel ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. Jul 13 14:48:38 kernel module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806209b0, 0) error 1 Jul 13 14:48:38 kernel ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jul 13 14:48:38 kernel ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. Jul 13 14:48:38 kernel wlan: mac acl policy registered Jul 13 14:48:38 kernel ioapic0 <version 2.0=""> irqs 0-23 on motherboard Jul 13 14:48:38 kernel random: <software, yarrow=""> initialized Jul 13 14:48:38 kernel cpu1 (AP): APIC ID: 2 Jul 13 14:48:38 kernel cpu0 (BSP): APIC ID: 0 Jul 13 14:48:38 kernel FreeBSD/SMP: 1 package(s) x 2 core(s) Jul 13 14:48:38 kernel FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs Jul 13 14:48:38 kernel ACPI APIC Table: <core coreboot=""> Jul 13 14:48:38 kernel Event timer "LAPIC" quality 600 Jul 13 14:48:38 kernel avail memory = 4060024832 (3871 MB) Jul 13 14:48:38 kernel real memory = 6442450944 (6144 MB) Jul 13 14:48:38 kernel TSC: P-state invariant, performance statistics Jul 13 14:48:38 kernel VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID Jul 13 14:48:38 kernel Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jul 13 14:48:38 kernel AMD Features2=0x101 <lahf,prefetch>Jul 13 14:48:38 kernel AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jul 13 14:48:38 kernel Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jul 13 14:48:38 kernel Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jul 13 14:48:38 kernel Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Jul 13 14:48:38 kernel CPU: Intel(R) Atom(TM) CPU C2358 @ 1.74GHz (1750.04-MHz K8-class CPU) Jul 13 14:48:38 kernel FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 Jul 13 14:48:38 kernel root@factory23-amd64-builder:/builder/factory-234/tmp/obj/builder/factory-234/tmp/FreeBSD-src/sys/pfSense amd64 Jul 13 14:48:38 kernel FreeBSD 10.3-RELEASE-p19 #0 bbfdb9a1d(RELENG_2_3_4): Wed May 3 16:12:24 CDT 2017 Jul 13 14:48:38 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Jul 13 14:48:38 kernel The Regents of the University of California. All rights reserved. Jul 13 14:48:38 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jul 13 14:48:38 kernel Copyright (c) 1992-2016 The FreeBSD Project. Jul 13 14:48:38 syslogd kernel boot file is /boot/kernel/kernel Jul 13 14:46:53 syslogd exiting on signal 15 Jul 13 14:46:53 sshlockout 56044 sshlockout/webConfigurator v3.0 starting up Jul 13 14:46:53 shutdown reboot by root:</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></core></software,></version></software></core></acpi></acpi></at></at></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></processor></intel(r)></intel(r)></intel(r)></intel(r)></intel></intel></ahci></ahci></intel></ahci></ahci></pci-isa></isa></isa></enhanced></enhanced></intel></intel></vendor></vendor></generic></generic></generic></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></sendpacket></sendpacket></sendpacket> ```</sendpacket>