NAT not working in Multi-WAN environment
-
Hello,
Upgraded from latest snapshot 2.3.3 to latest snapshot 2.4.
I'm using Multi WAN with 4 ISP's.
NAT in (or out?) doesn't work since 2.4.
When I manually set the default gateway to 1 of the ISP WAN (which are NAT'ed to internal LAN) the services on the LAN are available again (for that interface only).
There are no error messages in the logs.
It seems the packets aren't going out the same interface there are coming in.2.3.3 doesn't have this problem.
How to troubleshoot?
System LOG:
Dec 28 18:37:34 kernel cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1 Dec 28 18:35:16 kernel cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1 Dec 28 18:35:05 kernel cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1 Dec 28 18:29:25 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:29:25 xinetd 61893 readjusting service 6969-udp Dec 28 18:29:25 xinetd 61893 Swapping defaults Dec 28 18:29:25 xinetd 61893 Starting reconfiguration Dec 28 18:29:23 check_reload_status Reloading filter Dec 28 18:29:21 check_reload_status Syncing firewall Dec 28 18:28:59 kernel cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1 Dec 28 18:28:18 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:28:18 xinetd 61893 readjusting service 6969-udp Dec 28 18:28:18 xinetd 61893 Swapping defaults Dec 28 18:28:18 xinetd 61893 Starting reconfiguration Dec 28 18:28:16 check_reload_status Reloading filter Dec 28 18:28:14 check_reload_status Syncing firewall Dec 28 18:23:47 sshd 61738 Accepted keyboard-interactive/pam for root from 10.0.0.15 port 61108 ssh2 Dec 28 18:22:55 sshlockout 39270 sshlockout/webConfigurator v3.0 starting up Dec 28 18:22:55 php-fpm 50539 /index.php: Successful login for user 'admin' from: 10.0.0.15 Dec 28 18:17:48 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:17:48 xinetd 61893 readjusting service 6969-udp Dec 28 18:17:48 xinetd 61893 Swapping defaults Dec 28 18:17:48 xinetd 61893 Starting reconfiguration Dec 28 18:17:46 check_reload_status Reloading filter Dec 28 18:17:46 php-fpm 23793 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing. Dec 28 18:17:36 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:17:36 xinetd 61893 readjusting service 6969-udp Dec 28 18:17:36 xinetd 61893 Swapping defaults Dec 28 18:17:36 xinetd 61893 Starting reconfiguration Dec 28 18:17:33 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:17:33 xinetd 61893 readjusting service 6969-udp Dec 28 18:17:33 xinetd 61893 Swapping defaults Dec 28 18:17:33 xinetd 61893 Starting reconfiguration Dec 28 18:17:31 check_reload_status Reloading filter Dec 28 18:17:30 php-fpm 94130 /rc.start_packages: [squid] Starting a proxy monitor script Dec 28 18:17:29 php-fpm 94130 /rc.start_packages: [squid] Starting service... Dec 28 18:17:29 check_reload_status Reloading filter Dec 28 18:17:29 check_reload_status Restarting OpenVPN tunnels/interfaces Dec 28 18:17:29 check_reload_status Restarting ipsec tunnels Dec 28 18:17:29 check_reload_status updating dyndns WAN2IPV6_TUNNELV6 Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Starting C-ICAP... Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Starting ClamAV... Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Creating 'clamd.sh' rc script. Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Updating ClamAV definitions now... This will take a while. Check freshclam log on the 'Real Time' tab for progress information. Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] ClamAV will be automatically notified about the new definitions when finished. No manual action necessary. Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Google Safe Browsing is enabled but missing safebrowsing.cvd definitions. Running freshclam in background. Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: Checked cron job for /usr/local/bin/freshclam --config-file=/usr/local/etc/freshclam.conf, no change needed Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Adding freshclam cronjob. Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: Checked cron job for /usr/local/pkg/swapstate_check.php, no change needed Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: Checked cron job for /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf, no change needed Dec 28 18:17:28 php-fpm 94130 /rc.start_packages: [squid] Adding cronjobs ... Dec 28 18:17:29 php-fpm 94130 /rc.start_packages: [squid] - squid_resync function call pr: bp: rpc:no Dec 28 18:17:29 kernel done. Dec 28 18:17:29 php-fpm 94130 /rc.start_packages: Restarting/Starting all packages. Dec 28 18:17:29 syslogd kernel boot file is /boot/kernel/kernel Dec 28 18:17:29 syslogd exiting on signal 15 Dec 28 18:17:28 kernel done. Dec 28 18:17:28 kernel done. Dec 28 18:17:28 php-cgi rc.bootup: miniupnpd: Starting service on interface: lan Dec 28 18:17:28 php-cgi rc.bootup: Creating rrd update script Dec 28 18:17:26 kernel done Dec 28 18:17:25 kernel .done. Dec 28 18:17:24 kernel .. Dec 28 18:17:24 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'filter' rules. Dec 28 18:17:24 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules. Dec 28 18:17:24 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:17:24 xinetd 61893 readjusting service 6969-udp Dec 28 18:17:24 kernel . Dec 28 18:17:24 xinetd 61893 Swapping defaults Dec 28 18:17:24 xinetd 61893 Starting reconfiguration Dec 28 18:17:24 kernel . Dec 28 18:17:24 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'nat' rules. Dec 28 18:17:23 check_reload_status Updating all dyndns Dec 28 18:17:23 dhcpleases kqueue error: unkown Dec 28 18:17:23 kernel done. Dec 28 18:17:23 kernel done. Dec 28 18:17:22 kernel done. Dec 28 18:17:22 php-cgi rc.bootup: NTPD is starting up. Dec 28 18:17:22 kernel done. Dec 28 18:17:22 php-fpm 71729 /rc.filter_configure_sync: [squid] Installed but not started. Not installing 'filter' rules. Dec 28 18:17:22 php-fpm 71729 /rc.filter_configure_sync: [squid] Installed but not started. Not installing 'pfearly' rules. Dec 28 18:17:22 xinetd 61893 Reconfigured: new=0 old=1 dropped=0 (services) Dec 28 18:17:22 xinetd 61893 readjusting service 6969-udp Dec 28 18:17:22 xinetd 61893 Swapping defaults Dec 28 18:17:22 xinetd 61893 Starting reconfiguration Dec 28 18:17:22 php-fpm 71729 /rc.filter_configure_sync: [squid] Installed but not started. Not installing 'nat' rules. Dec 28 18:17:21 kernel done. Dec 28 18:17:21 kernel done. Dec 28 18:17:20 php-cgi rc.bootup: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1 Dec 28 18:17:20 php-cgi rc.bootup: ROUTING: setting default route to 10.0.5.1 Dec 28 18:17:20 kernel done. Dec 28 18:17:20 php-cgi rc.bootup: Error starting gateway monitor for WAN4IPV6_TUNNELV6 Dec 28 18:17:20 php-cgi rc.bootup: The command '/usr/local/bin/dpinger -S -r 0 -i WAN4IPV6_TUNNELV6 -B 2001:470:1f12:980::2 -p /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.pid -u /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f12:980::1 >/dev/null' returned exit code '1', the output was '' Dec 28 18:17:19 check_reload_status Reloading filter Dec 28 18:17:19 check_reload_status Restarting OpenVPN tunnels/interfaces Dec 28 18:17:19 check_reload_status Restarting ipsec tunnels Dec 28 18:17:19 check_reload_status updating dyndns WAN2IPV6_TUNNELV6 Dec 28 18:17:19 php-fpm 71729 /rc.newwanip: Error starting gateway monitor for WAN3IPV6_TUNNELV6 Dec 28 18:17:19 php-fpm 71729 /rc.newwanip: The command '/usr/local/bin/dpinger -S -r 0 -i WAN3IPV6_TUNNELV6 -B 2001:470:1f08:74::2 -p /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.pid -u /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f08:74::1 >/dev/null' returned exit code '1', the output was '' Dec 28 18:17:18 php-cgi rc.bootup: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1 Dec 28 18:17:18 php-cgi rc.bootup: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1 Dec 28 18:17:18 php-cgi rc.bootup: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1 Dec 28 18:17:18 kernel .done. Dec 28 18:17:18 php-fpm 71729 /rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1 Dec 28 18:17:18 php-fpm 71729 /rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1 Dec 28 18:17:18 php-fpm 71729 /rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1 Dec 28 18:17:18 php-fpm 71729 /rc.newwanip: rc.newwanip: on (IP address: 10.0.6.3) (interface: MOBILE[opt6]) (real interface: vmx3). Dec 28 18:17:18 php-fpm 71729 /rc.newwanip: rc.newwanip: Info: starting on vmx3. Dec 28 18:17:18 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'filter' rules. Dec 28 18:17:18 php-cgi rc.bootup: dpinger: No dpinger session running for gateway WAN3_DHCP Dec 28 18:17:18 php-cgi rc.bootup: dpinger: No dpinger session running for gateway WAN2_DHCP Dec 28 18:17:18 php-fpm 40048 /rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1 Dec 28 18:17:18 php-fpm 40048 /rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1 Dec 28 18:17:18 kernel gif3: link state changed to UP Dec 28 18:17:18 kernel gif3: link state changed to DOWN Dec 28 18:17:18 php-fpm 40048 /rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1 Dec 28 18:17:18 php-fpm 40048 /rc.newwanip: rc.newwanip: on (IP address: 10.0.5.2) (interface: WAN4[opt5]) (real interface: vmx1). Dec 28 18:17:18 php-fpm 40048 /rc.newwanip: rc.newwanip: Info: starting on vmx1. Dec 28 18:17:18 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules. Dec 28 18:17:18 xinetd 61893 Started working: 1 available service Dec 28 18:17:18 kernel . Dec 28 18:17:18 xinetd 61893 xinetd Version 2.3.15 started with libwrap loadavg options compiled in. Dec 28 18:17:18 kernel . Dec 28 18:17:18 php-cgi rc.bootup: [squid] Installed but not started. Not installing 'nat' rules. Dec 28 18:17:17 kernel pflog0: promiscuous mode enabled Dec 28 18:17:17 kernel gif3: link state changed to UP Dec 28 18:17:17 kernel gif2: link state changed to UP Dec 28 18:17:17 kernel gif2: link state changed to DOWN Dec 28 18:17:17 kernel gif1: link state changed to UP Dec 28 18:17:17 kernel gif1: link state changed to DOWN Dec 28 18:17:17 kernel gif0: link state changed to UP Dec 28 18:17:17 kernel gif0: link state changed to DOWN Dec 28 18:17:17 php-cgi rc.bootup: Resyncing OpenVPN instances. Dec 28 18:17:17 check_reload_status rc.newwanip starting vmx3 Dec 28 18:17:17 kernel vmx3: link state changed to UP Dec 28 18:17:17 check_reload_status Linkup starting vmx3 Dec 28 18:17:17 check_reload_status rc.newwanip starting vmx1 Dec 28 18:17:15 kernel gif2: link state changed to UP Dec 28 18:17:15 php-fpm 290 /rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1 Dec 28 18:17:15 php-fpm 290 /rc.newwanip: rc.newwanip: on (IP address: 10.0.4.2) (interface: WAN3[opt2]) (real interface: vmx5). Dec 28 18:17:15 php-fpm 290 /rc.newwanip: rc.newwanip: Info: starting on vmx5. Dec 28 18:17:14 php-fpm 30103 /rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1 Dec 28 18:17:14 kernel gif1: link state changed to UP Dec 28 18:17:14 php-fpm 30103 /rc.newwanip: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1 Dec 28 18:17:14 php-fpm 30103 /rc.newwanip: rc.newwanip: on (IP address: 94.211.248.93) (interface: WAN2[opt1]) (real interface: vmx4). Dec 28 18:17:14 php-fpm 30103 /rc.newwanip: rc.newwanip: Info: starting on vmx4. Dec 28 18:17:14 kernel vmx1: link state changed to UP Dec 28 18:17:14 check_reload_status Linkup starting vmx1 Dec 28 18:17:14 check_reload_status rc.newwanip starting vmx5 Dec 28 18:17:14 kernel gif0: link state changed to UP Dec 28 18:17:14 php-fpm 290 /rc.newwanip: rc.newwanip: on (IP address: 94.211.250.11) (interface: WAN1[wan]) (real interface: vmx2). Dec 28 18:17:14 php-fpm 290 /rc.newwanip: rc.newwanip: Info: starting on vmx2. Dec 28 18:17:14 sshlockout 21235 sshlockout/webConfigurator v3.0 starting up Dec 28 18:17:14 sshd 20906 Server listening on 0.0.0.0 port 22. Dec 28 18:17:14 sshd 20906 Server listening on :: port 22. Dec 28 18:17:13 kernel vmx5: link state changed to UP Dec 28 18:17:13 check_reload_status Linkup starting vmx5 Dec 28 18:17:13 check_reload_status rc.newwanip starting vmx4 Dec 28 18:17:13 kernel vmx4: link state changed to UP Dec 28 18:17:13 check_reload_status Linkup starting vmx4 Dec 28 18:17:13 kernel vmx0: link state changed to UP Dec 28 18:17:13 check_reload_status Linkup starting vmx0 Dec 28 18:17:13 check_reload_status rc.newwanip starting vmx2 Dec 28 18:17:13 check_reload_status Linkup starting vmx2 Dec 28 18:17:13 kernel vmx2: link state changed to UP Dec 28 18:17:13 kernel uhub2: 7 ports with 7 removable, self powered Dec 28 18:17:13 kernel random: unblocking device. Dec 28 18:17:13 kernel uhub1: 6 ports with 6 removable, self powered Dec 28 18:17:13 kernel Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]... Dec 28 18:17:13 kernel da0: quirks=0x40 <retry_busy>Dec 28 18:17:13 kernel da0: 8192MB (16777216 512 byte sectors) Dec 28 18:17:13 kernel da0: Command Queueing enabled Dec 28 18:17:13 kernel da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) Dec 28 18:17:13 kernel SMP: AP CPU #3 Launched! Dec 28 18:17:13 kernel SMP: AP CPU #2 Launched! Dec 28 18:17:13 kernel SMP: AP CPU #1 Launched! Dec 28 18:17:13 kernel da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device Dec 28 18:17:13 kernel da0 at mpt0 bus 0 scbus2 target 0 lun 0 Dec 28 18:17:13 kernel (da0:mpt0:0:0:0): UNMAPPED Dec 28 18:17:13 kernel uhub2: <vmware virtual="" usb="" hub="">on usbus0 Dec 28 18:17:13 kernel ugen0.3: <vendor 0x0e0f="">at usbus0 Dec 28 18:17:13 kernel uhid1: <vmware>on usbus0 Dec 28 18:17:13 kernel uhid0: <vmware>on usbus0 Dec 28 18:17:13 kernel ugen0.2: <vmware>at usbus0 Dec 28 18:17:13 kernel uhub0: 2 ports with 2 removable, self powered Dec 28 18:17:13 kernel uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 Dec 28 18:17:13 kernel ugen1.1: <0x15ad> at usbus1 Dec 28 18:17:13 kernel uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 Dec 28 18:17:13 kernel ugen0.1: <0x15ad> at usbus0 Dec 28 18:17:13 kernel usbus1: 480Mbps High Speed USB v2.0 Dec 28 18:17:13 kernel usbus0: 12Mbps Full Speed USB v1.0 Dec 28 18:17:13 kernel nvme cam probe device init Dec 28 18:17:13 kernel Timecounters tick every 1.000 msec Dec 28 18:17:13 kernel ppc0: cannot reserve I/O port range Dec 28 18:17:13 kernel vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Dec 28 18:17:13 kernel orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0 Dec 28 18:17:13 kernel qpi0: <qpi system="" bus="">on motherboard Dec 28 18:17:13 kernel psm0: model IntelliMouse, device ID 3 Dec 28 18:17:13 kernel psm0: [GIANT-LOCKED] Dec 28 18:17:13 kernel psm0: <ps 2="" mouse="">irq 12 on atkbdc0</ps></qpi></isa></generic></vmware></vmware></vmware></vendor></vmware></vmware></retry_busy>dmesg -a in terminal
Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-RELEASE-p5 #19 e26feba(RELENG_2_4): Wed Dec 28 09:46:51 CST 2016 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) VT(vga): text 80x25 CPU: Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz (3197.73-MHz K8-class CPU) Origin="GenuineIntel" Id=0x106a5 Family=0x6 Model=0x1a Stepping=5 Features=0x1fa3fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>Features2=0x81b82221 <sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>Structured Extended Features=0x2 <tscadj>VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID TSC: P-state invariant Hypervisor: Origin = "VMwareVMware" real memory = 2147483648 (2048 MB) avail memory = 2023337984 (1929 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <ptltd ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 2 package(s) x 2 core(s) MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <version 1.1="">irqs 0-23 on motherboard ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80675440, 0) error 1 random: entropy device external interface ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806754f0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff806755a0, 0) error 1 wlan: mac acl policy registered iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069e9d0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069ea80, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069eb30, 0) error 1 kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff8122a980, 0) error 19 vtvga0: <vt vga="" driver="">on motherboard cryptosoft0: <software crypto="">on motherboard padlock0: No ACE support. acpi0: <intel 440bx="">on motherboard acpi0: Power Button (fixed) Timecounter "HPET" frequency 14318180 Hz quality 950 cpu0: <acpi cpu="">numa-domain 0 on acpi0 cpu1: <acpi cpu="">numa-domain 0 on acpi0 cpu2: <acpi cpu="">numa-domain 0 on acpi0 cpu3: <acpi cpu="">numa-domain 0 on acpi0 attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0 pci0: <acpi pci="" bus="">on pcib0 pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0 pci1: <acpi pci="" bus="">on pcib1 isab0: <pci-isa bridge="">at device 7.0 on pci0 isa0: <isa bus="">on isab0 atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0 ata0: <ata channel="">at channel 0 on atapci0 ata1: <ata channel="">at channel 1 on atapci0 pci0: <bridge>at device 7.3 (no driver attached) vgapci0: <vga-compatible display="">port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0 vgapci0: Boot video device mpt0: <lsilogic 1030="" ultra4="" adapter="">port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0 mpt0: MPI Version=1.2.0.0 pcib2: <acpi pci-pci="" bridge="">at device 17.0 on pci0 pci2: <acpi pci="" bus="">on pcib2 uhci0: <uhci (generic)="" usb="" controller="">port 0x2000-0x201f irq 19 at device 1.0 on pci2 usbus0 on uhci0 ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfd5ff000-0xfd5fffff irq 16 at device 2.0 on pci2 usbus1: EHCI version 1.0 usbus1 on ehci0 pcib3: <acpi pci-pci="" bridge="">at device 21.0 on pci0 pcib3: [GIANT-LOCKED] pci3: <acpi pci="" bus="">on pcib3 vmx0: <vmware vmxnet3="" ethernet="" adapter="">port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3 vmx0: Ethernet address: 00:50:56:85:19:ce pcib4: <acpi pci-pci="" bridge="">at device 21.1 on pci0 pcib4: [GIANT-LOCKED] pci4: <acpi pci="" bus="">on pcib4 vmx1: <vmware vmxnet3="" ethernet="" adapter="">port 0x8000-0x800f mem 0xfd0fc000-0xfd0fcfff,0xfd0fd000-0xfd0fdfff,0xfd0fe000-0xfd0fffff irq 18 at device 0.0 on pci4 vmx1: Ethernet address: fc:d4:f2:df:00:04 pcib5: <acpi pci-pci="" bridge="">at device 21.2 on pci0 pcib5: [GIANT-LOCKED] pcib6: <acpi pci-pci="" bridge="">at device 21.3 on pci0 pcib6: [GIANT-LOCKED] pcib7: <acpi pci-pci="" bridge="">at device 21.4 on pci0 pcib7: [GIANT-LOCKED] pcib8: <acpi pci-pci="" bridge="">at device 21.5 on pci0 pcib8: [GIANT-LOCKED] pcib9: <acpi pci-pci="" bridge="">at device 21.6 on pci0 pcib9: [GIANT-LOCKED] pcib10: <acpi pci-pci="" bridge="">at device 21.7 on pci0 pcib10: [GIANT-LOCKED] pcib11: <acpi pci-pci="" bridge="">at device 22.0 on pci0 pcib11: [GIANT-LOCKED] pci5: <acpi pci="" bus="">on pcib11 vmx2: <vmware vmxnet3="" ethernet="" adapter="">port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci5 vmx2: Ethernet address: fc:d4:f2:df:00:01 pcib12: <acpi pci-pci="" bridge="">at device 22.1 on pci0 pcib12: [GIANT-LOCKED] pci6: <acpi pci="" bus="">on pcib12 vmx3: <vmware vmxnet3="" ethernet="" adapter="">port 0x9000-0x900f mem 0xfcffc000-0xfcffcfff,0xfcffd000-0xfcffdfff,0xfcffe000-0xfcffffff irq 19 at device 0.0 on pci6 vmx3: Ethernet address: fc:d4:f2:df:00:05 pcib13: <acpi pci-pci="" bridge="">at device 22.2 on pci0 pcib13: [GIANT-LOCKED] pcib14: <acpi pci-pci="" bridge="">at device 22.3 on pci0 pcib14: [GIANT-LOCKED] pcib15: <acpi pci-pci="" bridge="">at device 22.4 on pci0 pcib15: [GIANT-LOCKED] pcib16: <acpi pci-pci="" bridge="">at device 22.5 on pci0 pcib16: [GIANT-LOCKED] pcib17: <acpi pci-pci="" bridge="">at device 22.6 on pci0 pcib17: [GIANT-LOCKED] pcib18: <acpi pci-pci="" bridge="">at device 22.7 on pci0 pcib18: [GIANT-LOCKED] pcib19: <acpi pci-pci="" bridge="">at device 23.0 on pci0 pcib19: [GIANT-LOCKED] pci7: <acpi pci="" bus="">on pcib19 vmx4: <vmware vmxnet3="" ethernet="" adapter="">port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci7 vmx4: Ethernet address: fc:d4:f2:df:00:02 pcib20: <acpi pci-pci="" bridge="">at device 23.1 on pci0 pcib20: [GIANT-LOCKED] pcib21: <acpi pci-pci="" bridge="">at device 23.2 on pci0 pcib21: [GIANT-LOCKED] pcib22: <acpi pci-pci="" bridge="">at device 23.3 on pci0 pcib22: [GIANT-LOCKED] pcib23: <acpi pci-pci="" bridge="">at device 23.4 on pci0 pcib23: [GIANT-LOCKED] pcib24: <acpi pci-pci="" bridge="">at device 23.5 on pci0 pcib24: [GIANT-LOCKED] pcib25: <acpi pci-pci="" bridge="">at device 23.6 on pci0 pcib25: [GIANT-LOCKED] pcib26: <acpi pci-pci="" bridge="">at device 23.7 on pci0 pcib26: [GIANT-LOCKED] pcib27: <acpi pci-pci="" bridge="">at device 24.0 on pci0 pcib27: [GIANT-LOCKED] pci8: <acpi pci="" bus="">on pcib27 vmx5: <vmware vmxnet3="" ethernet="" adapter="">port 0x7000-0x700f mem 0xfd1fc000-0xfd1fcfff,0xfd1fd000-0xfd1fdfff,0xfd1fe000-0xfd1fffff irq 17 at device 0.0 on pci8 vmx5: Ethernet address: fc:d4:f2:df:00:03 pcib28: <acpi pci-pci="" bridge="">at device 24.1 on pci0 pcib28: [GIANT-LOCKED] pcib29: <acpi pci-pci="" bridge="">at device 24.2 on pci0 pcib29: [GIANT-LOCKED] pcib30: <acpi pci-pci="" bridge="">at device 24.3 on pci0 pcib30: [GIANT-LOCKED] pcib31: <acpi pci-pci="" bridge="">at device 24.4 on pci0 pcib31: [GIANT-LOCKED] pcib32: <acpi pci-pci="" bridge="">at device 24.5 on pci0 pcib32: [GIANT-LOCKED] pcib33: <acpi pci-pci="" bridge="">at device 24.6 on pci0 pcib33: [GIANT-LOCKED] pcib34: <acpi pci-pci="" bridge="">at device 24.7 on pci0 pcib34: [GIANT-LOCKED] acpi_acad0: <ac adapter="">on acpi0 atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0 atkbd0: <at keyboard="">irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <ps 2="" mouse="">irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse, device ID 3 qpi0: <qpi system="" bus="">on motherboard orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0 vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range Timecounters tick every 1.000 msec nvme cam probe device init usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: <0x15ad> at usbus0 uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <0x15ad> at usbus1 uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub0: 2 ports with 2 removable, self powered ugen0.2: <vmware>at usbus0 uhid0: <vmware>on usbus0 uhid1: <vmware>on usbus0 ugen0.3: <vendor 0x0e0f="">at usbus0 uhub2: <vmware virtual="" usb="" hub="">on usbus0 (da0:mpt0:0:0:0): UNMAPPED da0 at mpt0 bus 0 scbus2 target 0 lun 0 da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors) da0: quirks=0x40 <retry_busy>Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]... uhub1: 6 ports with 6 removable, self powered Configuring crash dumps... Using /dev/label/swap0 for dump device. /dev/ufsid/55e0b2e79f855829: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufsid/55e0b2e79f855829: clean, 298143 free (5879 frags, 36533 blocks, 0.6% fragmentation) Filesystems are clean, continuing... Mounting filesystems... random: unblocking device. ___ ___/ f \134 / p \134___/ Sense \134___/ \134 \134___/ Welcome to pfSense 2.4.0-BETA... No core dumps found. ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.20/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 done. uhub2: 7 ports with 7 removable, self powered External config loader 1.0 is now starting... da0s1 da0s1a da0s1b Launching the init system....... done. Initializing.................. done. Starting device manager (devd)...done. Loading configuration......done. Updating configuration...done. Cleaning backup cache.................................done. Setting up extended sysctls...done. Setting timezone...done. Configuring loopback interface...done. Starting syslog...done. Starting Secure Shell Services...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Creating wireless clone interfaces...done. Configuring LAGG interfaces...done. Configuring VLAN interfaces...done. Configuring QinQ interfaces...done. Configuring WAN1 interface... vmx2: link state changed to UP done. Configuring LAN interface... vmx0: link state changed to UP done. Configuring WAN2 interface... vmx4: link state changed to UP done. Configuring WAN3 interface... vmx5: link state changed to UP gif0: link state changed to UP done. Configuring WAN4 interface... vmx1: link state changed to UP gif1: link state changed to UP gif2: link state changed to UP done. Configuring MOBILE interface... vmx3: link state changed to UP done. gif0: link state changed to DOWN gif0: link state changed to UP gif1: link state changed to DOWN gif1: link state changed to UP gif2: link state changed to DOWN gif2: link state changed to UP gif3: link state changed to UP Configuring WAN1IPV6 interface...done. Configuring WAN2IPV6 interface...done. Configuring WAN3IPV6 interface...done. Configuring WAN4IPV6 interface...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. pflog0: promiscuous mode enabled Configuring firewall... gif3: link state changed to DOWN gif3: link state changed to UP ...done. Starting PFLOG...done. Setting up gateway monitors...done. Starting DNS Resolver...done. Synchronizing user settings...done. Starting webConfigurator...done. Configuring CRON...done. Starting NTP time client...done. Starting DHCP service...done. Starting DHCPv6 service...done. Configuring firewall......done. Configuring IPsec VPN... done Generating RRD graphs...done. Starting UPnP service... done. Starting syslog...done. Starting CRON... done. Starting package Open-VM-Tools...done. Starting package squid3...done. Starting package nmap...done. Starting /usr/local/etc/rc.d/c-icap.sh...done. Starting /usr/local/etc/rc.d/clamd.sh...done. Starting /usr/local/etc/rc.d/sqp_monitor.sh...done. Starting /usr/local/etc/rc.d/vmware-guestd.sh...done. pfSense 2.4.0-BETA amd64 Wed Dec 28 09:41:49 CST 2016 Bootup complete cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1 cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1 cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1 cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1</retry_busy></vmware></vmware></vendor></vmware></vmware></vmware></generic></isa></qpi></ps></at></keyboard></ac></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></ehci></uhci></acpi></acpi></lsilogic></vga-compatible></bridge></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></intel></software></vt></version></ptltd ></tscadj></lahf></syscall,nx,rdtscp,lm></sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt> -
Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?
-
https://redmine.pfsense.org/issues/6986
-
Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?
All 5 WAN's are DHCP.
Yes, I configured the gateways under gateway groups, as on 2.3.3, on 2.4.0 they are still there. -
-
Is there are a workaround or can I do something to fix it?
Or do I just need to wait… -
Nothing can be done to workaround the problem. You'll have to wait for a fix from our side.
-
Oh uh…
Is this my problem too?
https://forum.pfsense.org/index.php?topic=123370.0 -
Most likely.
-
I'm waiting for this fix to…so i will test this problem.
-
One more thing…
I really dont know it its connected but still...See attached image.
I start to download some file hosted behind pfsense with 2 wans (download is happening only on DHCP).
It starts sloppy and interrupting as hell, slow dl speeds lots of retransmissions in packet capture.
But when I hit disconnect on pppoe, it starts to fly as expected...Can this 2 things somehow be connected?
-
Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
Maybe this is client problem and tries to download with ipv4 and ipv6 together? -
Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
Maybe this is client problem and tries to download with ipv4 and ipv6 together?This is confirmed.
When I disable pppoe ipv4 gateway transfers are OK.
This also happens on 2.3.3So 2 IPv4 gateways are causing major problems, at least for me.
-
Seems to be resolved now: https://redmine.pfsense.org/issues/6986
Don't know if the latest snapshots already have the fix, though. -
The latest snapshots have the fix (for almost a day now)
-
works great now :-)
-
Just upgraded from 2.3.3 to 2.4.0, bug is fixed, Problem solved! :D
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.