NAT not working in Multi-WAN environment

megapearl · Dec 28, 2016, 5:43 PM

Hello,

Upgraded from latest snapshot 2.3.3 to latest snapshot 2.4.

I'm using Multi WAN with 4 ISP's.
NAT in (or out?) doesn't work since 2.4.
When I manually set the default gateway to 1 of the ISP WAN (which are NAT'ed to internal LAN) the services on the LAN are available again (for that interface only).
There are no error messages in the logs.
It seems the packets aren't going out the same interface there are coming in.

2.3.3 doesn't have this problem.

How to troubleshoot?

System LOG:


Dec 28 18:37:34	kernel		cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
Dec 28 18:35:16	kernel		cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
Dec 28 18:35:05	kernel		cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1
Dec 28 18:29:25	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:29:25	xinetd	61893	readjusting service 6969-udp
Dec 28 18:29:25	xinetd	61893	Swapping defaults
Dec 28 18:29:25	xinetd	61893	Starting reconfiguration
Dec 28 18:29:23	check_reload_status		Reloading filter
Dec 28 18:29:21	check_reload_status		Syncing firewall
Dec 28 18:28:59	kernel		cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
Dec 28 18:28:18	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:28:18	xinetd	61893	readjusting service 6969-udp
Dec 28 18:28:18	xinetd	61893	Swapping defaults
Dec 28 18:28:18	xinetd	61893	Starting reconfiguration
Dec 28 18:28:16	check_reload_status		Reloading filter
Dec 28 18:28:14	check_reload_status		Syncing firewall
Dec 28 18:23:47	sshd	61738	Accepted keyboard-interactive/pam for root from 10.0.0.15 port 61108 ssh2
Dec 28 18:22:55	sshlockout	39270	sshlockout/webConfigurator v3.0 starting up
Dec 28 18:22:55	php-fpm	50539	/index.php: Successful login for user 'admin' from: 10.0.0.15
Dec 28 18:17:48	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:17:48	xinetd	61893	readjusting service 6969-udp
Dec 28 18:17:48	xinetd	61893	Swapping defaults
Dec 28 18:17:48	xinetd	61893	Starting reconfiguration
Dec 28 18:17:46	check_reload_status		Reloading filter
Dec 28 18:17:46	php-fpm	23793	/rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
Dec 28 18:17:36	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:17:36	xinetd	61893	readjusting service 6969-udp
Dec 28 18:17:36	xinetd	61893	Swapping defaults
Dec 28 18:17:36	xinetd	61893	Starting reconfiguration
Dec 28 18:17:33	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:17:33	xinetd	61893	readjusting service 6969-udp
Dec 28 18:17:33	xinetd	61893	Swapping defaults
Dec 28 18:17:33	xinetd	61893	Starting reconfiguration
Dec 28 18:17:31	check_reload_status		Reloading filter
Dec 28 18:17:30	php-fpm	94130	/rc.start_packages: [squid] Starting a proxy monitor script
Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: [squid] Starting service...
Dec 28 18:17:29	check_reload_status		Reloading filter
Dec 28 18:17:29	check_reload_status		Restarting OpenVPN tunnels/interfaces
Dec 28 18:17:29	check_reload_status		Restarting ipsec tunnels
Dec 28 18:17:29	check_reload_status		updating dyndns WAN2IPV6_TUNNELV6
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Starting C-ICAP...
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Starting ClamAV...
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Creating 'clamd.sh' rc script.
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Updating ClamAV definitions now... This will take a while. Check freshclam log on the 'Real Time' tab for progress information.
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] ClamAV will be automatically notified about the new definitions when finished. No manual action necessary.
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Google Safe Browsing is enabled but missing safebrowsing.cvd definitions. Running freshclam in background.
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/bin/freshclam --config-file=/usr/local/etc/freshclam.conf, no change needed
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Adding freshclam cronjob.
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/pkg/swapstate_check.php, no change needed
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf, no change needed
Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Adding cronjobs ...
Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: [squid] - squid_resync function call pr: bp: rpc:no
Dec 28 18:17:29	kernel		done.
Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: Restarting/Starting all packages.
Dec 28 18:17:29	syslogd		kernel boot file is /boot/kernel/kernel
Dec 28 18:17:29	syslogd		exiting on signal 15
Dec 28 18:17:28	kernel		done.
Dec 28 18:17:28	kernel		done.
Dec 28 18:17:28	php-cgi		rc.bootup: miniupnpd: Starting service on interface: lan
Dec 28 18:17:28	php-cgi		rc.bootup: Creating rrd update script
Dec 28 18:17:26	kernel		done
Dec 28 18:17:25	kernel		.done.
Dec 28 18:17:24	kernel		..
Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'filter' rules.
Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules.
Dec 28 18:17:24	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:17:24	xinetd	61893	readjusting service 6969-udp
Dec 28 18:17:24	kernel		.
Dec 28 18:17:24	xinetd	61893	Swapping defaults
Dec 28 18:17:24	xinetd	61893	Starting reconfiguration
Dec 28 18:17:24	kernel		.
Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'nat' rules.
Dec 28 18:17:23	check_reload_status		Updating all dyndns
Dec 28 18:17:23	dhcpleases		kqueue error: unkown
Dec 28 18:17:23	kernel		done.
Dec 28 18:17:23	kernel		done.
Dec 28 18:17:22	kernel		done.
Dec 28 18:17:22	php-cgi		rc.bootup: NTPD is starting up.
Dec 28 18:17:22	kernel		done.
Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'filter' rules.
Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'pfearly' rules.
Dec 28 18:17:22	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
Dec 28 18:17:22	xinetd	61893	readjusting service 6969-udp
Dec 28 18:17:22	xinetd	61893	Swapping defaults
Dec 28 18:17:22	xinetd	61893	Starting reconfiguration
Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'nat' rules.
Dec 28 18:17:21	kernel		done.
Dec 28 18:17:21	kernel		done.
Dec 28 18:17:20	php-cgi		rc.bootup: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1
Dec 28 18:17:20	php-cgi		rc.bootup: ROUTING: setting default route to 10.0.5.1
Dec 28 18:17:20	kernel		done.
Dec 28 18:17:20	php-cgi		rc.bootup: Error starting gateway monitor for WAN4IPV6_TUNNELV6
Dec 28 18:17:20	php-cgi		rc.bootup: The command '/usr/local/bin/dpinger -S -r 0 -i WAN4IPV6_TUNNELV6 -B 2001:470:1f12:980::2 -p /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.pid -u /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f12:980::1 >/dev/null' returned exit code '1', the output was ''
Dec 28 18:17:19	check_reload_status		Reloading filter
Dec 28 18:17:19	check_reload_status		Restarting OpenVPN tunnels/interfaces
Dec 28 18:17:19	check_reload_status		Restarting ipsec tunnels
Dec 28 18:17:19	check_reload_status		updating dyndns WAN2IPV6_TUNNELV6
Dec 28 18:17:19	php-fpm	71729	/rc.newwanip: Error starting gateway monitor for WAN3IPV6_TUNNELV6
Dec 28 18:17:19	php-fpm	71729	/rc.newwanip: The command '/usr/local/bin/dpinger -S -r 0 -i WAN3IPV6_TUNNELV6 -B 2001:470:1f08:74::2 -p /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.pid -u /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f08:74::1 >/dev/null' returned exit code '1', the output was ''
Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
Dec 28 18:17:18	kernel		.done.
Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: rc.newwanip: on (IP address: 10.0.6.3) (interface: MOBILE[opt6]) (real interface: vmx3).
Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: rc.newwanip: Info: starting on vmx3.
Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'filter' rules.
Dec 28 18:17:18	php-cgi		rc.bootup: dpinger: No dpinger session running for gateway WAN3_DHCP
Dec 28 18:17:18	php-cgi		rc.bootup: dpinger: No dpinger session running for gateway WAN2_DHCP
Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
Dec 28 18:17:18	kernel		gif3: link state changed to UP
Dec 28 18:17:18	kernel		gif3: link state changed to DOWN
Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: rc.newwanip: on (IP address: 10.0.5.2) (interface: WAN4[opt5]) (real interface: vmx1).
Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: rc.newwanip: Info: starting on vmx1.
Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules.
Dec 28 18:17:18	xinetd	61893	Started working: 1 available service
Dec 28 18:17:18	kernel		.
Dec 28 18:17:18	xinetd	61893	xinetd Version 2.3.15 started with libwrap loadavg options compiled in.
Dec 28 18:17:18	kernel		.
Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'nat' rules.
Dec 28 18:17:17	kernel		pflog0: promiscuous mode enabled
Dec 28 18:17:17	kernel		gif3: link state changed to UP
Dec 28 18:17:17	kernel		gif2: link state changed to UP
Dec 28 18:17:17	kernel		gif2: link state changed to DOWN
Dec 28 18:17:17	kernel		gif1: link state changed to UP
Dec 28 18:17:17	kernel		gif1: link state changed to DOWN
Dec 28 18:17:17	kernel		gif0: link state changed to UP
Dec 28 18:17:17	kernel		gif0: link state changed to DOWN
Dec 28 18:17:17	php-cgi		rc.bootup: Resyncing OpenVPN instances.
Dec 28 18:17:17	check_reload_status		rc.newwanip starting vmx3
Dec 28 18:17:17	kernel		vmx3: link state changed to UP
Dec 28 18:17:17	check_reload_status		Linkup starting vmx3
Dec 28 18:17:17	check_reload_status		rc.newwanip starting vmx1
Dec 28 18:17:15	kernel		gif2: link state changed to UP
Dec 28 18:17:15	php-fpm	290	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
Dec 28 18:17:15	php-fpm	290	/rc.newwanip: rc.newwanip: on (IP address: 10.0.4.2) (interface: WAN3[opt2]) (real interface: vmx5).
Dec 28 18:17:15	php-fpm	290	/rc.newwanip: rc.newwanip: Info: starting on vmx5.
Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
Dec 28 18:17:14	kernel		gif1: link state changed to UP
Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1
Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: rc.newwanip: on (IP address: 94.211.248.93) (interface: WAN2[opt1]) (real interface: vmx4).
Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: rc.newwanip: Info: starting on vmx4.
Dec 28 18:17:14	kernel		vmx1: link state changed to UP
Dec 28 18:17:14	check_reload_status		Linkup starting vmx1
Dec 28 18:17:14	check_reload_status		rc.newwanip starting vmx5
Dec 28 18:17:14	kernel		gif0: link state changed to UP
Dec 28 18:17:14	php-fpm	290	/rc.newwanip: rc.newwanip: on (IP address: 94.211.250.11) (interface: WAN1[wan]) (real interface: vmx2).
Dec 28 18:17:14	php-fpm	290	/rc.newwanip: rc.newwanip: Info: starting on vmx2.
Dec 28 18:17:14	sshlockout	21235	sshlockout/webConfigurator v3.0 starting up
Dec 28 18:17:14	sshd	20906	Server listening on 0.0.0.0 port 22.
Dec 28 18:17:14	sshd	20906	Server listening on :: port 22.
Dec 28 18:17:13	kernel		vmx5: link state changed to UP
Dec 28 18:17:13	check_reload_status		Linkup starting vmx5
Dec 28 18:17:13	check_reload_status		rc.newwanip starting vmx4
Dec 28 18:17:13	kernel		vmx4: link state changed to UP
Dec 28 18:17:13	check_reload_status		Linkup starting vmx4
Dec 28 18:17:13	kernel		vmx0: link state changed to UP
Dec 28 18:17:13	check_reload_status		Linkup starting vmx0
Dec 28 18:17:13	check_reload_status		rc.newwanip starting vmx2
Dec 28 18:17:13	check_reload_status		Linkup starting vmx2
Dec 28 18:17:13	kernel		vmx2: link state changed to UP
Dec 28 18:17:13	kernel		uhub2: 7 ports with 7 removable, self powered
Dec 28 18:17:13	kernel		random: unblocking device.
Dec 28 18:17:13	kernel		uhub1: 6 ports with 6 removable, self powered
Dec 28 18:17:13	kernel		Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]...
Dec 28 18:17:13	kernel		da0: quirks=0x40 <retry_busy>Dec 28 18:17:13	kernel		da0: 8192MB (16777216 512 byte sectors)
Dec 28 18:17:13	kernel		da0: Command Queueing enabled
Dec 28 18:17:13	kernel		da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
Dec 28 18:17:13	kernel		SMP: AP CPU #3 Launched!
Dec 28 18:17:13	kernel		SMP: AP CPU #2 Launched!
Dec 28 18:17:13	kernel		SMP: AP CPU #1 Launched!
Dec 28 18:17:13	kernel		da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device
Dec 28 18:17:13	kernel		da0 at mpt0 bus 0 scbus2 target 0 lun 0
Dec 28 18:17:13	kernel		(da0:mpt0:0:0:0): UNMAPPED
Dec 28 18:17:13	kernel		uhub2: <vmware virtual="" usb="" hub="">on usbus0
Dec 28 18:17:13	kernel		ugen0.3: <vendor 0x0e0f="">at usbus0
Dec 28 18:17:13	kernel		uhid1: <vmware>on usbus0
Dec 28 18:17:13	kernel		uhid0: <vmware>on usbus0
Dec 28 18:17:13	kernel		ugen0.2: <vmware>at usbus0
Dec 28 18:17:13	kernel		uhub0: 2 ports with 2 removable, self powered
Dec 28 18:17:13	kernel		uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Dec 28 18:17:13	kernel		ugen1.1: <0x15ad> at usbus1
Dec 28 18:17:13	kernel		uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Dec 28 18:17:13	kernel		ugen0.1: <0x15ad> at usbus0
Dec 28 18:17:13	kernel		usbus1: 480Mbps High Speed USB v2.0
Dec 28 18:17:13	kernel		usbus0: 12Mbps Full Speed USB v1.0
Dec 28 18:17:13	kernel		nvme cam probe device init
Dec 28 18:17:13	kernel		Timecounters tick every 1.000 msec
Dec 28 18:17:13	kernel		ppc0: cannot reserve I/O port range
Dec 28 18:17:13	kernel		vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Dec 28 18:17:13	kernel		orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
Dec 28 18:17:13	kernel		qpi0: <qpi system="" bus="">on motherboard
Dec 28 18:17:13	kernel		psm0: model IntelliMouse, device ID 3
Dec 28 18:17:13	kernel		psm0: [GIANT-LOCKED]
Dec 28 18:17:13	kernel		psm0: <ps 2="" mouse="">irq 12 on atkbdc0</ps></qpi></isa></generic></vmware></vmware></vmware></vendor></vmware></vmware></retry_busy>

dmesg -a in terminal


Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p5 #19 e26feba(RELENG_2_4): Wed Dec 28 09:46:51 CST 2016
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): text 80x25
CPU: Intel(R) Core(TM) i7 CPU         960  @ 3.20GHz (3197.73-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0x1fa3fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>Features2=0x81b82221 <sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>Structured Extended Features=0x2 <tscadj>VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant
Hypervisor: Origin = "VMwareVMware"
real memory  = 2147483648 (2048 MB)
avail memory = 2023337984 (1929 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <ptltd   ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <version 1.1="">irqs 0-23 on motherboard
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80675440, 0) error 1
random: entropy device external interface
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806754f0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff806755a0, 0) error 1
wlan: mac acl policy registered
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069e9d0, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069ea80, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069eb30, 0) error 1
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff8122a980, 0) error 19
vtvga0: <vt vga="" driver="">on motherboard
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <intel 440bx="">on motherboard
acpi0: Power Button (fixed)
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0: <acpi cpu="">numa-domain 0 on acpi0
cpu1: <acpi cpu="">numa-domain 0 on acpi0
cpu2: <acpi cpu="">numa-domain 0 on acpi0
cpu3: <acpi cpu="">numa-domain 0 on acpi0
attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
isab0: <pci-isa bridge="">at device 7.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
ata0: <ata channel="">at channel 0 on atapci0
ata1: <ata channel="">at channel 1 on atapci0
pci0: <bridge>at device 7.3 (no driver attached)
vgapci0: <vga-compatible display="">port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
vgapci0: Boot video device
mpt0: <lsilogic 1030="" ultra4="" adapter="">port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
mpt0: MPI Version=1.2.0.0
pcib2: <acpi pci-pci="" bridge="">at device 17.0 on pci0
pci2: <acpi pci="" bus="">on pcib2
uhci0: <uhci (generic)="" usb="" controller="">port 0x2000-0x201f irq 19 at device 1.0 on pci2
usbus0 on uhci0
ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfd5ff000-0xfd5fffff irq 16 at device 2.0 on pci2
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib3: <acpi pci-pci="" bridge="">at device 21.0 on pci0
pcib3: [GIANT-LOCKED]
pci3: <acpi pci="" bus="">on pcib3
vmx0: <vmware vmxnet3="" ethernet="" adapter="">port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
vmx0: Ethernet address: 00:50:56:85:19:ce
pcib4: <acpi pci-pci="" bridge="">at device 21.1 on pci0
pcib4: [GIANT-LOCKED]
pci4: <acpi pci="" bus="">on pcib4
vmx1: <vmware vmxnet3="" ethernet="" adapter="">port 0x8000-0x800f mem 0xfd0fc000-0xfd0fcfff,0xfd0fd000-0xfd0fdfff,0xfd0fe000-0xfd0fffff irq 18 at device 0.0 on pci4
vmx1: Ethernet address: fc:d4:f2:df:00:04
pcib5: <acpi pci-pci="" bridge="">at device 21.2 on pci0
pcib5: [GIANT-LOCKED]
pcib6: <acpi pci-pci="" bridge="">at device 21.3 on pci0
pcib6: [GIANT-LOCKED]
pcib7: <acpi pci-pci="" bridge="">at device 21.4 on pci0
pcib7: [GIANT-LOCKED]
pcib8: <acpi pci-pci="" bridge="">at device 21.5 on pci0
pcib8: [GIANT-LOCKED]
pcib9: <acpi pci-pci="" bridge="">at device 21.6 on pci0
pcib9: [GIANT-LOCKED]
pcib10: <acpi pci-pci="" bridge="">at device 21.7 on pci0
pcib10: [GIANT-LOCKED]
pcib11: <acpi pci-pci="" bridge="">at device 22.0 on pci0
pcib11: [GIANT-LOCKED]
pci5: <acpi pci="" bus="">on pcib11
vmx2: <vmware vmxnet3="" ethernet="" adapter="">port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci5
vmx2: Ethernet address: fc:d4:f2:df:00:01
pcib12: <acpi pci-pci="" bridge="">at device 22.1 on pci0
pcib12: [GIANT-LOCKED]
pci6: <acpi pci="" bus="">on pcib12
vmx3: <vmware vmxnet3="" ethernet="" adapter="">port 0x9000-0x900f mem 0xfcffc000-0xfcffcfff,0xfcffd000-0xfcffdfff,0xfcffe000-0xfcffffff irq 19 at device 0.0 on pci6
vmx3: Ethernet address: fc:d4:f2:df:00:05
pcib13: <acpi pci-pci="" bridge="">at device 22.2 on pci0
pcib13: [GIANT-LOCKED]
pcib14: <acpi pci-pci="" bridge="">at device 22.3 on pci0
pcib14: [GIANT-LOCKED]
pcib15: <acpi pci-pci="" bridge="">at device 22.4 on pci0
pcib15: [GIANT-LOCKED]
pcib16: <acpi pci-pci="" bridge="">at device 22.5 on pci0
pcib16: [GIANT-LOCKED]
pcib17: <acpi pci-pci="" bridge="">at device 22.6 on pci0
pcib17: [GIANT-LOCKED]
pcib18: <acpi pci-pci="" bridge="">at device 22.7 on pci0
pcib18: [GIANT-LOCKED]
pcib19: <acpi pci-pci="" bridge="">at device 23.0 on pci0
pcib19: [GIANT-LOCKED]
pci7: <acpi pci="" bus="">on pcib19
vmx4: <vmware vmxnet3="" ethernet="" adapter="">port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci7
vmx4: Ethernet address: fc:d4:f2:df:00:02
pcib20: <acpi pci-pci="" bridge="">at device 23.1 on pci0
pcib20: [GIANT-LOCKED]
pcib21: <acpi pci-pci="" bridge="">at device 23.2 on pci0
pcib21: [GIANT-LOCKED]
pcib22: <acpi pci-pci="" bridge="">at device 23.3 on pci0
pcib22: [GIANT-LOCKED]
pcib23: <acpi pci-pci="" bridge="">at device 23.4 on pci0
pcib23: [GIANT-LOCKED]
pcib24: <acpi pci-pci="" bridge="">at device 23.5 on pci0
pcib24: [GIANT-LOCKED]
pcib25: <acpi pci-pci="" bridge="">at device 23.6 on pci0
pcib25: [GIANT-LOCKED]
pcib26: <acpi pci-pci="" bridge="">at device 23.7 on pci0
pcib26: [GIANT-LOCKED]
pcib27: <acpi pci-pci="" bridge="">at device 24.0 on pci0
pcib27: [GIANT-LOCKED]
pci8: <acpi pci="" bus="">on pcib27
vmx5: <vmware vmxnet3="" ethernet="" adapter="">port 0x7000-0x700f mem 0xfd1fc000-0xfd1fcfff,0xfd1fd000-0xfd1fdfff,0xfd1fe000-0xfd1fffff irq 17 at device 0.0 on pci8
vmx5: Ethernet address: fc:d4:f2:df:00:03
pcib28: <acpi pci-pci="" bridge="">at device 24.1 on pci0
pcib28: [GIANT-LOCKED]
pcib29: <acpi pci-pci="" bridge="">at device 24.2 on pci0
pcib29: [GIANT-LOCKED]
pcib30: <acpi pci-pci="" bridge="">at device 24.3 on pci0
pcib30: [GIANT-LOCKED]
pcib31: <acpi pci-pci="" bridge="">at device 24.4 on pci0
pcib31: [GIANT-LOCKED]
pcib32: <acpi pci-pci="" bridge="">at device 24.5 on pci0
pcib32: [GIANT-LOCKED]
pcib33: <acpi pci-pci="" bridge="">at device 24.6 on pci0
pcib33: [GIANT-LOCKED]
pcib34: <acpi pci-pci="" bridge="">at device 24.7 on pci0
pcib34: [GIANT-LOCKED]
acpi_acad0: <ac adapter="">on acpi0
atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <ps 2="" mouse="">irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
qpi0: <qpi system="" bus="">on motherboard
orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
nvme cam probe device init
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <0x15ad> at usbus0
uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <0x15ad> at usbus1
uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
ugen0.2: <vmware>at usbus0
uhid0: <vmware>on usbus0
uhid1: <vmware>on usbus0
ugen0.3: <vendor 0x0e0f="">at usbus0
uhub2: <vmware virtual="" usb="" hub="">on usbus0
(da0:mpt0:0:0:0): UNMAPPED
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors)
da0: quirks=0x40 <retry_busy>Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]...
uhub1: 6 ports with 6 removable, self powered
Configuring crash dumps...
Using /dev/label/swap0 for dump device.
/dev/ufsid/55e0b2e79f855829: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufsid/55e0b2e79f855829: clean, 298143 free (5879 frags, 36533 blocks, 0.6% fragmentation)
Filesystems are clean, continuing...
Mounting filesystems...
random: unblocking device.

     ___
 ___/ f \134
/ p \134___/ Sense
\134___/   \134
    \134___/

Welcome to pfSense 2.4.0-BETA...

No core dumps found.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.20/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
done.
uhub2: 7 ports with 7 removable, self powered
External config loader 1.0 is now starting... da0s1 da0s1a da0s1b
Launching the init system....... done.
Initializing.................. done.
Starting device manager (devd)...done.
Loading configuration......done.
Updating configuration...done.
Cleaning backup cache.................................done.
Setting up extended sysctls...done.
Setting timezone...done.
Configuring loopback interface...done.
Starting syslog...done.
Starting Secure Shell Services...done.
Setting up interfaces microcode...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring QinQ interfaces...done.
Configuring WAN1 interface...
vmx2: link state changed to UP
done.
Configuring LAN interface...
vmx0: link state changed to UP
done.
Configuring WAN2 interface...
vmx4: link state changed to UP
done.
Configuring WAN3 interface...
vmx5: link state changed to UP
gif0: link state changed to UP
done.
Configuring WAN4 interface...
vmx1: link state changed to UP
gif1: link state changed to UP
gif2: link state changed to UP
done.
Configuring MOBILE interface...
vmx3: link state changed to UP
done.
gif0: link state changed to DOWN
gif0: link state changed to UP
gif1: link state changed to DOWN
gif1: link state changed to UP
gif2: link state changed to DOWN
gif2: link state changed to UP
gif3: link state changed to UP
Configuring WAN1IPV6 interface...done.
Configuring WAN2IPV6 interface...done.
Configuring WAN3IPV6 interface...done.
Configuring WAN4IPV6 interface...done.
Configuring CARP settings...done.
Syncing OpenVPN settings...done.
pflog0: promiscuous mode enabled
Configuring firewall...
gif3: link state changed to DOWN
gif3: link state changed to UP
...done.
Starting PFLOG...done.
Setting up gateway monitors...done.
Starting DNS Resolver...done.
Synchronizing user settings...done.
Starting webConfigurator...done.
Configuring CRON...done.
Starting NTP time client...done.
Starting DHCP service...done.
Starting DHCPv6 service...done.
Configuring firewall......done.
Configuring IPsec VPN... done
Generating RRD graphs...done.
Starting UPnP service... done.
Starting syslog...done.
Starting CRON... done.
 Starting package Open-VM-Tools...done.
 Starting package squid3...done.
 Starting package nmap...done.
 Starting /usr/local/etc/rc.d/c-icap.sh...done.
 Starting /usr/local/etc/rc.d/clamd.sh...done.
 Starting /usr/local/etc/rc.d/sqp_monitor.sh...done.
 Starting /usr/local/etc/rc.d/vmware-guestd.sh...done.
pfSense 2.4.0-BETA amd64 Wed Dec 28 09:41:49 CST 2016
Bootup complete
cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1
cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1</retry_busy></vmware></vmware></vendor></vmware></vmware></vmware></generic></isa></qpi></ps></at></keyboard></ac></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></ehci></uhci></acpi></acpi></lsilogic></vga-compatible></bridge></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></intel></software></vt></version></ptltd ></tscadj></lahf></syscall,nx,rdtscp,lm></sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>

PiBa · Dec 28, 2016, 9:36 PM

Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?

heper · Dec 28, 2016, 11:20 PM

https://redmine.pfsense.org/issues/6986

megapearl · Dec 29, 2016, 12:16 AM

@PiBa:

Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?

All 5 WAN's are DHCP.
Yes, I configured the gateways under gateway groups, as on 2.3.3, on 2.4.0 they are still there.

megapearl · Dec 29, 2016, 12:17 AM

@heper:

https://redmine.pfsense.org/issues/6986

That's exactly the problem I have.

megapearl · Jan 4, 2017, 6:38 PM

Is there are a workaround or can I do something to fix it?
Or do I just need to wait…

jimp · Jan 4, 2017, 8:42 PM

Nothing can be done to workaround the problem. You'll have to wait for a fix from our side.

maverick_slo · Jan 4, 2017, 9:27 PM

Oh uh…
Is this my problem too?
https://forum.pfsense.org/index.php?topic=123370.0

jimp · Jan 4, 2017, 9:33 PM

Most likely.

Raul Ramos · Jan 4, 2017, 10:41 PM

I'm waiting for this fix to…so i will test this problem.

maverick_slo · Jan 5, 2017, 1:19 PM

One more thing…
I really dont know it its connected but still...

See attached image.
I start to download some file hosted behind pfsense with 2 wans (download is happening only on DHCP).
It starts sloppy and interrupting as hell, slow dl speeds lots of retransmissions in packet capture.
But when I hit disconnect on pppoe, it starts to fly as expected...

Can this 2 things somehow be connected?

maverick_slo · Jan 5, 2017, 2:18 PM

Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
Maybe this is client problem and tries to download with ipv4 and ipv6 together?

maverick_slo · Jan 6, 2017, 7:18 AM

@maverick_slo:

Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
Maybe this is client problem and tries to download with ipv4 and ipv6 together?

This is confirmed.
When I disable pppoe ipv4 gateway transfers are OK.
This also happens on 2.3.3

So 2 IPv4 gateways are causing major problems, at least for me.

athurdent · Jan 11, 2017, 7:18 AM

Seems to be resolved now: https://redmine.pfsense.org/issues/6986
Don't know if the latest snapshots already have the fix, though.

jimp · Jan 11, 2017, 12:16 PM

The latest snapshots have the fix (for almost a day now)

techy82 · Jan 11, 2017, 7:12 PM

works great now :-)

megapearl · Jan 11, 2017, 11:03 PM

Just upgraded from 2.3.3 to 2.4.0, bug is fixed, Problem solved! :D