NAT not working in Multi-WAN environment



  • Hello,

    Upgraded from latest snapshot 2.3.3 to latest snapshot 2.4.

    I'm using Multi WAN with 4 ISP's.
    NAT in (or out?) doesn't work since 2.4.
    When I manually set the default gateway to 1 of the ISP WAN (which are NAT'ed to internal LAN) the services on the LAN are available again (for that interface only).
    There are no error messages in the logs.
    It seems the packets aren't going out the same interface there are coming in.

    2.3.3 doesn't have this problem.

    How to troubleshoot?

    System LOG:

    
    Dec 28 18:37:34	kernel		cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    Dec 28 18:35:16	kernel		cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    Dec 28 18:35:05	kernel		cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1
    Dec 28 18:29:25	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:29:25	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:29:25	xinetd	61893	Swapping defaults
    Dec 28 18:29:25	xinetd	61893	Starting reconfiguration
    Dec 28 18:29:23	check_reload_status		Reloading filter
    Dec 28 18:29:21	check_reload_status		Syncing firewall
    Dec 28 18:28:59	kernel		cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    Dec 28 18:28:18	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:28:18	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:28:18	xinetd	61893	Swapping defaults
    Dec 28 18:28:18	xinetd	61893	Starting reconfiguration
    Dec 28 18:28:16	check_reload_status		Reloading filter
    Dec 28 18:28:14	check_reload_status		Syncing firewall
    Dec 28 18:23:47	sshd	61738	Accepted keyboard-interactive/pam for root from 10.0.0.15 port 61108 ssh2
    Dec 28 18:22:55	sshlockout	39270	sshlockout/webConfigurator v3.0 starting up
    Dec 28 18:22:55	php-fpm	50539	/index.php: Successful login for user 'admin' from: 10.0.0.15
    Dec 28 18:17:48	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:17:48	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:17:48	xinetd	61893	Swapping defaults
    Dec 28 18:17:48	xinetd	61893	Starting reconfiguration
    Dec 28 18:17:46	check_reload_status		Reloading filter
    Dec 28 18:17:46	php-fpm	23793	/rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Dec 28 18:17:36	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:17:36	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:17:36	xinetd	61893	Swapping defaults
    Dec 28 18:17:36	xinetd	61893	Starting reconfiguration
    Dec 28 18:17:33	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:17:33	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:17:33	xinetd	61893	Swapping defaults
    Dec 28 18:17:33	xinetd	61893	Starting reconfiguration
    Dec 28 18:17:31	check_reload_status		Reloading filter
    Dec 28 18:17:30	php-fpm	94130	/rc.start_packages: [squid] Starting a proxy monitor script
    Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: [squid] Starting service...
    Dec 28 18:17:29	check_reload_status		Reloading filter
    Dec 28 18:17:29	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Dec 28 18:17:29	check_reload_status		Restarting ipsec tunnels
    Dec 28 18:17:29	check_reload_status		updating dyndns WAN2IPV6_TUNNELV6
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Starting C-ICAP...
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Starting ClamAV...
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Creating 'clamd.sh' rc script.
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Updating ClamAV definitions now... This will take a while. Check freshclam log on the 'Real Time' tab for progress information.
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] ClamAV will be automatically notified about the new definitions when finished. No manual action necessary.
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Google Safe Browsing is enabled but missing safebrowsing.cvd definitions. Running freshclam in background.
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/bin/freshclam --config-file=/usr/local/etc/freshclam.conf, no change needed
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Adding freshclam cronjob.
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/pkg/swapstate_check.php, no change needed
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: Checked cron job for /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf, no change needed
    Dec 28 18:17:28	php-fpm	94130	/rc.start_packages: [squid] Adding cronjobs ...
    Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: [squid] - squid_resync function call pr: bp: rpc:no
    Dec 28 18:17:29	kernel		done.
    Dec 28 18:17:29	php-fpm	94130	/rc.start_packages: Restarting/Starting all packages.
    Dec 28 18:17:29	syslogd		kernel boot file is /boot/kernel/kernel
    Dec 28 18:17:29	syslogd		exiting on signal 15
    Dec 28 18:17:28	kernel		done.
    Dec 28 18:17:28	kernel		done.
    Dec 28 18:17:28	php-cgi		rc.bootup: miniupnpd: Starting service on interface: lan
    Dec 28 18:17:28	php-cgi		rc.bootup: Creating rrd update script
    Dec 28 18:17:26	kernel		done
    Dec 28 18:17:25	kernel		.done.
    Dec 28 18:17:24	kernel		..
    Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'filter' rules.
    Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules.
    Dec 28 18:17:24	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:17:24	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:17:24	kernel		.
    Dec 28 18:17:24	xinetd	61893	Swapping defaults
    Dec 28 18:17:24	xinetd	61893	Starting reconfiguration
    Dec 28 18:17:24	kernel		.
    Dec 28 18:17:24	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'nat' rules.
    Dec 28 18:17:23	check_reload_status		Updating all dyndns
    Dec 28 18:17:23	dhcpleases		kqueue error: unkown
    Dec 28 18:17:23	kernel		done.
    Dec 28 18:17:23	kernel		done.
    Dec 28 18:17:22	kernel		done.
    Dec 28 18:17:22	php-cgi		rc.bootup: NTPD is starting up.
    Dec 28 18:17:22	kernel		done.
    Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'filter' rules.
    Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'pfearly' rules.
    Dec 28 18:17:22	xinetd	61893	Reconfigured: new=0 old=1 dropped=0 (services)
    Dec 28 18:17:22	xinetd	61893	readjusting service 6969-udp
    Dec 28 18:17:22	xinetd	61893	Swapping defaults
    Dec 28 18:17:22	xinetd	61893	Starting reconfiguration
    Dec 28 18:17:22	php-fpm	71729	/rc.filter_configure_sync: [squid] Installed but not started. Not installing 'nat' rules.
    Dec 28 18:17:21	kernel		done.
    Dec 28 18:17:21	kernel		done.
    Dec 28 18:17:20	php-cgi		rc.bootup: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1
    Dec 28 18:17:20	php-cgi		rc.bootup: ROUTING: setting default route to 10.0.5.1
    Dec 28 18:17:20	kernel		done.
    Dec 28 18:17:20	php-cgi		rc.bootup: Error starting gateway monitor for WAN4IPV6_TUNNELV6
    Dec 28 18:17:20	php-cgi		rc.bootup: The command '/usr/local/bin/dpinger -S -r 0 -i WAN4IPV6_TUNNELV6 -B 2001:470:1f12:980::2 -p /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.pid -u /var/run/dpinger_WAN4IPV6_TUNNELV6~2001:470:1f12:980::2~2001:470:1f12:980::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f12:980::1 >/dev/null' returned exit code '1', the output was ''
    Dec 28 18:17:19	check_reload_status		Reloading filter
    Dec 28 18:17:19	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Dec 28 18:17:19	check_reload_status		Restarting ipsec tunnels
    Dec 28 18:17:19	check_reload_status		updating dyndns WAN2IPV6_TUNNELV6
    Dec 28 18:17:19	php-fpm	71729	/rc.newwanip: Error starting gateway monitor for WAN3IPV6_TUNNELV6
    Dec 28 18:17:19	php-fpm	71729	/rc.newwanip: The command '/usr/local/bin/dpinger -S -r 0 -i WAN3IPV6_TUNNELV6 -B 2001:470:1f08:74::2 -p /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.pid -u /var/run/dpinger_WAN3IPV6_TUNNELV6~2001:470:1f08:74::2~2001:470:1f08:74::1.sock -C "/etc/rc.gateway_alarm" -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2001:470:1f08:74::1 >/dev/null' returned exit code '1', the output was ''
    Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
    Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
    Dec 28 18:17:18	php-cgi		rc.bootup: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
    Dec 28 18:17:18	kernel		.done.
    Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
    Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
    Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
    Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: rc.newwanip: on (IP address: 10.0.6.3) (interface: MOBILE[opt6]) (real interface: vmx3).
    Dec 28 18:17:18	php-fpm	71729	/rc.newwanip: rc.newwanip: Info: starting on vmx3.
    Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'filter' rules.
    Dec 28 18:17:18	php-cgi		rc.bootup: dpinger: No dpinger session running for gateway WAN3_DHCP
    Dec 28 18:17:18	php-cgi		rc.bootup: dpinger: No dpinger session running for gateway WAN2_DHCP
    Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 62.140.145.58 and adding a new route through 10.0.6.1
    Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 94.211.234.1 and adding a new route through 10.0.5.1
    Dec 28 18:17:18	kernel		gif3: link state changed to UP
    Dec 28 18:17:18	kernel		gif3: link state changed to DOWN
    Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
    Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: rc.newwanip: on (IP address: 10.0.5.2) (interface: WAN4[opt5]) (real interface: vmx1).
    Dec 28 18:17:18	php-fpm	40048	/rc.newwanip: rc.newwanip: Info: starting on vmx1.
    Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'pfearly' rules.
    Dec 28 18:17:18	xinetd	61893	Started working: 1 available service
    Dec 28 18:17:18	kernel		.
    Dec 28 18:17:18	xinetd	61893	xinetd Version 2.3.15 started with libwrap loadavg options compiled in.
    Dec 28 18:17:18	kernel		.
    Dec 28 18:17:18	php-cgi		rc.bootup: [squid] Installed but not started. Not installing 'nat' rules.
    Dec 28 18:17:17	kernel		pflog0: promiscuous mode enabled
    Dec 28 18:17:17	kernel		gif3: link state changed to UP
    Dec 28 18:17:17	kernel		gif2: link state changed to UP
    Dec 28 18:17:17	kernel		gif2: link state changed to DOWN
    Dec 28 18:17:17	kernel		gif1: link state changed to UP
    Dec 28 18:17:17	kernel		gif1: link state changed to DOWN
    Dec 28 18:17:17	kernel		gif0: link state changed to UP
    Dec 28 18:17:17	kernel		gif0: link state changed to DOWN
    Dec 28 18:17:17	php-cgi		rc.bootup: Resyncing OpenVPN instances.
    Dec 28 18:17:17	check_reload_status		rc.newwanip starting vmx3
    Dec 28 18:17:17	kernel		vmx3: link state changed to UP
    Dec 28 18:17:17	check_reload_status		Linkup starting vmx3
    Dec 28 18:17:17	check_reload_status		rc.newwanip starting vmx1
    Dec 28 18:17:15	kernel		gif2: link state changed to UP
    Dec 28 18:17:15	php-fpm	290	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
    Dec 28 18:17:15	php-fpm	290	/rc.newwanip: rc.newwanip: on (IP address: 10.0.4.2) (interface: WAN3[opt2]) (real interface: vmx5).
    Dec 28 18:17:15	php-fpm	290	/rc.newwanip: rc.newwanip: Info: starting on vmx5.
    Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: Removing static route for monitor 212.127.170.1 and adding a new route through 10.0.4.1
    Dec 28 18:17:14	kernel		gif1: link state changed to UP
    Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: ROUTING: setting IPv6 default route to 2001:470:1f14:1225::1
    Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: rc.newwanip: on (IP address: 94.211.248.93) (interface: WAN2[opt1]) (real interface: vmx4).
    Dec 28 18:17:14	php-fpm	30103	/rc.newwanip: rc.newwanip: Info: starting on vmx4.
    Dec 28 18:17:14	kernel		vmx1: link state changed to UP
    Dec 28 18:17:14	check_reload_status		Linkup starting vmx1
    Dec 28 18:17:14	check_reload_status		rc.newwanip starting vmx5
    Dec 28 18:17:14	kernel		gif0: link state changed to UP
    Dec 28 18:17:14	php-fpm	290	/rc.newwanip: rc.newwanip: on (IP address: 94.211.250.11) (interface: WAN1[wan]) (real interface: vmx2).
    Dec 28 18:17:14	php-fpm	290	/rc.newwanip: rc.newwanip: Info: starting on vmx2.
    Dec 28 18:17:14	sshlockout	21235	sshlockout/webConfigurator v3.0 starting up
    Dec 28 18:17:14	sshd	20906	Server listening on 0.0.0.0 port 22.
    Dec 28 18:17:14	sshd	20906	Server listening on :: port 22.
    Dec 28 18:17:13	kernel		vmx5: link state changed to UP
    Dec 28 18:17:13	check_reload_status		Linkup starting vmx5
    Dec 28 18:17:13	check_reload_status		rc.newwanip starting vmx4
    Dec 28 18:17:13	kernel		vmx4: link state changed to UP
    Dec 28 18:17:13	check_reload_status		Linkup starting vmx4
    Dec 28 18:17:13	kernel		vmx0: link state changed to UP
    Dec 28 18:17:13	check_reload_status		Linkup starting vmx0
    Dec 28 18:17:13	check_reload_status		rc.newwanip starting vmx2
    Dec 28 18:17:13	check_reload_status		Linkup starting vmx2
    Dec 28 18:17:13	kernel		vmx2: link state changed to UP
    Dec 28 18:17:13	kernel		uhub2: 7 ports with 7 removable, self powered
    Dec 28 18:17:13	kernel		random: unblocking device.
    Dec 28 18:17:13	kernel		uhub1: 6 ports with 6 removable, self powered
    Dec 28 18:17:13	kernel		Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]...
    Dec 28 18:17:13	kernel		da0: quirks=0x40 <retry_busy>Dec 28 18:17:13	kernel		da0: 8192MB (16777216 512 byte sectors)
    Dec 28 18:17:13	kernel		da0: Command Queueing enabled
    Dec 28 18:17:13	kernel		da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
    Dec 28 18:17:13	kernel		SMP: AP CPU #3 Launched!
    Dec 28 18:17:13	kernel		SMP: AP CPU #2 Launched!
    Dec 28 18:17:13	kernel		SMP: AP CPU #1 Launched!
    Dec 28 18:17:13	kernel		da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device
    Dec 28 18:17:13	kernel		da0 at mpt0 bus 0 scbus2 target 0 lun 0
    Dec 28 18:17:13	kernel		(da0:mpt0:0:0:0): UNMAPPED
    Dec 28 18:17:13	kernel		uhub2: <vmware virtual="" usb="" hub="">on usbus0
    Dec 28 18:17:13	kernel		ugen0.3: <vendor 0x0e0f="">at usbus0
    Dec 28 18:17:13	kernel		uhid1: <vmware>on usbus0
    Dec 28 18:17:13	kernel		uhid0: <vmware>on usbus0
    Dec 28 18:17:13	kernel		ugen0.2: <vmware>at usbus0
    Dec 28 18:17:13	kernel		uhub0: 2 ports with 2 removable, self powered
    Dec 28 18:17:13	kernel		uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
    Dec 28 18:17:13	kernel		ugen1.1: <0x15ad> at usbus1
    Dec 28 18:17:13	kernel		uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
    Dec 28 18:17:13	kernel		ugen0.1: <0x15ad> at usbus0
    Dec 28 18:17:13	kernel		usbus1: 480Mbps High Speed USB v2.0
    Dec 28 18:17:13	kernel		usbus0: 12Mbps Full Speed USB v1.0
    Dec 28 18:17:13	kernel		nvme cam probe device init
    Dec 28 18:17:13	kernel		Timecounters tick every 1.000 msec
    Dec 28 18:17:13	kernel		ppc0: cannot reserve I/O port range
    Dec 28 18:17:13	kernel		vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    Dec 28 18:17:13	kernel		orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
    Dec 28 18:17:13	kernel		qpi0: <qpi system="" bus="">on motherboard
    Dec 28 18:17:13	kernel		psm0: model IntelliMouse, device ID 3
    Dec 28 18:17:13	kernel		psm0: [GIANT-LOCKED]
    Dec 28 18:17:13	kernel		psm0: <ps 2="" mouse="">irq 12 on atkbdc0</ps></qpi></isa></generic></vmware></vmware></vmware></vendor></vmware></vmware></retry_busy> 
    

    dmesg -a in terminal

    
    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.0-RELEASE-p5 #19 e26feba(RELENG_2_4): Wed Dec 28 09:46:51 CST 2016
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    VT(vga): text 80x25
    CPU: Intel(R) Core(TM) i7 CPU         960  @ 3.20GHz (3197.73-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
      Features=0x1fa3fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>Features2=0x81b82221 <sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>Structured Extended Features=0x2 <tscadj>VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
      TSC: P-state invariant
    Hypervisor: Origin = "VMwareVMware"
    real memory  = 2147483648 (2048 MB)
    avail memory = 2023337984 (1929 MB)
    Event timer "LAPIC" quality 400
    ACPI APIC Table: <ptltd   ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 2 package(s) x 2 core(s)
    MADT: Forcing active-low polarity and level trigger for SCI
    ioapic0 <version 1.1="">irqs 0-23 on motherboard
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80675440, 0) error 1
    random: entropy device external interface
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806754f0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff806755a0, 0) error 1
    wlan: mac acl policy registered
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069e9d0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069ea80, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069eb30, 0) error 1
    kbd1 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff8122a980, 0) error 19
    vtvga0: <vt vga="" driver="">on motherboard
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <intel 440bx="">on motherboard
    acpi0: Power Button (fixed)
    Timecounter "HPET" frequency 14318180 Hz quality 950
    cpu0: <acpi cpu="">numa-domain 0 on acpi0
    cpu1: <acpi cpu="">numa-domain 0 on acpi0
    cpu2: <acpi cpu="">numa-domain 0 on acpi0
    cpu3: <acpi cpu="">numa-domain 0 on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    isab0: <pci-isa bridge="">at device 7.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
    ata0: <ata channel="">at channel 0 on atapci0
    ata1: <ata channel="">at channel 1 on atapci0
    pci0: <bridge>at device 7.3 (no driver attached)
    vgapci0: <vga-compatible display="">port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
    vgapci0: Boot video device
    mpt0: <lsilogic 1030="" ultra4="" adapter="">port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
    mpt0: MPI Version=1.2.0.0
    pcib2: <acpi pci-pci="" bridge="">at device 17.0 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    uhci0: <uhci (generic)="" usb="" controller="">port 0x2000-0x201f irq 19 at device 1.0 on pci2
    usbus0 on uhci0
    ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfd5ff000-0xfd5fffff irq 16 at device 2.0 on pci2
    usbus1: EHCI version 1.0
    usbus1 on ehci0
    pcib3: <acpi pci-pci="" bridge="">at device 21.0 on pci0
    pcib3: [GIANT-LOCKED]
    pci3: <acpi pci="" bus="">on pcib3
    vmx0: <vmware vmxnet3="" ethernet="" adapter="">port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
    vmx0: Ethernet address: 00:50:56:85:19:ce
    pcib4: <acpi pci-pci="" bridge="">at device 21.1 on pci0
    pcib4: [GIANT-LOCKED]
    pci4: <acpi pci="" bus="">on pcib4
    vmx1: <vmware vmxnet3="" ethernet="" adapter="">port 0x8000-0x800f mem 0xfd0fc000-0xfd0fcfff,0xfd0fd000-0xfd0fdfff,0xfd0fe000-0xfd0fffff irq 18 at device 0.0 on pci4
    vmx1: Ethernet address: fc:d4:f2:df:00:04
    pcib5: <acpi pci-pci="" bridge="">at device 21.2 on pci0
    pcib5: [GIANT-LOCKED]
    pcib6: <acpi pci-pci="" bridge="">at device 21.3 on pci0
    pcib6: [GIANT-LOCKED]
    pcib7: <acpi pci-pci="" bridge="">at device 21.4 on pci0
    pcib7: [GIANT-LOCKED]
    pcib8: <acpi pci-pci="" bridge="">at device 21.5 on pci0
    pcib8: [GIANT-LOCKED]
    pcib9: <acpi pci-pci="" bridge="">at device 21.6 on pci0
    pcib9: [GIANT-LOCKED]
    pcib10: <acpi pci-pci="" bridge="">at device 21.7 on pci0
    pcib10: [GIANT-LOCKED]
    pcib11: <acpi pci-pci="" bridge="">at device 22.0 on pci0
    pcib11: [GIANT-LOCKED]
    pci5: <acpi pci="" bus="">on pcib11
    vmx2: <vmware vmxnet3="" ethernet="" adapter="">port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci5
    vmx2: Ethernet address: fc:d4:f2:df:00:01
    pcib12: <acpi pci-pci="" bridge="">at device 22.1 on pci0
    pcib12: [GIANT-LOCKED]
    pci6: <acpi pci="" bus="">on pcib12
    vmx3: <vmware vmxnet3="" ethernet="" adapter="">port 0x9000-0x900f mem 0xfcffc000-0xfcffcfff,0xfcffd000-0xfcffdfff,0xfcffe000-0xfcffffff irq 19 at device 0.0 on pci6
    vmx3: Ethernet address: fc:d4:f2:df:00:05
    pcib13: <acpi pci-pci="" bridge="">at device 22.2 on pci0
    pcib13: [GIANT-LOCKED]
    pcib14: <acpi pci-pci="" bridge="">at device 22.3 on pci0
    pcib14: [GIANT-LOCKED]
    pcib15: <acpi pci-pci="" bridge="">at device 22.4 on pci0
    pcib15: [GIANT-LOCKED]
    pcib16: <acpi pci-pci="" bridge="">at device 22.5 on pci0
    pcib16: [GIANT-LOCKED]
    pcib17: <acpi pci-pci="" bridge="">at device 22.6 on pci0
    pcib17: [GIANT-LOCKED]
    pcib18: <acpi pci-pci="" bridge="">at device 22.7 on pci0
    pcib18: [GIANT-LOCKED]
    pcib19: <acpi pci-pci="" bridge="">at device 23.0 on pci0
    pcib19: [GIANT-LOCKED]
    pci7: <acpi pci="" bus="">on pcib19
    vmx4: <vmware vmxnet3="" ethernet="" adapter="">port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci7
    vmx4: Ethernet address: fc:d4:f2:df:00:02
    pcib20: <acpi pci-pci="" bridge="">at device 23.1 on pci0
    pcib20: [GIANT-LOCKED]
    pcib21: <acpi pci-pci="" bridge="">at device 23.2 on pci0
    pcib21: [GIANT-LOCKED]
    pcib22: <acpi pci-pci="" bridge="">at device 23.3 on pci0
    pcib22: [GIANT-LOCKED]
    pcib23: <acpi pci-pci="" bridge="">at device 23.4 on pci0
    pcib23: [GIANT-LOCKED]
    pcib24: <acpi pci-pci="" bridge="">at device 23.5 on pci0
    pcib24: [GIANT-LOCKED]
    pcib25: <acpi pci-pci="" bridge="">at device 23.6 on pci0
    pcib25: [GIANT-LOCKED]
    pcib26: <acpi pci-pci="" bridge="">at device 23.7 on pci0
    pcib26: [GIANT-LOCKED]
    pcib27: <acpi pci-pci="" bridge="">at device 24.0 on pci0
    pcib27: [GIANT-LOCKED]
    pci8: <acpi pci="" bus="">on pcib27
    vmx5: <vmware vmxnet3="" ethernet="" adapter="">port 0x7000-0x700f mem 0xfd1fc000-0xfd1fcfff,0xfd1fd000-0xfd1fdfff,0xfd1fe000-0xfd1fffff irq 17 at device 0.0 on pci8
    vmx5: Ethernet address: fc:d4:f2:df:00:03
    pcib28: <acpi pci-pci="" bridge="">at device 24.1 on pci0
    pcib28: [GIANT-LOCKED]
    pcib29: <acpi pci-pci="" bridge="">at device 24.2 on pci0
    pcib29: [GIANT-LOCKED]
    pcib30: <acpi pci-pci="" bridge="">at device 24.3 on pci0
    pcib30: [GIANT-LOCKED]
    pcib31: <acpi pci-pci="" bridge="">at device 24.4 on pci0
    pcib31: [GIANT-LOCKED]
    pcib32: <acpi pci-pci="" bridge="">at device 24.5 on pci0
    pcib32: [GIANT-LOCKED]
    pcib33: <acpi pci-pci="" bridge="">at device 24.6 on pci0
    pcib33: [GIANT-LOCKED]
    pcib34: <acpi pci-pci="" bridge="">at device 24.7 on pci0
    pcib34: [GIANT-LOCKED]
    acpi_acad0: <ac adapter="">on acpi0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    psm0: <ps 2="" mouse="">irq 12 on atkbdc0
    psm0: [GIANT-LOCKED]
    psm0: model IntelliMouse, device ID 3
    qpi0: <qpi system="" bus="">on motherboard
    orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    Timecounters tick every 1.000 msec
    nvme cam probe device init
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: <0x15ad> at usbus0
    uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
    ugen1.1: <0x15ad> at usbus1
    uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
    uhub0: 2 ports with 2 removable, self powered
    ugen0.2: <vmware>at usbus0
    uhid0: <vmware>on usbus0
    uhid1: <vmware>on usbus0
    ugen0.3: <vendor 0x0e0f="">at usbus0
    uhub2: <vmware virtual="" usb="" hub="">on usbus0
    (da0:mpt0:0:0:0): UNMAPPED
    da0 at mpt0 bus 0 scbus2 target 0 lun 0
    da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #3 Launched!
    da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
    da0: Command Queueing enabled
    da0: 8192MB (16777216 512 byte sectors)
    da0: quirks=0x40 <retry_busy>Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]...
    uhub1: 6 ports with 6 removable, self powered
    Configuring crash dumps...
    Using /dev/label/swap0 for dump device.
    /dev/ufsid/55e0b2e79f855829: FILE SYSTEM CLEAN; SKIPPING CHECKS
    /dev/ufsid/55e0b2e79f855829: clean, 298143 free (5879 frags, 36533 blocks, 0.6% fragmentation)
    Filesystems are clean, continuing...
    Mounting filesystems...
    random: unblocking device.
    
         ___
     ___/ f \134
    / p \134___/ Sense
    \134___/   \134
        \134___/
    
    Welcome to pfSense 2.4.0-BETA...
    
    No core dumps found.
    ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.20/mach/CORE
    32-bit compatibility ldconfig path: /usr/lib32
    done.
    uhub2: 7 ports with 7 removable, self powered
    External config loader 1.0 is now starting... da0s1 da0s1a da0s1b
    Launching the init system....... done.
    Initializing.................. done.
    Starting device manager (devd)...done.
    Loading configuration......done.
    Updating configuration...done.
    Cleaning backup cache.................................done.
    Setting up extended sysctls...done.
    Setting timezone...done.
    Configuring loopback interface...done.
    Starting syslog...done.
    Starting Secure Shell Services...done.
    Setting up interfaces microcode...done.
    Configuring loopback interface...done.
    Creating wireless clone interfaces...done.
    Configuring LAGG interfaces...done.
    Configuring VLAN interfaces...done.
    Configuring QinQ interfaces...done.
    Configuring WAN1 interface...
    vmx2: link state changed to UP
    done.
    Configuring LAN interface...
    vmx0: link state changed to UP
    done.
    Configuring WAN2 interface...
    vmx4: link state changed to UP
    done.
    Configuring WAN3 interface...
    vmx5: link state changed to UP
    gif0: link state changed to UP
    done.
    Configuring WAN4 interface...
    vmx1: link state changed to UP
    gif1: link state changed to UP
    gif2: link state changed to UP
    done.
    Configuring MOBILE interface...
    vmx3: link state changed to UP
    done.
    gif0: link state changed to DOWN
    gif0: link state changed to UP
    gif1: link state changed to DOWN
    gif1: link state changed to UP
    gif2: link state changed to DOWN
    gif2: link state changed to UP
    gif3: link state changed to UP
    Configuring WAN1IPV6 interface...done.
    Configuring WAN2IPV6 interface...done.
    Configuring WAN3IPV6 interface...done.
    Configuring WAN4IPV6 interface...done.
    Configuring CARP settings...done.
    Syncing OpenVPN settings...done.
    pflog0: promiscuous mode enabled
    Configuring firewall...
    gif3: link state changed to DOWN
    gif3: link state changed to UP
    ...done.
    Starting PFLOG...done.
    Setting up gateway monitors...done.
    Starting DNS Resolver...done.
    Synchronizing user settings...done.
    Starting webConfigurator...done.
    Configuring CRON...done.
    Starting NTP time client...done.
    Starting DHCP service...done.
    Starting DHCPv6 service...done.
    Configuring firewall......done.
    Configuring IPsec VPN... done
    Generating RRD graphs...done.
    Starting UPnP service... done.
    Starting syslog...done.
    Starting CRON... done.
     Starting package Open-VM-Tools...done.
     Starting package squid3...done.
     Starting package nmap...done.
     Starting /usr/local/etc/rc.d/c-icap.sh...done.
     Starting /usr/local/etc/rc.d/clamd.sh...done.
     Starting /usr/local/etc/rc.d/sqp_monitor.sh...done.
     Starting /usr/local/etc/rc.d/vmware-guestd.sh...done.
    pfSense 2.4.0-BETA amd64 Wed Dec 28 09:41:49 CST 2016
    Bootup complete
    cannot forward src fe80:1::20b:82ff:fe63:de14, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe7c:879a, dst 2a00:1288:12c:2::100c, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1</retry_busy></vmware></vmware></vendor></vmware></vmware></vmware></generic></isa></qpi></ps></at></keyboard></ac></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></ehci></uhci></acpi></acpi></lsilogic></vga-compatible></bridge></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></intel></software></vt></version></ptltd ></tscadj></lahf></syscall,nx,rdtscp,lm></sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt> 
    


  • Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?





  • @PiBa:

    Have you configured a gateway on the interfaces/wan edit page? Are you using static or dhcp on wan? If you check /tmp/rules.debug the pass rule should contain 'reply-to' i suppose that is not the case?

    All 5 WAN's are DHCP.
    Yes, I configured the gateways under gateway groups, as on 2.3.3, on 2.4.0 they are still there.



  • @heper:

    https://redmine.pfsense.org/issues/6986

    That's exactly the problem I have.



  • Is there are a workaround or can I do something to fix it?
    Or do I just need to wait…


  • Rebel Alliance Developer Netgate

    Nothing can be done to workaround the problem. You'll have to wait for a fix from our side.



  • Oh uh…
    Is this my problem too?
    https://forum.pfsense.org/index.php?topic=123370.0


  • Rebel Alliance Developer Netgate

    Most likely.



  • I'm waiting for this fix to…so i will test this problem.



  • One more thing…
    I really dont know it its connected but still...

    See attached image.
    I start to download some file hosted behind pfsense with 2 wans (download is happening only on DHCP).
    It starts sloppy and interrupting as hell, slow dl speeds lots of retransmissions in packet capture.
    But when I hit disconnect on pppoe, it starts to fly as expected...

    Can this 2 things somehow be connected?




  • Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
    Maybe this is client problem and tries to download with ipv4 and ipv6 together?



  • @maverick_slo:

    Forgot to mention that pppoe has ipv6 and client that downloads has ipv6 too. Webserver has also ipv6.
    Maybe this is client problem and tries to download with ipv4 and ipv6 together?

    This is confirmed.
    When I disable pppoe ipv4 gateway transfers are OK.
    This also happens on 2.3.3

    So 2 IPv4 gateways are causing major problems, at least for me.



  • Seems to be resolved now: https://redmine.pfsense.org/issues/6986
    Don't know if the latest snapshots already have the fix, though.


  • Rebel Alliance Developer Netgate

    The latest snapshots have the fix (for almost a day now)



  • works great now :-)



  • Just upgraded from 2.3.3 to 2.4.0, bug is fixed, Problem solved! :D


Log in to reply