Upgraded from 2.3.3 to 2.4.0 Ipsec routing error

  • Hi,

    Updated yesterday from 2.3.3 to 2.4.0 (I was waiting for the Multi Wan Reply-To bug fix).
    But got another problem.

    Services on this side of IPSec are not reachable from the other side.

    Can't find any errors except in dmesg -a

    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.0-RELEASE-p6 #55 2ede8a24166(RELENG_2_4): Thu Jan 12 07:49:59 CST 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    VT(vga): text 80x25
    CPU: Intel(R) Core(TM) i7 CPU         960  @ 3.20GHz (3197.73-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
      Features=0x1fa3fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>Features2=0x81b82221 <sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>Structured Extended Features=0x2 <tscadj>VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
      TSC: P-state invariant
    Hypervisor: Origin = "VMwareVMware"
    real memory  = 2147483648 (2048 MB)
    avail memory = 2023337984 (1929 MB)
    Event timer "LAPIC" quality 400
    ACPI APIC Table: <ptltd   ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 2 package(s) x 2 core(s)
    MADT: Forcing active-low polarity and level trigger for SCI
    ioapic0 <version 1.1="">irqs 0-23 on motherboard
    random: entropy device external interface
    wlan: mac acl policy registered
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069e9d0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069ea80, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069eb30, 0) error 1
    kbd1 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff8122a980, 0) error 19
    vtvga0: <vt vga="" driver="">on motherboard
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <intel 440bx="">on motherboard
    acpi0: Power Button (fixed)
    Timecounter "HPET" frequency 14318180 Hz quality 950
    cpu0: <acpi cpu="">numa-domain 0 on acpi0
    cpu1: <acpi cpu="">numa-domain 0 on acpi0
    cpu2: <acpi cpu="">numa-domain 0 on acpi0
    cpu3: <acpi cpu="">numa-domain 0 on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    isab0: <pci-isa bridge="">at device 7.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
    ata0: <ata channel="">at channel 0 on atapci0
    ata1: <ata channel="">at channel 1 on atapci0
    pci0: <bridge>at device 7.3 (no driver attached)
    vgapci0: <vga-compatible display="">port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
    vgapci0: Boot video device
    mpt0: <lsilogic 1030="" ultra4="" adapter="">port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
    mpt0: MPI Version=
    pcib2: <acpi pci-pci="" bridge="">at device 17.0 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    uhci0: <uhci (generic)="" usb="" controller="">port 0x2000-0x201f irq 19 at device 1.0 on pci2
    usbus0 on uhci0
    ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfd5ff000-0xfd5fffff irq 16 at device 2.0 on pci2
    usbus1: EHCI version 1.0
    usbus1 on ehci0
    pcib3: <acpi pci-pci="" bridge="">at device 21.0 on pci0
    pcib3: [GIANT-LOCKED]
    pci3: <acpi pci="" bus="">on pcib3
    vmx0: <vmware vmxnet3="" ethernet="" adapter="">port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
    vmx0: Ethernet address: 00:50:56:85:19:ce
    pcib4: <acpi pci-pci="" bridge="">at device 21.1 on pci0
    pcib4: [GIANT-LOCKED]
    pci4: <acpi pci="" bus="">on pcib4
    vmx1: <vmware vmxnet3="" ethernet="" adapter="">port 0x8000-0x800f mem 0xfd0fc000-0xfd0fcfff,0xfd0fd000-0xfd0fdfff,0xfd0fe000-0xfd0fffff irq 18 at device 0.0 on pci4
    vmx1: Ethernet address: fc:d4:f2:df:00:04
    pcib5: <acpi pci-pci="" bridge="">at device 21.2 on pci0
    pcib5: [GIANT-LOCKED]
    pcib6: <acpi pci-pci="" bridge="">at device 21.3 on pci0
    pcib6: [GIANT-LOCKED]
    pcib7: <acpi pci-pci="" bridge="">at device 21.4 on pci0
    pcib7: [GIANT-LOCKED]
    pcib8: <acpi pci-pci="" bridge="">at device 21.5 on pci0
    pcib8: [GIANT-LOCKED]
    pcib9: <acpi pci-pci="" bridge="">at device 21.6 on pci0
    pcib9: [GIANT-LOCKED]
    pcib10: <acpi pci-pci="" bridge="">at device 21.7 on pci0
    pcib10: [GIANT-LOCKED]
    pcib11: <acpi pci-pci="" bridge="">at device 22.0 on pci0
    pcib11: [GIANT-LOCKED]
    pci5: <acpi pci="" bus="">on pcib11
    vmx2: <vmware vmxnet3="" ethernet="" adapter="">port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci5
    vmx2: Ethernet address: fc:d4:f2:df:00:01
    pcib12: <acpi pci-pci="" bridge="">at device 22.1 on pci0
    pcib12: [GIANT-LOCKED]
    pci6: <acpi pci="" bus="">on pcib12
    vmx3: <vmware vmxnet3="" ethernet="" adapter="">port 0x9000-0x900f mem 0xfcffc000-0xfcffcfff,0xfcffd000-0xfcffdfff,0xfcffe000-0xfcffffff irq 19 at device 0.0 on pci6
    vmx3: Ethernet address: fc:d4:f2:df:00:05
    pcib13: <acpi pci-pci="" bridge="">at device 22.2 on pci0
    pcib13: [GIANT-LOCKED]
    pcib14: <acpi pci-pci="" bridge="">at device 22.3 on pci0
    pcib14: [GIANT-LOCKED]
    pcib15: <acpi pci-pci="" bridge="">at device 22.4 on pci0
    pcib15: [GIANT-LOCKED]
    pcib16: <acpi pci-pci="" bridge="">at device 22.5 on pci0
    pcib16: [GIANT-LOCKED]
    pcib17: <acpi pci-pci="" bridge="">at device 22.6 on pci0
    pcib17: [GIANT-LOCKED]
    pcib18: <acpi pci-pci="" bridge="">at device 22.7 on pci0
    pcib18: [GIANT-LOCKED]
    pcib19: <acpi pci-pci="" bridge="">at device 23.0 on pci0
    pcib19: [GIANT-LOCKED]
    pci7: <acpi pci="" bus="">on pcib19
    vmx4: <vmware vmxnet3="" ethernet="" adapter="">port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci7
    vmx4: Ethernet address: fc:d4:f2:df:00:02
    pcib20: <acpi pci-pci="" bridge="">at device 23.1 on pci0
    pcib20: [GIANT-LOCKED]
    pcib21: <acpi pci-pci="" bridge="">at device 23.2 on pci0
    pcib21: [GIANT-LOCKED]
    pcib22: <acpi pci-pci="" bridge="">at device 23.3 on pci0
    pcib22: [GIANT-LOCKED]
    pcib23: <acpi pci-pci="" bridge="">at device 23.4 on pci0
    pcib23: [GIANT-LOCKED]
    pcib24: <acpi pci-pci="" bridge="">at device 23.5 on pci0
    pcib24: [GIANT-LOCKED]
    pcib25: <acpi pci-pci="" bridge="">at device 23.6 on pci0
    pcib25: [GIANT-LOCKED]
    pcib26: <acpi pci-pci="" bridge="">at device 23.7 on pci0
    pcib26: [GIANT-LOCKED]
    pcib27: <acpi pci-pci="" bridge="">at device 24.0 on pci0
    pcib27: [GIANT-LOCKED]
    pci8: <acpi pci="" bus="">on pcib27
    vmx5: <vmware vmxnet3="" ethernet="" adapter="">port 0x7000-0x700f mem 0xfd1fc000-0xfd1fcfff,0xfd1fd000-0xfd1fdfff,0xfd1fe000-0xfd1fffff irq 17 at device 0.0 on pci8
    vmx5: Ethernet address: fc:d4:f2:df:00:03
    pcib28: <acpi pci-pci="" bridge="">at device 24.1 on pci0
    pcib28: [GIANT-LOCKED]
    pcib29: <acpi pci-pci="" bridge="">at device 24.2 on pci0
    pcib29: [GIANT-LOCKED]
    pcib30: <acpi pci-pci="" bridge="">at device 24.3 on pci0
    pcib30: [GIANT-LOCKED]
    pcib31: <acpi pci-pci="" bridge="">at device 24.4 on pci0
    pcib31: [GIANT-LOCKED]
    pcib32: <acpi pci-pci="" bridge="">at device 24.5 on pci0
    pcib32: [GIANT-LOCKED]
    pcib33: <acpi pci-pci="" bridge="">at device 24.6 on pci0
    pcib33: [GIANT-LOCKED]
    pcib34: <acpi pci-pci="" bridge="">at device 24.7 on pci0
    pcib34: [GIANT-LOCKED]
    acpi_acad0: <ac adapter="">on acpi0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    psm0: <ps 2="" mouse="">irq 12 on atkbdc0
    psm0: [GIANT-LOCKED]
    psm0: model IntelliMouse, device ID 3
    qpi0: <qpi system="" bus="">on motherboard
    orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    Timecounters tick every 1.000 msec
    nvme cam probe device init
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: <0x15ad> at usbus0
    uhub0: <0x15ad UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
    ugen1.1: <0x15ad> at usbus1
    uhub1: <0x15ad EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
    uhub0: 2 ports with 2 removable, self powered
    ugen0.2: <vmware>at usbus0
    uhid0: <vmware>on usbus0
    uhid1: <vmware>on usbus0
    ugen0.3: <vendor 0x0e0f="">at usbus0
    uhub2: <vmware virtual="" usb="" hub="">on usbus0
    (da0:mpt0:0:0:0): UNMAPPED
    da0 at mpt0 bus 0 scbus2 target 0 lun 0
    da0: <vmware virtual="" disk="" 2.0="">Fixed Direct Access SPC-4 SCSI device
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #1 Launched!
    da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
    da0: Command Queueing enabled
    da0: 8192MB (16777216 512 byte sectors)
    da0: quirks=0x40 <retry_busy>Trying to mount root from ufs:/dev/ufsid/55e0b2e79f855829 [rw]...
    Configuring crash dumps...
    uhub1: 6 ports with 6 removable, self powered
    Using /dev/label/swap0 for dump device.
    /dev/ufsid/55e0b2e79f855829: FILE SYSTEM CLEAN; SKIPPING CHECKS
    /dev/ufsid/55e0b2e79f855829: clean, 278972 free (10220 frags, 33594 blocks, 1.0% fragmentation)
    Filesystems are clean, continuing...
    Mounting filesystems...
    random: unblocking device.
     ___/ f \134
    / p \134___/ Sense
    \134___/   \134
    Welcome to pfSense 2.4.0-BETA...
    No core dumps found.
    ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
    32-bit compatibility ldconfig path: /usr/lib32
    uhub2: 7 ports with 7 removable, self powered
    External config loader 1.0 is now starting... da0s1 da0s1a da0s1b
    Launching the init system....... done.
    Initializing.................. done.
    Starting device manager (devd)...done.
    Loading configuration......done.
    Updating configuration...done.
    Cleaning backup cache.................................done.
    Setting up extended sysctls...done.
    Setting timezone...done.
    Configuring loopback interface...done.
    Starting syslog...done.
    Starting Secure Shell Services...done.
    Setting up interfaces microcode...done.
    Configuring loopback interface...done.
    Creating wireless clone interfaces...done.
    Configuring LAGG interfaces...done.
    Configuring VLAN interfaces...done.
    Configuring QinQ interfaces...done.
    Configuring WAN1 interface...
    vmx2: link state changed to UP
    Configuring LAN interface...
    vmx0: link state changed to UP
    Configuring WAN2 interface...
    vmx4: link state changed to UP
    Configuring WAN3 interface...
    vmx5: link state changed to UP
    gif0: link state changed to UP
    Configuring WAN4 interface...
    vmx1: link state changed to UP
    gif1: link state changed to UP
    arp: moved from 20:8c:d3:5f:00:f8 to 00:0b:82:63:de:14 on vmx0
    gif2: link state changed to UP
    Configuring MOBILE interface...
    vmx3: link state changed to UP
    gif0: link state changed to DOWN
    gif0: link state changed to UP
    gif1: link state changed to DOWN
    gif1: link state changed to UP
    gif2: link state changed to DOWN
    gif2: link state changed to UP
    gif3: link state changed to UP
    Configuring WAN1IPV6 interface...done.
    Configuring WAN2IPV6 interface...done.
    Configuring WAN3IPV6 interface...done.
    Configuring WAN4IPV6 interface...done.
    Configuring CARP settings...done.
    Syncing OpenVPN settings...done.
    pflog0: promiscuous mode enabled
    Configuring firewall...
    gif3: link state changed to DOWN
    gif3: link state changed to UP
    Starting PFLOG...done.
    Setting up gateway monitors...done.
    Starting DNS Resolver...done.
    Synchronizing user settings...done.
    Starting webConfigurator...done.
    Configuring CRON...done.
    Starting NTP time client...done.
    Starting DHCP service...done.
    Starting DHCPv6 service...done.
    Configuring firewall......done.
    Configuring IPsec VPN... route: writing to routing socket: Invalid argument
    route: writing to routing socket: Invalid argument
    Generating RRD graphs...done.
    Starting UPnP service... done.
    Starting syslog...done.
    route: writing to routing socket: Invalid argument
    route: writing to routing socket: Invalid argument
    Starting CRON... done.
     Starting package Open-VM-Tools...done.
     Starting package squid3...done.
     Starting package nmap...done.
     Starting /usr/local/etc/rc.d/c-icap.sh...done.
     Starting /usr/local/etc/rc.d/clamd.sh...done.
     Starting /usr/local/etc/rc.d/sqp_monitor.sh...done.
     Starting /usr/local/etc/rc.d/vmware-guestd.sh...done.
    pfSense 2.4.0-BETA amd64 Thu Jan 12 07:45:16 CST 2017
    Bootup complete
    cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de15, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:58:4904::4000, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2a00:1288:12c:2::4001, nxt 6, rcvif vmx0, outif gif1
    cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1</retry_busy></vmware></vmware></vendor></vmware></vmware></vmware></generic></isa></qpi></ps></at></keyboard></ac></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></vmware></acpi></acpi></vmware></acpi></acpi></ehci></uhci></acpi></acpi></lsilogic></vga-compatible></bridge></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></intel></software></vt></version></ptltd ></tscadj></lahf></syscall,nx,rdtscp,lm></sse3,vmx,ssse3,cx16,sse4.1,sse4.2,x2apic,popcnt,tscdlt,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt> 

    The error "route: writing to routing socket: Invalid argument
    route: writing to routing socket: Invalid argument" occurs multiple times.

    Does it come from the IPSec config? where to start searching?
    What can I do about "cannot forward src fe80:1::20b:82ff:fe63:de13, dst 2001:4998:c:e33::6000, nxt 6, rcvif vmx0, outif gif1"
    I see that same errors also on my FreeBSD VM which is running at the same ESXi host as pfSense does.

    Any help would be appreciated.


  • Rebel Alliance Developer Netgate

    Those errors are unrelated to IPsec.

    The "cannot forward" message is because your system is attempting to send non-link-local traffic using a link-local source. fe80 addresses cannot talk to anything outside the current L2, they are not routeable, so they can never reach that destination.

  • Maybe, I'll try to disable IPSec and see if the errors disappear.

    Can anyone tell me something about the "route: writing to routing socket: Invalid argument" error?
    Didn't have that one on 2.3.3

  • This one is related, and exactly the same problem: https://forum.pfsense.org/index.php?topic=117827.0
    The error "route: writing to routing socket: Invalid argument" is something else, and disappears when I disable one of my IPSec Tunnels (net2net).
    Roadwarrior IPSec (The other tunnel) is not working (tunnel itself does work, bus traffic does not flow), and gives the default deny error in firewall logs, as above topic.
    Tried to create the sloppy state floating rule, but is not working for me.

  • I see the same message on 2.4.0-BETA on my new sg-1000.
    Does anyone have a solution for this?

    edit: I also created that sloppy rule … does not work here.

    IPSEC tunnel(s) up, but traffic doesn't get through.
    The imported config works on another pfsense-2.3.3

  • Yeah, I went OpenVPN…

  • @maverick_slo:

    Yeah, I went OpenVPN…

    not a valid option for everyone. I have customers with IPSEC only.

  • I  know but I had no other option.. Migrated all to openvpn.
    Was pain in the ass but it was worth it…

  • Well, I assume if it works in 2.3.3 it should be solvable in 2.4.x as well.

  • What is recommended? Should I file a ticket for that issue or simply wait … ? ;-)