MultiWan - OPT1 (WAN2) doesn't work [Solved]

Raul Ramos · Dec 1, 2015, 8:15 PM

Hi

I have 2 WAN one pppoe other is a static connection (try DHCP) to a LTE modem. From pFsense Box i can ping and do DNS lookups fine. From clients i can't.

I have installed pFsense 2.3 on a hyper-V Machine like the version 2.2.5, backup and restore from 2.2.5 to 2.3 built on Tue Dec 01 11:48:28 CST 2015.

Try to add a route to the LTE modem subnet and nothing from the client, no ping or lookup to LTE IP or outside

Netstat

[2.3-ALPHA][admin@pfSense.*****]/root: netstat -r
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            dial-b1-****** UGS      pppoe1
10.0.0.0/22        link#6             U           hn1
pfSense            link#6             UHS         lo0
localhost          link#4             UH          lo0
bl****.dsl.tele link#8             UHS         lo0
192.168.2.0        link#7             U           hn2
192.168.2.3        link#7             UHS         lo0
dial-b1-******.te link#8             UHS      pppoe1
212.55.154.174     dial-b1-*****.te UGHS     pppoe1
212.55.154.190     dial-b1-*****.te UGHS     pppoe1
tserv1.lon2.he.net dial-b1-*****.te UGHS     pppoe1

Internet6:
Destination        Gateway            Flags      Netif Expire
default            *******-1.tunnel.ts UGS        gif1
localhost          link#4             UH          lo0
*******-1.tunnel.ts link#9             UH         gif1
*******-1-pt.tunnel link#9             UHS         lo0
2001:4**:****:**b: link#6             U           hn1
pfSense.*****      link#6             UHS         lo0
fe80::%lo0         link#4             U           lo0
fe80::1%lo0        link#4             UHS         lo0
fe80::%hn0         link#5             U           hn0
fe80::2**:**ff:fe0 link#5             UHS         lo0
fe80::%hn1         link#6             U           hn1
fe80::2**:**ff:fe0 link#6             UHS         lo0
fe80::%hn2         link#7             U           hn2
fe80::2**:**ff:fe0 link#7             UHS         lo0
fe80::%pppoe1      link#8             U        pppoe1
fe80::2**:**ff:fe0 link#8             UHS         lo0
fe80::%gif1        link#9             U          gif1
fe80::2**:**ff:fe0 link#9             UHS         lo0
ff01::%lo0         localhost          U           lo0
ff01::%hn0         fe80::2**:**ff:fe0 U           hn0
ff01::%hn1         fe80::2**:**ff:fe0 U           hn1
ff01::%hn2         fe80::2**:**ff:fe0 U           hn2
ff01::%pppoe1      fe80::2**:**ff:fe0 U        pppoe1
ff01::%gif1        *******-1-pt.tunnel U          gif1
ff02::%lo0         localhost          U           lo0
ff02::%hn0         fe80::2**:**ff:fe0 U           hn0
ff02::%hn1         fe80::2**:**ff:fe0 U           hn1
ff02::%hn2         fe80::2**:**ff:fe0 U           hn2
ff02::%pppoe1      fe80::2**:**ff:fe0 U        pppoe1
ff02::%gif1        ******-1-pt.tunnel U          gif1

ifconfig

[2.3-ALPHA][admin@pfSense.*****]/root: ifconfig
pflog0: flags=100 <promisc>metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
	syncpeer: 224.0.0.240 maxupd: 128 defer: on
	syncok: 1
enc0: flags=41 <up,running>metric 0 mtu 1536
	nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	nd6 options=21 <performnud,auto_linklocal>hn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=31b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6>ether 00:**:**:**:**:1d
	inet6 fe80::2**:**ff:fe00:**1d%hn0 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>hn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=31b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6>ether 00:**:**:**:**:1e
	inet6 fe80::2**:**ff:fe00:**1e%hn1 prefixlen 64 scopeid 0x6 
	inet 10.0.0.1 netmask 0xfffffc00 broadcast 10.0.3.255 
	inet6 2001:470:****:****::1 prefixlen 64 
	nd6 options=21 <performnud,auto_linklocal>hn2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=31b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6>ether 00:**:**:**:**:20
	inet6 fe80::2**:**ff:fe00:**20%hn2 prefixlen 64 scopeid 0x7 
	inet 192.168.2.3 netmask 0xffffff00 broadcast 192.168.2.255 
	nd6 options=21 <performnud,auto_linklocal>pppoe1: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
	inet 188.80.*.** --> 194.65.**.**** netmask 0xffffffff 
	inet6 fe80::22**:**ff:fe00:**1d%pppoe1 prefixlen 64 scopeid 0x8 
	nd6 options=21 <performnud,auto_linklocal>gif1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1452
	options=80000 <linkstate>tunnel inet 188.80.*.** --> 216.66.**.**
	inet6 2001:470:****:****::2 --> 2001:470:****:****::1 prefixlen 128 
	inet6 fe80::22**:**ff:fe00:**1d%gif1 prefixlen 64 scopeid 0x9 
	nd6 options=21<performnud,auto_linklocal></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></up,running></promisc>

Packets installed:

Snort (disable interfaces)
pfBlockerNG (disable)\
FreeRADIUS (doesn't start)
Cron

This version (2.3 built on Tue Dec 01 11:48:28 CST 2015) dosen't show packets Installed or available at this time (1 Dec 2015, 20:23 GMT). I want to uninstall some or all of them and nothing.

Some more info i can put here?

Tomorrow, maybe, i will configure the interfaces from default and report back.

EDIT: Chante Gateway monitor IP from modem IP to Google IP ~~i have Internet,~~ nevertheless no webGUI from modem (192.168.2.1).
But i have ping (dont know what i change to have it).
I have to reboot the LTE modem to check if the problem is the modem. Is not a easy access, and have battery to 3 hours.

Edit2: scratch "having internet" from wan2. I have but not from Wan2. Some rule (advanced option - Gateway) not working?
Thanks. Cya

cmb · Dec 1, 2015, 11:25 PM

You able to test using something other than Hyper-V? I suspect there is a problem of some sort with Hyper-V in 2.3's base OS judging by this and another report, which I haven't had a chance to dig into yet, but should soon. All the functionality you're describing works fine in general.

Raul Ramos · Dec 2, 2015, 12:51 AM

Hi

I have trunking wan and wan2 same physical connection using vlan in a switch to separate them, this works fine on 2.2.5.

Tomorrow i will configure interfaces without importing the 2.2.5 config and i have a AMD fusion motherboard maybe with a pcie nic + asix for wan's and integrated for lan i can test this topology.

I will be back

Cya

Guest · Dec 2, 2015, 8:36 AM

I have trunking wan and wan2 same physical connection using vlan in a switch to separate them, this works fine on 2.2.5.

Why? You could be able to use one of the Load balancing method;

session based load balancing
service based load balancing
policy based load balancing

You might be also doing this compared with a ratio pending on the throughput of each Internet connection
and together with two or three fail over rules you might be getting on top a fail over set up on top.

Raul Ramos · Dec 2, 2015, 11:39 AM

Hi

Wan and Wan2 are different kind of Wan's one adsl and the other is LTE. The ADSL is a 2.64 max download but very stable, i need this for a home webserver and one very prioritized service in the other hand the LTE with random download speeds (500kbs to 20Mbps ) and from 7:00PM to 2:00am rarely pass 2Mbps download in a good day and they block most of the standard server incoming ports. So load balancing this is a bad experience and a probably a bad idea(?), in my opinion. I decide to use the adsl to the services and the LTE for local clients usage in failover to adsl with lots of shaping.

But if i could, your idea would be great, thanks.

\offtopic

Today i create a new Hyper-V switch only for the wan2 and is a no go.

Thanks

fragged · Dec 2, 2015, 12:54 PM

This is likely a Hyper-V issue as per thread https://forum.pfsense.org/index.php?topic=102814.0

I've noticed similar issues but haven't tested enough to report anything in detail. Ping ok, pfSense GUI ok, but no internet access on virtual machine behind virtual pfSense on Hyper-V.

Raul Ramos · Dec 2, 2015, 1:41 PM

Well i can access pfsense webGui and have internet through WAN but not in OPT1 (WAN2) i can ping but no internet or the Interface of the modem on the opt1.

Raul Ramos · Dec 7, 2015, 5:26 PM

Some more testing with the last builds in this day (12/07/2015).

With pfsense in native hardware i can have multi gateway in my case with 3 Vlan in one interface (re0). This should be a Hyper-V problem.

Raul Ramos · Dec 12, 2015, 2:12 AM

Trying to shake this topic.

Some Packet Capture:

18:58:25.093420 00:15:5d:xx:xx:xx > b0:5b:67:xx:xx:xx, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 61609, offset 0, flags [DF], proto TCP (6), length 64)
192.168.2.3.50852 > 192.168.2.1.80: Flags (S), cksum 0x0557 (correct), seq 4202729516, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 411910690 ecr 0,sackOK,eol], length 0

192.168.2.3.50852 from router to modem through opt1 (Vlan11 on Hyper-v Machine settings network adapter hn2).

Build:
2.3-ALPHA (amd64)
built on Fri Dec 11 10:16:22 CST 2015

Thanks Cya

Raul Ramos · Dec 12, 2015, 2:24 AM

OK, for some reason thought i had tested disable all Checksum offload in all interfaces. Apparently not.

Disable TCP/UDP Checksum Offload and IPv4 Checksum offload in the Intel NIC 1000 PT in the host, not the interface created by the hyper-v Switch, i have network again in the OPT1 interface. No need to disable Offload on the pFsense or Hyper-v Interface Adaptor.

Always learning. I need to be more accurate in this testes next time.

Thanks. Cya