Weird LAN behaviour - LAN to internet



  • Trying to discover whats going on here....

    My LAN seems to work fine via Ethernet connection when the wifi is left turned on (on the linux laptop Im using) - ie laptop reports via wicd, connected via cable. I thought it was all set up could be left alone. BUT - Turn wifi off on laptop, via manual switch and no internet - webconfig still connecting. (LAN not set as gateway and receiving the correct LAN ip address from dchp)
    Any ides


  • Netgate Administrator

    Not using pfSense for DHCP, some other rogue DHCP server?

    Does it appear in the pfSense lease table with the correct MAC address?

    Is the pfSense LAN dhcp server at it's defaults still?

    What is the wifi access point here? Something that isn't pfSense and using the same subnet perhaps?

    Steve



  • My setup is pretty much this

    pfsense.jpg

    NAT out

    	WAN 	127.0.0.0/8 	* 	* 	* 	WAN address 	* 		Localhost to WAN 	
    	WAN 	10.0.0.0/24 	* 	* 	* 	WAN address 	* 		LAN TO WAN 	
    	WAN 	10.0.10.0/24 	* 	* 	* 	WAN address 	* 		VL10_MGMT to WAN 	
    	WAN 	10.0.20.0/24 	* 	* 	* 	WAN address 	* 		VL20_VPN to WAN 	
    	WAN 	10.0.30.0/24 	* 	* 	* 	WAN address 	* 		VL30_CLRNET to WAN 	
    	WAN 	10.0.40.0/24 	* 	* 	* 	WAN address 	* 		VL40_GUEST to WAN 	
    	VPN_WAN2 	10.0.20.0/24 	* 	* 	* 	VPN_WAN2 address 	* 		VL20_VPN to VPN_WAN 	
    	VPN_WAN1 	10.0.20.0/24 	* 	* 	* 	VPN_WAN1 address 	* 		VL20_VPN to VPN1_WAN 
    

    All sub nets on dhcp separate range and rules for sub nets and ports
    LAN giving out 10.0.0.19 via Ethernet cable to LAN interface igb1
    only noticed the problem when I turned off the wifi adapter on the notebook which had been connected to either vlan30(clear net) or vlan20(OPENVPN)

    States when pinging 9.9.9.9 from LAN ethernet

    LAN icmp 10.0.0.19:15326 -> 9.9.9.9:15326 0:0 21 / 21 2 KiB / 2 KiB
    WAN icmp 73.245.92.68:7006 (10.0.0.19:15326) -> 9.9.9.9:7006 0:0 21 / 21 2 KiB / 2 KiB

    pings are going out via isp not openvpn only 12ms
    But no http or https via lan
    LAN is connected to the correct MAC on the laptop in the table


  • Netgate Administrator

    So before disconnecting the wifi the client had two connections to the same subnet?

    That seems invalid.

    Steve



  • It would seem so. Ping was going to the isp I guess as the return was quick -12ms from US - and http/s was via the VPN which at this time is Netherlands for leaktest.com - ping from European servers sem to be around 220ms

    ifconfig gives

    Link encap:Ethernet HWaddr f0:dxxxxxxx
    inet addr:10.0.0.19 Bcast:10.0.0.255 Mask:255.255.255.0
    inet6 addr: fe80::abe4:c19:4c94:98f4/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:55304 errors:0 dropped:0 overruns:0 frame:0
    TX packets:50052 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:24017385 (24.0 MB) TX bytes:6618596 (6.6 MB)
    Interrupt:20 Memory:f2600000-f2620000

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:65536 Metric:1
    RX packets:785515 errors:0 dropped:0 overruns:0 frame:0
    TX packets:785515 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:107283334 (107.2 MB) TX bytes:107283334 (107.2 MB)

    wlan0 Link encap:Ethernet HWaddr 00:27:10:87:4xxxx
    inet addr:10.0.20.17 Bcast:10.0.20.255 Mask:255.255.255.0
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1



  • ifconfig gives two addresses at the same time eth0 10.0.0.19 and wlan0 10.0.20.17

    Link encap:Ethernet HWaddr xxxxxxxxxxx
    inet addr:10.0.0.19 Bcast:10.0.0.255 Mask:255.255.255.0
    inet6 addr: fe80::abe4:c19:4c94:98f4/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:55304 errors:0 dropped:0 overruns:0 frame:0
    TX packets:50052 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:24017385 (24.0 MB) TX bytes:6618596 (6.6 MB)
    Interrupt:20 Memory:f2600000-f2620000

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:65536 Metric:1
    RX packets:785515 errors:0 dropped:0 overruns:0 frame:0
    TX packets:785515 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:107283334 (107.2 MB) TX bytes:107283334 (107.2 MB)

    wlan0 Link encap:Ethernet HWaddr 00:27:xxxxxxxxx
    inet addr:10.0.20.17 Bcast:10.0.20.255 Mask:255.255.255.0
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


  • Netgate Administrator

    Hmm, but not in the same subnet. What happens if you turn off the wifi before you connect the Ethernet?
    I would check the routing table on the client, it seems like it just lost it's default route.

    Steve



  • Yep realized the subnet thing - eventually.
    if wifi turned off first the only ip is the ethernet 10.0.0.0/24

    States for all when attempting to connect to dnsleaktest.com via ip address of (and filtered for that ip) 23.239.16.110 from firefox

    LAN tcp 10.0.0.19:52364 -> 23.239.16.110:80 CLOSED:SYN_SENT 6 / 0 360 B / 0 B
    VPN_WAN2 tcp 10.0.0.19:52364 -> 23.239.16.110:80 SYN_SENT:CLOSED 6 / 0 360 B / 0 B
    LAN tcp 10.0.0.19:52370 -> 23.239.16.110:80 CLOSED:SYN_SENT 5 / 0 300 B / 0 B
    VPN_WAN2 tcp 10.0.0.19:52370 -> 23.239.16.110:80 SYN_SENT:CLOSED 5 / 0 300 B / 0 B
    LAN tcp 10.0.0.19:37646 -> 23.239.16.110:443 CLOSED:SYN_SENT 5 / 0 300 B / 0 B
    VPN_WAN2 tcp 10.0.0.19:37646 -> 23.239.16.110:443 SYN_SENT:CLOSED 5 / 0 300 B / 0 B
    LAN tcp 10.0.0.19:37648 -> 23.239.16.110:443 CLOSED:SYN_SENT 5 / 0 300 B / 0 B
    VPN_WAN2 tcp 10.0.0.19:37648 -> 23.239.16.110:443 SYN_SENT:CLOSED 5 / 0 300 B / 0 B

    ARP routing table

    VLAN_10_OPT4 10.0.10.11 18:e8:xxxxxxx:8b AP200 Expires in 604 seconds vlan
    WAN 73.245xx 18:8b:xxx:xx:28:xx c-73-245xxx.hsd1.fl.comcast.net Expires in 157 seconds ethernet
    WAN 73.245.92.xx 00:1b:21:ba:xxxx c-73-xxx-9xxxx.hsd1.fl.comcast.net Permanent ethernet
    VLAN_20_OPT5 10.0.20.12 34:a3:95:88:xxxc iPhone Expires in 1165 seconds vlan
    VLAN_20_OPT5 10.0.20.11 60:f4:45xxxx iPhone.localdomain Expires in 1146 seconds vlan
    LAN 10.0.0.10 00:1b:21:xxxxxxx pfSense.localdomain Permanent ethernet
    LAN 10.0.0.19 f0🇩🇪f1:0dxxxxx user-ThinkPad-T410.localdomain Expires in 1176 seconds ethernet
    VLAN_20_OPT5 10.0.20.16 04:52:f3:8xxxx users-iPad Expires in 1173 seconds vlan
    VLAN_40_OPT7 10.0.40.1 00:1b:21xxxxxx Permanent vlan
    VLAN_30_OPT6 10.0.30.1 00:1b:21xxxxxx Permanent vlan
    VLAN_30_OPT6 10.0.30.10 00:27:10:8xxxx Expires in 623 seconds vlan
    VLAN_20_OPT5 10.0.20.1 00:1b:21:bxxxxxx Permanent vlan
    VLAN_10_OPT4 10.0.10.1 00:1b:21:xxxxxxxx Permanent vlan
    IGB3 10.0.50.1 00:1b:2xxxxxx


  • LAYER 8 Global Moderator

    You know multihoming devices with both ethernet and wireless at the same time is normally not a good idea... This is why its common for the wifi interface to turn off when connected to a wire... This is very common in laptops. Generally multihoming devices be it wired/wired wireless/wireless or wired/wireless is normally not good setup.

    If not directly supported in the wifi driver you can do it like this
    https://www.dell.com/support/article/us/en/19/how11386/how-to-disable-a-wireless-network-connection-when-a-wired-connection-is-detected?lang=en

    linux has multiple options for doing this as well.

    Yes normally the routing metrics will have the box prefer the wired connection since it should be faster, etc.



  • Thanks. Most of this is above my pay grade and trying to learn as much as I can to get the box working as I want.
    Took some time to realize that the wifi was still connecting even though the Wicd network manager was only saying wired connection - would have helped if Id looked at ifconfig full results rather than just eth0

    As it stands, (wired connection to LAN, wifii off) seems no matter how I fiddle with the rules there is nothing coming in on a browser page, though ipcm pings are ok from the linux client and from within the pfsense ping window

    The vlans and openvpn are all working as they should - just the LAN that wont play


  • LAYER 8 Global Moderator

    well "lan" default rules are any any - did you mess with that?... Can you ping pfsense lan IP? What is the clients route table when only connected the lan... You have pfsense set as your gateway?

    What is being handed out for dns when only on the wired lan?



  • OK – A bit confused here and apologies for my lack of understanding, but can ping to 10.0.0.10 which is the webconfig addresss
    Can ping to say 9.9.9.9 form client linux pc wired to LAN.
    And also from inside pfsense ping window
    dhcp on psfense giving addesses as set

    LAN rules have been set the same as rules in the VLAN20 (OPENVPN) subnet as I wanted the same results (could be totally screwed up her thru a lack of understanding)
    Removing all those rules and just having allow all to all makes no difference except it stops ping from my linux client pc – which I didn’t understand as I thought all meant everything?

    Getting routes for igb1 (LAN interface) not set as gateway – default gateway as set by pfsense

    pfSense.localdomain   Firewall  Rules  LAN.png

    Unsure what client route table was unless its

    Destination Gateway Flags Use Mtu Netif Expire
    10.0.0.0/24 link#2 U 656 1500 igb1

    Additionally, was not sure how to see “What is being handed out for dns when only on the wired lan?”
    Another strange thing, when connected to an windows7 PC still no web pages but windows reports the connection has internet - so could some ports or protocol be blocked? Tho 80 and 443 are allowed ports. And the subnets and ports are fine on the VPN subnet


  • LAYER 8 Global Moderator

    there should be a default route..

    Your on linux right? what does netstat -rn show?

    root@uc:/tmp# netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         192.168.2.253   0.0.0.0         UG        0 0          0 ens3
    192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 ens3
    root@uc:/tmp# 
    

    windows would be
    route print from cmd line

    as to what your using for dns.. you can normally look in /etc/resolv.conf

    are you using the network manager gui? It should show you what using.

    As to your rules not sure what you then think that vpn_wan2 address is ever going to be a source of traffic into your lan interface



  • The dns is from the isp provider either comcast or the VPN not set on the connection cant seem to get it to respond to a set dns in pfense setup

    I thought the vpn_wan2 address rule would route the lan thru the VPN - but as I said am out of my depth here
    netstat.....

    user@user-ThinkPad-T410 ~ $ netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.0.0.10       0.0.0.0         UG        0 0          0 eth0
    10.0.0.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    user@user-ThinkPad-T410 ~ $ 
    
    
    
    

  • Netgate Administrator

    If you can ping 9.9.9.9 from the client then the routes are good.

    That 'wan out' rule is almost certainly wrong on the LAN interface. You should never see traffic sourced from the VPN_WAN2 address arrive at the LAN interface.

    The only rule that can pass a connection to a general website in the unlabelled rule. That is passing everything via the VPN gateway. There are no states shown on it though. It looks like no traffic is hitting that.

    Steve


  • LAYER 8 Global Moderator

    not sure why you would have that 169.254 address there.. APIPA??? you shouldn't have that - but its not going to hurt anything.

    But don't see any hits on your rules going anywhere..

    Are you running pfblocker? That runs a vip on your lan and can mess with your ! rule you have their with the local subnets that should allow access out to the internet... But its forcing traffic out your vpn.



  • @stephenw10
    Thanks
    Ive disabled all the rules on LAN except lockout and a new rule IvP4 allow all to all
    ping still gets reply but no other sign of internet via wired LAN (with client VPN disabled and running same result)
    I did notice when connected to windows7 which I have\ dual boot with linux that windows reports there in an internet connection tho no web pages are avaialble.



  • @johnpoz
    Thanks, Ive no idea where the 192.254 is coming from - it only appears when wired LAN connected, nothing visible in network settings manager.

    Not running pfblocker or any addon (gave upon pfblocker on an other Atom 2.4g box as it seemed to be causing the cpu to run at between 80-100% and couldn't find a fix)

    With no rules other than anitlockout and a pass all to all rule on LAN no webpages are getting thru but ICMP is fine


  • LAYER 8 Global Moderator

    Are you running proxy?

    Those 169.254 can often show up when you have set for dhcp but don't get an address..

    Does any box on your wired lan work? Do you see devices in your dhcp lease table on your lan? Do you have any floating rules?

    Your rules are forcing traffic out your vpn!! Do you see hits on the rules? What does your outbound nat look like if your trying to send traffic out a vpn.

    Why don't you make your rules any any - default out of the box on lan... And now do something as simple as a wget on your linux box to say www.google.com... Does it grab anyway.



  • No, no proxy
    wget via lan is not resolved - I'm thinking its a dns issue on the lan
    perhaps the dns resolver and forwarder settings for the vlans, which I think prevent dns lookup when the vpn is down is preventing dns resolving on the lan.
    From inside pfsense the dns lookup is ok and comes from the settings in general setup

    I'm trying to recreate this https://nguvu.org/pfsense/pfsense-baseline-setup/ but there is no lan in the description.

    Is there a way to give the lan dns servers without affecting anything else? That is, if that's the issue as the vlans work well.


  • LAYER 8 Global Moderator

    out of the box pfsense resolver will listen on all interfaces... But your rules don't even allow your client to talk to pfsense - since your shoving everything out your vpn!

    Remove that policy route.. And allow for lan to talk to pfsense lan IP tcp/udp port 53 Directly query pfsense lan IP for something.. Does it resolve?

    use dig or host or nslookup on your client - pointing directly to pfsense lan IP.

    Well maybe you do - but you have alias for both source and dest on the rule that might allow it.



  • Thanks for all of the help!

    I changed the lan interface nic assignment just in case it had some problem (no idea if that's a possibility as icmp was running ok but anyways things seemed intermittent - sometimes ping to google.com was resolved then ping to say bbc.co.uk was not resolved and no names were found - all the time ping to number address worked -.9.9.9.9. for eg.
    Ping and dns lookup always work from inside the pfsense box.

    Removed all rules on lan except allow all to all - rebooted a few times, changed cables - restarted client pc and set new connection to run
    And.... the thing came to life with www available.
    Thanks again this forum is a great resource!!


  • Netgate Administrator

    Hmm, well always slightly disappointing not to find a definite problem but I'm glad you got it up and running. 👍

    Steve



  • It was quire odd as I'd tried all the removing rules on LAN except all to all before and got nowhere.
    Had just thought that maybe the connection on my linux client might have been the problem as Ive just noticed a dns tab which I had set to auto on the working connection. And the dns servers set in general on pfsense were listed. Previously it was off - I have no idea if anything needs to be set on a linux cable connection to get to www pages, I just assumed it just took whatever was sent from the router box as seemed to be the case with my last router (asus68u) which was really slow at vpn - speeds are 3x faster on this, which was the reason to change. I"m still thinking it was some failure of dns, however that might happen.
    Anways, its good to know one can get help when needed _thanks again!



  • Final note on this problem......

    Using linux mint 18.3 client to check pfsense setup

    what I found was the ethernet connection to pfsense was very fussy on how it was set - I wasted so much time hunting for errors on the pfsense box when it was the connection which was at fault - If set to the subnet address of 10.0.20.16 and auto dns for eg there was no internet, even though the connection looked ok ( not really sure what was going on here but thinking the dns was not getting in/out) pings to number addresses ok named not found.

    If set to dchp and auto dns the connection was good and things worked as they should.

    Things got very confusing when setting thre cisco sg3500 switch too as that would not show its webconfig without being set to a fixed ip.
    The moral here is check the function of the connection before attempting to change the pfsense settings!


  • LAYER 8 Global Moderator

    @fin1000 said in Weird LAN behaviour - LAN to internet:

    what I found was the ethernet connection to pfsense was very fussy on how it was set

    huh?? What is there to set.. Just leave it in auto... The only thing you should be doing on an interface in pfsense is settings its ip to be honest..


  • Netgate Administrator

    Hmm, that sounds a lot like something has a bad subnet mask set.



  • Ive no idea having very little understanding of networking "black magic"
    But have had many problems over the years with connections failing to connect and the "connecting" icon "churning" until it times out. Which then leads to messing with settings to see if it can be connected. Try a fixed address in the right range - which normally connects but then there is nothing visible at the other end.
    sudo ip route flush cache, sudo ip route flush table main, sudo iptables -F did seem to allow a connection to start BUT im still in the dark as to why sometimes ethernet wont connect.


  • LAYER 8 Global Moderator

    @fin1000 said in Weird LAN behaviour - LAN to internet:

    Ive no idea having very little understanding of networking "black magic"

    WTF are you running pfsense and sg350 switches then???

    You shouldn't be setting static IPs if your dhcp is not working you should fix the dhcp issue!

    I have been using ethernet from day one... Back in the thick and thinnet days... Vampire connections and bnc... Way before rj45 and cat level cables..

    Sounds like you have trouble at layer 1 if you ask me if your having issues with stuff connecting or stuff like the basics of dhcp to even work. Maybe you have bad cables - are you making your own? Or buy prefab?


  • Netgate Administrator

    Mmm, I would start basic here and work up.

    With close to default pfSense settings any client set to DHCP should pull a lease and be given valid settings.

    Steve



  • Actually I've learnt a lot form the guidance given here. But it has to be infuriating to deal with the problems of numpties who don't really understand how stuff works but non the less want to set up something which is beyond them.

    Cables are bought and changed about to check for errors tho not got a connection checker here.

    What I meant to be saying in my final note on the problem was that a good deal of time was wasted because I hadn't realised Id got a problem with dhcp on the client and restarting linux didnt flush the network if that’s even the correct phrase. sudo ip route flush table main – after which it got better -or maybe I just imagined it ;) Anyways, its doing what it should now - Thanks


Log in to reply