Remote access Layer 2 works, Layer 3 no

Summer

@johnpoz thanks, client log are same as yours in Traceroute I can see:

ovpn.server
SITEA.1

So the gateway in use should be the correct one even if in main VPN connection info it show N/A.

Now if I open the SITEA/login in the browser I can see: HTTP SERVER LOG:

ovpn_client2 - - [28/Aug/2023:16:48:41 +0200] "GET /login HTTP/1.1" 404 45099 "-" "Mozilla/5.0 (Linux; Android 13; Nokia X10 Build/TKQ1.220807.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36"
ovpn_client2 - - [28/Aug/2023:16:48:41 +0200] "GET / HTTP/1.1" 403 94 "-" "Mozilla/5.0 (Linux; Android 13; Nokia X10 Build/TKQ1.220807.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36"
ovpn_client2 - - [28/Aug/2023:16:49:41 +0200] "GET /favicon.ico HTTP/1.1" 404 46502 "http://SITEA.1/" "Mozilla/5.0 (Linux; Android 13; Nokia X10 Build/TKQ1.220807.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36"

So routing incoming traffic to LAN should be good.

But on Pfsense LOG:

ovpn.client/PFSENSEWAN:40039 MULTI: bad source address from client [10.19.51.88], packet dropped

Now I'm reading but not understanding:

https://serverfault.com/questions/646130/multi-bad-source-address-from-client-any-one-off-solutions

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

bad source address from client [10.19.51.88],

What is this 10.19.51.88 address? If your remote site routes traffic down to the vpn server, how does the openvpn server know about this 10.19.51 network? If its not a known IP, and it sees traffic from it - then yeah it should drop it.

There is pretty good write up in what you linked too, if you setup a s2s - did you create the appropriate routes on each site for what is available on each site?

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

What is this 10.19.51.88 address?

This is mobile client CELL CONNECTION (has changed from Information screenshot was 10.12.225.115 ) ip is not External ipv4.

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

f your remote site routes traffic down to the vpn server, how does the openvpn server know about this 10.19.51 network?

If there's a way to look to iroutes of pfsense maybe we can discover it.

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

did you create the appropriate routes on each site for what is available on each site

I've setup Remote Access is not a site to site, just a road warrior that should be able to reach LAN.

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

This is mobile client CELL CONNECTION

if you have sitea and siteb - then they should a s2s connection. If you are going to have road warriors, ie cell out and about - then those should be a remote access setup..

You can setup both, you can have more than 1 openvpn server running on each pfsense.

If your going to have say phones using the wifi at say siteA, then you need to setup routes for this network the the phones will be on at either site a or site b, c etc..

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

if you have sitea and siteb - then they should a s2s connection. If you are going to have road warriors, ie cell out and about - then those should be a remote access setup..

That's exactly how it is, now the issue is road warrior can access siteb but not sitea that is pfsense LAN.

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

If your going to have say phones using the wifi at say siteA, then you need to setup routes for this network the the phones will be on at either site a or site b, c etc..

No wifi, only traffic from outside but I've only made a client specific override and with this I can reach SITEB but not SITEA

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

now the issue is road warrior can access siteb but not sitea that is pfsense LAN.

If you want your road warrior connection to get to some site other than the one they are connecting to for vpn. Then you have to create the routes for whatever tunnel network the roadwarrior is going to use.

If some rw IP comes in say 10.10.10.x and wants to go to site A from site B, even if site B has route to go to site A. how does site A know to get back?

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

Then you have to create the routes for whatever tunnel network the roadwarrior is going to use.

I don't undertstand route should be created by client specific overrides, however these are current routes:

based on this:

if roadwarrior.2 (10.7.31.2) open http://SITEA (http://10.7.208.1) page is loaded.
if roadwarrior.2 (10.7.31.2) open http://SITEA/login (http://10.7.208.1/login) error:

ovpn.client/192.168.100.100:40039 MULTI: bad source address from client [10.19.51.88], packet dropped

instead if roadwarrior.2 (10.7.31.2) open http://SITEB/login (http://10.7.218.1/login) correctly load the page.

Could it be that default Gateway 192.168.100.100 that is managed here interfere with SITEA routing?

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

if roadwarrior.2 (10.7.31.2) open http://SITEA/login (http://10.7.208.1/login) error:

Again not sure why you think that has anything to do with some pfsense problem.. to this road warrior there is no difference between http://something and http://something/otherthing/etc

There is no difference in those when it comes to networking that something will resolve to an IP.. the IP is the same in both.. what the full url is has nothing to do with anything unless your running through a proxy.. IP is IP the only thing that is in play here.. The syn will be sent to whatever that something IP is.. anything that happens after that has zero to do with routing..

Be it you do a get to just something or something/otherthing has nothing to do with routing or firewall rules since they are both to the same IP, which is whatever that something resolves too.

Now it could be a permissions thing on whatever is serving up something httpd.. But from a networking perspective those 2 things are exactly the same.. the something is an IP, which would be the same in either.

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

Now it could be a permissions thing on whatever is serving up something httpd.. But from a networking perspective those 2 things are exactly the same.. the something is an IP, which would be the same in either.

I'm just observing this behavior with layer 3 vpn, thanks for confirming this should not happen.

So routes are all here and they seems fine.

So next step for debug is turn off layer3 and androids clients and wait for next pfsense upgrade.

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

So next step for debug is turn off layer3 and androids clients and wait for next pfsense upgrade.

Sure you do you - but if you can get to http://something and not http://something/else has zero to do with being layer2 or layer3 or anything at all to do with openvpn.. So you could wait for the next 20 upgrades, not going to fix whatever problem is your seeing.

That is not a networking/openvpn issue -- to networking, firewalling and openvpn those 2 things are exactly the same. Something resolves to an IP or it doesn't - talking to that IP is no different if its just something or something/otherthing

Now you could have a permissions thing on your httpd server, where client from IP has permission to the root folder that something is suppose to serve up, but something/other that IP does not. This could explain why both work with layer2 because your going to be on the same IP the httpd is on most likely. But when your routed your going to be from different IP than the httpd server. And maybe this other IP on the httpd doesn't have permission to look at something/other, but it does have permission to look at just something.

what version your running of pfsense would have zero to do with that..

Summer

@johnpoz thank you for the advices

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

But when your routed your going to be from different IP than the httpd server. And maybe this other IP on the httpd does have permission to look at something/other, but it does have permission to look at just something.

I will inspect but based on pfsense log MULTI : bad source the request got to http server host and it register to access log not error log then I suppose packet is dropped by pfsense but there's a way to understand if this is coming from Firewall rules or elsewhere?

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

what version your running of pfsense would have zero to do with that..

I Hope it, but layer3 from pfsense 22.01 version with same config showed on GUI and road warriors where working.

johnpoz

@Summer Again from a networking perspective, routing/firewalling even openvpn there is NO difference between http://something and http://something/otherthing

The only thing that comes into play with both is the IP that something resolves too.. pfsense, the firewall nor openvpn have any care to what the full url is, only the IP the "something" resolves too.. It is not possible for something to resolve to something different if you go to something/other

Now if you were running traffic through proxy - the proxy would or could care and could do things differently depending on the full url. Your httpd could care what IP your coming from when talking to something/other vs just something.

maybe when you go to something/other you getting redirected to some other IP?? on the httpd server? But from a networking perspective that something is going to resolve to an IP.. What the full url is has nothing to do with if you can talk to that IP or not through your routing / firewalling

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

Now if you were running traffic through proxy - the proxy would or could care and could do things differently depending on the full url

Is there any Cache / Proxy enabled by default, how can I look from command line if there's anything running this kind of controls?

I've found docs about Reverse Proxy this could be the case but I don't have any package installed.

johnpoz

@Summer no there is no proxy installed forward or reverse out of the box, you would have to install the squid or the ha proxy package.

And then you would of had to configure them..

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

and then you would of had to configure them..

In previous version squid and squidgard were installed and running, but I've removed after 23.05 upgrade to simplify debug.

I've made a backup and it contains those configs:

<cron>
<item>
<task_name>lightsquid_parser_today</task_name>
<minute>45</minute>
<hour>23</hour>
<mday>*</mday>
<month>*</month>
<wday>*</wday>
<who>root</who>
<command>/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today</command>
</item>
<item>
<task_name>lightsquid_parser_yesterday</task_name>
<minute>15</minute>
<hour>0</hour>
<mday>*</mday>
<month>*</month>
<wday>*</wday>
<who>root</who>
<command>/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday</command>
</item>
<item>
<minute>0</minute>
<hour>0</hour>
<mday>*</mday>
<month>*</month>
<wday>*</wday>
<who>root</who>
<command>/usr/bin/nice -n20 /usr/pbi/squidguard-i386/etc/rc.d/squidGuard_logrotate</command>


<squidcache>
<config>
<nocache/>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache>MTAuNy4yMDguMTIxDQoxMC43LjIxOC4x</donotcache>
<enable_offline/>
<ext_cachemanager/>
<harddisk_cache_size>100</harddisk_cache_size>
<harddisk_cache_system>ufs</harddisk_cache_system>
<level1_subdirs>16</level1_subdirs>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4</maximum_object_size>
<memory_cache_size>64</memory_cache_size>
<maximum_objsize_in_mem>256</maximum_objsize_in_mem>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_dynamic_content/>
<custom_refresh_patterns/>
</config>
</squidcache>
<squidauth>
<config>
<auth_method>none</auth_method>
</config>
</squidauth>
<squid>
<config>
<enable_squid/>
<keep_squid_data/>
<listenproto>inet</listenproto>
<carpstatusvid>none</carpstatusvid>
<active_interface>lan,opt1,opt3,opt5</active_interface>
<outgoing_interface>auto</outgoing_interface>
<proxy_port>3128</proxy_port>
<icp_port/>
<allow_interface>on</allow_interface>
<dns_v4_first/>
<disable_pinger/>
<dns_nameservers/>
<extraca>none</extraca>
<transparent_proxy>on</transparent_proxy>
<transparent_active_interface>lan,opt1,opt3,opt5</transparent_active_interface>
<private_subnet_proxy_off/>
<defined_ip_proxy_off>10.7.208.1;</defined_ip_proxy_off>
<defined_ip_proxy_off_dest>10.7.208.0</defined_ip_proxy_off_dest>
<ssl_proxy/>
<sslproxy_mitm_mode>splicewhitelist</sslproxy_mitm_mode>
<ssl_active_interface>lan</ssl_active_interface>
<ssl_proxy_port/>
<sslproxy_compatibility_mode>modern</sslproxy_compatibility_mode>
<dhparams_size>2048</dhparams_size>
<dca>none</dca>
<sslcrtd_children/>
<interception_checks/>
<interception_adapt/>
<log_enabled/>
<log_dir>/var/squid/logs</log_dir>
<log_rotate/>
<log_sqd/>
<visible_hostname>localhost</visible_hostname>
<admin_email>admin@localhost</admin_email>
<error_language>en</error_language>
<xforward_mode>on</xforward_mode>
<disable_via/>
<uri_whitespace>strip</uri_whitespace>
<disable_squidversion>on</disable_squidversion>
<custom_options/>
<custom_options_squid3/>
<custom_options2_squid3/>
<custom_options3_squid3/>
</config>
</squid>
<squidguardgeneral>
<config>
<squidguard_enable>on</squidguard_enable>
<ldap_enable/>
<ldapbinddn/>
<ldapbindpass/>
<ldapcachetime>0</ldapcachetime>
<stripntdomain/>
<striprealm/>
<ldapversion>3</ldapversion>
<rewrite_children>16</rewrite_children>
<rewrite_children_startup>8</rewrite_children_startup>
<rewrite_children_idle>4</rewrite_children_idle>
<enable_guilog/>
<enable_log/>
<log_rotation/>
<adv_blankimg>on</adv_blankimg>
<blacklist>on</blacklist>
<blacklist_proxy/>
<blacklist_url>http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz</blacklist_url>
</config>
</squidguardgeneral>
<squidguarddefault>
<config>
<dest>!blocca !blk_blacklists_adult !blk_blacklists_agressif !blk_blacklists_arjel !blk_blacklists_associations_religieuses !blk_blacklists_astrology !blk_blacklists_cryptojacking !blk_blacklists_dating !blk_blacklists_ddos !blk_blacklists_doh !blk_blacklists_download !blk_blacklists_examen_pix !blk_blacklists_exceptions_liste_bu !blk_blacklists_forums !blk_blacklists_games !blk_blacklists_malware !blk_blacklists_marketingware !blk_blacklists_mixed_adult !blk_blacklists_phishing !blk_blacklists_radio !blk_blacklists_redirector !blk_blacklists_remote-control !blk_blacklists_stalkerware !blk_blacklists_strict_redirector !blk_blacklists_strong_redirector !blk_blacklists_tricheur !blk_blacklists_update !blk_blacklists_warez all</dest>
<notallowingip/>
<deniedmessage>proxy denied errorr</deniedmessage>
<redirect_mode>rmod_int</redirect_mode>
<redirect/>
<safesearch>on</safesearch>
<rewrite>safesearch</rewrite>
<enablelog>on</enablelog>
</config>
</squidguarddefault>
<squidreversegeneral>
<config>
<reverse_listenproto>inet</reverse_listenproto>
<reverse_interface>wan</reverse_interface>
<reverse_ip/>
<reverse_external_fqdn/>
<deny_info_tcp_reset/>
<reverse_http/>
<reverse_http_port/>
<reverse_http_defsite/>
<reverse_https/>
<reverse_https_port/>
<reverse_https_defsite/>
<reverse_ssl_cert>none</reverse_ssl_cert>
<reverse_int_ca/>
<reverse_ignore_ssl_valid/>
<reverse_check_clientca/>
<reverse_ssl_clientca>none</reverse_ssl_clientca>
<reverse_ssl_clientcrl>none</reverse_ssl_clientcrl>
<reverse_compatibility_mode>modern</reverse_compatibility_mode>
<dhparams_size>2048</dhparams_size>
<disable_session_reuse/>
<reverse_owa/>
<reverse_owa_ip/>
<reverse_owa_activesync/>
<reverse_owa_rpchttp/>
<reverse_owa_mapihttp/>
<reverse_owa_webservice/>
<reverse_owa_autodiscover/>
</config>
</squidreversegeneral>
<squidguarddest>
<config>
<name>blocca</name>
<domains>peppapig.com anycast.dns.nextdns.io</domains>
<urls/>
<expressions/>
<redirect_mode>rmod_none</redirect_mode>
<redirect>badbadsitepleasecontactadmin</redirect>
<description>
<![CDATA[ block ]]>
</description>
<enablelog/>
</config>
</squidguarddest>
<squidnac>
<config>
<allowed_subnets/>
<unrestricted_hosts>MTAuNy4yMDguNDAvMjQNCjEwLjcuMjA4LjE0MC8yNA0KMTAuNy4yMDguMTQxLzI0</unrestricted_hosts>
<banned_hosts/>
<whitelist/>
<blacklist/>
<block_user_agent/>
<block_reply_mime_type/>
<addtl_ports/>
<addtl_sslports/>
<google_accounts/>
<youtube_restrict>none</youtube_restrict>
</config>
</squidnac>
<squidantivirus>
<config>
<enable/>
<client_info>none</client_info>
<enable_advanced>disabled</enable_advanced>
<clamav_url/>
<clamav_scan_type>all</clamav_scan_type>
<clamav_disable_stream_scanning>on</clamav_disable_stream_scanning>
<clamav_block_pua>on</clamav_block_pua>
<clamav_update>24</clamav_update>
<clamav_dbregion>uk</clamav_dbregion>
<clamav_dbservers/>
<urlhaus_sig/>
<interserver_sig/>
<securiteinfo_sig/>
<securiteinfo_premium/>
<securiteinfo_id/>
<raw_squidclamav_conf/>
<raw_cicap_conf/>
<raw_cicap_magic/>
<raw_freshclam_conf/>
<raw_clamd_conf/>
</config>
</squidantivirus>

Could it be that some misconfiguration is still present even if the packages are not installed?

Summer

I've collected info on the client:

chrome://net-export/

Now I've found out that it ask to switcht to https

{"params":{"headers":["Host: 10.7.208.1","Connection: keep-alive","DNT: 1","Upgrade-Insecure-Requests: 1","User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36","Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","Referer: http://10.7.208.1/static/build/html/test.html","Accept-Encoding: gzip, deflate","Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7"],"line":"GET /newrap/ HTTP/1.1\r\n"},"phase":0,"source":{"id":364,"start_time":"55561518","type":1},"time":"55561591","type":173},

Local server got no HTTPS capabilities.

Found out also this to test dns resolve:

chrome://net-internals/#proxy

johnpoz

@Summer well turn that off on your browser.. Possible when your on the same network as the server, maybe it doesn't do that - not really a big chrome user. Could be related to your hsts policy, could be "privacy" setting to switch everything to https, but you would think it smart enough that if the resource not available, not to switch.

But that for sure would explain your problem ;)

edit: is this set?

That could possible explain why it works in incognito mode - since it doesn't switch when your in that mode? Again not a big chrome user.

Summer

@johnpoz the "Always use secure connections" was already deactivate.

Anyway still got file share samba access issue it just stays in a wait forever state too but looking at server log:

-from remote : /path/samba_OVPN.TUNNEL.CLIENT.log show:

[2023/08/30 16:35:17.335894,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "lanman auth" option is deprecated
[2023/08/30 16:35:17.335946,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "raw NTLMv2 auth" option is deprecated
[2023/08/30 16:35:17.336055,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "encrypt passwords" option is deprecated

from local lan /path/samba_LAN.client.log show:

[2023/08/30 16:35:55.314534,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "lanman auth" option is deprecated
[2023/08/30 16:35:55.314580,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "raw NTLMv2 auth" option is deprecated
[2023/08/30 16:35:55.314671,  1] ../../lib/param/loadparm.c:1911(lpcfg_do_global_parameter)
  lpcfg_do_global_parameter: WARNING: The "encrypt passwords" option is deprecated

But with last it access instantaneusly the files...

johnpoz

@Summer said in Remote access Layer 2 works, Layer 3 no:

WARNING: The "lanman auth" option is deprecated

This has zero to do with the networking/firewall/openvpn aspect of connectivity.. What version of samba are you using, what client - those errors yeah point to trying to use old deprecated auth methods..

pfsense would be involved in the routing of the traffic, firewall rules based on ports IPs, openvpn doesn't care it just moves packets through its tunnel..

How you could think that the update of pfsense could have anything to do with such errors in your samba setup?

Summer

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

pfsense would be involved in the routing of the traffic, firewall rules based on ports IPs, openvpn doesn't care it just moves packets through its tunnel..

From pftop I can see traffic from outside that reach LAN with ESTABLISHED status, but can't send back replies:

request: 
roadwarrior.client > roadwarrior.tunnel > roadwarrior.pfsense > lan

reply: lan > REMOTECLIENT.INTERNETPROVIDER.NETWORK (nowhere)

Instead of:

request: roadwarrior.client > roadwarrior.tunnel > roadwarrior.pfsense > lan

reply: lan  > roadwarrior.pfsense  > roadwarrior.client

So if routes are fine why this happen and only for samba and webbrowsing (i.e. VNC and Voip are working)?

@johnpoz said in Remote access Layer 2 works, Layer 3 no:

How you could think that the update of pfsense could have anything to do with such errors in your samba setup?

The same server works with layer2 because no routing is involved, what manage routing? Actually is still pfsense.

I'm just trying to find out the reasons and understand how to fix something.