Strange Routing Issue

[ahole4sure]

So you would think that I would be able to do this at this point.

I am setting up my 4th or 5th pfsense system. It will be remote to my home office but I am trying to get it setup locally before deploying.

I am using an ATT modem (but with an assigned static IP address and in bridgem mode) for my temporary internet source. The WAN interface reports the proper public IP adreess and I have internet access in general.

The problem was recognized whem I could not get a connection with a setup site to site WG connection. As I troubleshot , I found weird behavior -- I could go into the diagnostics and ping google.com or cloudflare.com sucessfully - however if I tried to ping the static IP of my home office it would timeout (that address is easily able to be pinged from third PC, outside of the network).

What should I look for to find out why I can't ping an address that would be the endpoint of one my WG tunnel?

EDIT
It is so ODD -- I can ping almost any address (IP or name - like google.com, chase.com, etc). But when I try to ping the IP (numerical) or the DNS associated with that IP address of MYDOMAIN.com - it fails

[ahole4sure]

So I think I may have found at least part of the issue --- any help would be greatly appreciated.

I have a static IP (for my ATT modem)

I took the pfsense device out of the equation and with it directly connected to a PC I am able to reach certain websites , like :
google.com
cloudflare.com

but my domain which has DNS thorugh Cloudflare - the modem is not able to reach the site.
the attached are the traceroute results:

are there any suggestions ??

I assume only ATT can fix

[patient0]

@ahole4sure traceroute may or may not work (the devices on the way to the target may not support it). I'd say a ping is preferred for a quick test.

What are you referring to by '... able to reach certain websites'? Do some open when you access them in the web browser? And others not, and if not what error do you get?

I have a static IP (for my ATT modem)

How did you have pfSense configured, static IP and gateway (information you got from ATT?)? What DNS server have you set to be used?

And I'd think you have a good chance searching the forum for ATT, maybe someone else had a similar issue with this ISP.

[SteveITS]

@ahole4sure This is a cellular connection?

Does the firewall on your remote endpoint allow ICMP?

[stephenw10]

Mmm, I can't ping that either so it looks like it's blocking at least some sources.

[ahole4sure]

@stephenw10
@patient0
@SteveITS

THANKS for trying to help.
My head is spinning because I have too many isssues., lol
(they may be partly realted , maybe not completely)

The first thing that I have done more investigation is that there is for sure an issue with my ATT wireless modem
The way ATT has you to accomplish the static IP - is to give you a particular APN - and if you enter that data APN in the modem and it matches your SIM card it will give you access to your static IP address.
So to test things - (when I remove it from the pfsense situation and connect it directly to my Mac) and then enter or use the specific APN - then I get the weird issues where none of my normally hosted servers can be reached (if by web browser it just has no response - just locks page ) -- XXX.mydomain.com (DNS by cloudfare and access to my network only on 443 with haproxy - no open ports)

If I remove the specific APN and just use the normal Broadband APN then I can access my sites just fine.
So I concluded that my main issue is with ATT

What I am trying to accomplish is to set up Site to Site Wireguard VPN's so that the work is mostly done before I travel 5hrs to deploy for my daughter.
With the ATT modem set in the normal mode I can put it in IP Passthrough mode - but it is acting weird (I guess CGNAT ?) in that IP that the WAN is getting by DHCP on the status page reports as 10.168.25.44 - , but if I do what is my ip from a web browser and if I set up DDNS on pfsense - it reports IP as - 166.199.150.73
I have VPN pass through checked on my wireless modem - but I assume that is just not going to work for trying to setup a Site to Site -- I never a get a handshake from EITHER side

Is there a workaround to test and setup a Site to Site when you only have one internet source (or the second one is a cellular modem that I assume wont work becasue of CGNAT) ??

[ahole4sure]

@patient0
@stephenw10
@SteveITS

I was initally able to do my modified Site to Site using Tailscale and things were accesible -- but that isn't as reliable or fast as setting up a Site to Site Wireguard

With that said I have 2 site to sites setup with 2 of our locations -- there ahouldn't be a problem with creating a third as long as I seperate all subnets -- correct?

[SteveITS]

Yeah CGNAT breaks the inbound. If the other end has a public IP you can connect out to that, though. Do you get IPv6?

You could use a third device as a VPN server and connect both ends to that.

Somewhere recently I thought I saw something saying customers could not use 10.0.0.0/8 internally, and I think it was from AT&T. 20+ years ago we had a T1 ISP that used that for their network but still routed public IPs to the customer.

[ahole4sure]

@SteveITS

So in my existing I have site to site from A to B

Could I set up my new test to connect to site B (server) and have access from site A to the new test ???

[stephenw10]

Unclear what you're asking. You should be able to establish the tunnel from the cell modem side as long as the other end has a known public IP it can listen on. Once the tunnel is up you can access things across it either way.

[ahole4sure]

@stephenw10
OK. thank you - I know stupid question - but why do a site to site in the first place then if all can be accomplished the other way?

[stephenw10]

What other way do you mean? Using Tailscale? No reason if that works for you. As you said it may be faster with a direct WG tunnel. But you may not need that additional bandwidth.

[ahole4sure]

@stephenw10
No I meant that I didn't know I could establish a peer to site with the peer being behind CGNAT (in pfsense). -- is that what you are saying that I should be able to do that ?
But does that allow access from the non CGNAT site into the network of the CGNAT site ??

[stephenw10]

It's still site to site it's just that the tunnel can only be established from the side behind CGNAT. But, yes, once the tunnel is established connections can be opened across it both ways.

[ahole4sure]

@stephenw10

Thanks as always for your help!!!!

I am back to the setup again - and still trying to get past the same issue that I had initially (whhere I can't seem to get a hanshake)

I have a screenshot of the status page of the device that is behind CGNAT that should be initiating the tunnel

I relaize that there are many things like the WAN rule allowing 51825 in , etc
But is there a good way to troubleshoot WG connections that won't do a "handshake". (now that I know it should happpen going out from the CGNAT device)

1st screen is the remote CGNAT device status
2nd is the one with public IP

I can send screenshots of Rules or anything just to get me past the 15 to 20 hrs I've spent on this.
As you might can see -- I have been succesful in setting up WG connections before - both site to site and client to server

[stephenw10]

Are you now able to ping from the client to the server outside the tunnel?

I would check the states at both ends and filter by :51825. Make sure you're seeing WG opening states at both sides. And that they have two way traffic on them.

[ahole4sure]

@stephenw10

Going crazy here
I , TBH, am not sure whether I have been able to ping my public IP in the past or not. But I made sure I had created a WAN rule for allowing ping.
I am not able to ping from my laptop using another internet source.
I am not able to ping from the "client" pfsense box that is behind CGNAT

As far as the States - I sent screen shots form the running site to site on port 51821
AND I ssent screenshots from the non working connection with the client behind CGNAT 51825

Looks there is some partial traffic but not 2 -way traffic!

NOT SURE about the ping thing -- I have seemingly enabled ping to pass but I can't seem to ping my firewall from any device or any internet source BUT I have running services that are easily accesible from outside - that are coming in via https and haproxy

[SteveITS]

@ahole4sure ICMP to the public WAN IP from "any"/Internet would be allowed on the WAN interface which doesn't have that rule in your screenshot (may not be the right router).

ICMP to the public WAN IP from LAN would be a rule on the LAN interface, though that has a default allow all rule.

ICMP from within the VPN would be a rule on the Wireguard interface.

ICMP/ping is not required for other services to work.

[ahole4sure]

@stephenw10

Ok so some weird info --

I continued to not be able to ping - even with the rule (I think the screenshot you were referring to was my other router that I had not made that rule yet)
That made no sense -- so

I just temporarily disabled packet filtering all together -- result

1. I was then able to ping my main router
2. the previously "non-connecting" tunnel from the router behind CGNAT made the connection

I then renabled packet filtering

• became unable to ping my main router - depspite rule being in place
• the WG tunnel remained connected!

Seems like whatever is responsible for my "firewall" is not working correctly. I had rebooted last night and that didn't cause the incoming from CGNAT 51825 traffic to start -- but disabling the filtering did
?? - any suggestions
PS -- this pfsense install was from a complicated rough reinstall because the interface names were different -- the backup was hard to use ( that restore was a few months ago)

[SteveITS]

@ahole4sure What are the 42 alerts at the top? ;)

The 0/0 in the Status column means no matching connections at least since counting started.

Try a Diagnostics>Filter Reload.

https://docs.netgate.com/pfsense/en/latest/troubleshooting/firewall.html#new-rules-are-not-applied