Strange Routing Issue

[ahole4sure]

So you would think that I would be able to do this at this point.

I am setting up my 4th or 5th pfsense system. It will be remote to my home office but I am trying to get it setup locally before deploying.

I am using an ATT modem (but with an assigned static IP address and in bridgem mode) for my temporary internet source. The WAN interface reports the proper public IP adreess and I have internet access in general.

The problem was recognized whem I could not get a connection with a setup site to site WG connection. As I troubleshot , I found weird behavior -- I could go into the diagnostics and ping google.com or cloudflare.com sucessfully - however if I tried to ping the static IP of my home office it would timeout (that address is easily able to be pinged from third PC, outside of the network).

What should I look for to find out why I can't ping an address that would be the endpoint of one my WG tunnel?

EDIT
It is so ODD -- I can ping almost any address (IP or name - like google.com, chase.com, etc). But when I try to ping the IP (numerical) or the DNS associated with that IP address of MYDOMAIN.com - it fails

[ahole4sure]

So I think I may have found at least part of the issue --- any help would be greatly appreciated.

I have a static IP (for my ATT modem)

I took the pfsense device out of the equation and with it directly connected to a PC I am able to reach certain websites , like :
google.com
cloudflare.com

but my domain which has DNS thorugh Cloudflare - the modem is not able to reach the site.
the attached are the traceroute results:

are there any suggestions ??

I assume only ATT can fix

[patient0]

@ahole4sure traceroute may or may not work (the devices on the way to the target may not support it). I'd say a ping is preferred for a quick test.

What are you referring to by '... able to reach certain websites'? Do some open when you access them in the web browser? And others not, and if not what error do you get?

I have a static IP (for my ATT modem)

How did you have pfSense configured, static IP and gateway (information you got from ATT?)? What DNS server have you set to be used?

And I'd think you have a good chance searching the forum for ATT, maybe someone else had a similar issue with this ISP.

[SteveITS]

@ahole4sure This is a cellular connection?

Does the firewall on your remote endpoint allow ICMP?

[stephenw10]

Mmm, I can't ping that either so it looks like it's blocking at least some sources.

[ahole4sure]

@stephenw10
@patient0
@SteveITS

THANKS for trying to help.
My head is spinning because I have too many isssues., lol
(they may be partly realted , maybe not completely)

The first thing that I have done more investigation is that there is for sure an issue with my ATT wireless modem
The way ATT has you to accomplish the static IP - is to give you a particular APN - and if you enter that data APN in the modem and it matches your SIM card it will give you access to your static IP address.
So to test things - (when I remove it from the pfsense situation and connect it directly to my Mac) and then enter or use the specific APN - then I get the weird issues where none of my normally hosted servers can be reached (if by web browser it just has no response - just locks page ) -- XXX.mydomain.com (DNS by cloudfare and access to my network only on 443 with haproxy - no open ports)

If I remove the specific APN and just use the normal Broadband APN then I can access my sites just fine.
So I concluded that my main issue is with ATT

What I am trying to accomplish is to set up Site to Site Wireguard VPN's so that the work is mostly done before I travel 5hrs to deploy for my daughter.
With the ATT modem set in the normal mode I can put it in IP Passthrough mode - but it is acting weird (I guess CGNAT ?) in that IP that the WAN is getting by DHCP on the status page reports as 10.168.25.44 - , but if I do what is my ip from a web browser and if I set up DDNS on pfsense - it reports IP as - 166.199.150.73
I have VPN pass through checked on my wireless modem - but I assume that is just not going to work for trying to setup a Site to Site -- I never a get a handshake from EITHER side

Is there a workaround to test and setup a Site to Site when you only have one internet source (or the second one is a cellular modem that I assume wont work becasue of CGNAT) ??

[ahole4sure]

@patient0
@stephenw10
@SteveITS

I was initally able to do my modified Site to Site using Tailscale and things were accesible -- but that isn't as reliable or fast as setting up a Site to Site Wireguard

With that said I have 2 site to sites setup with 2 of our locations -- there ahouldn't be a problem with creating a third as long as I seperate all subnets -- correct?

[SteveITS]

Yeah CGNAT breaks the inbound. If the other end has a public IP you can connect out to that, though. Do you get IPv6?

You could use a third device as a VPN server and connect both ends to that.

Somewhere recently I thought I saw something saying customers could not use 10.0.0.0/8 internally, and I think it was from AT&T. 20+ years ago we had a T1 ISP that used that for their network but still routed public IPs to the customer.

[ahole4sure]

@SteveITS

So in my existing I have site to site from A to B

Could I set up my new test to connect to site B (server) and have access from site A to the new test ???