Strange Routing Issue

patient0

@ahole4sure traceroute may or may not work (the devices on the way to the target may not support it). I'd say a ping is preferred for a quick test.

What are you referring to by '... able to reach certain websites'? Do some open when you access them in the web browser? And others not, and if not what error do you get?

I have a static IP (for my ATT modem)

How did you have pfSense configured, static IP and gateway (information you got from ATT?)? What DNS server have you set to be used?

And I'd think you have a good chance searching the forum for ATT, maybe someone else had a similar issue with this ISP.

SteveITS

@ahole4sure This is a cellular connection?

Does the firewall on your remote endpoint allow ICMP?

stephenw10

Mmm, I can't ping that either so it looks like it's blocking at least some sources.

ahole4sure

@stephenw10
@patient0
@SteveITS

THANKS for trying to help.
My head is spinning because I have too many isssues., lol
(they may be partly realted , maybe not completely)

The first thing that I have done more investigation is that there is for sure an issue with my ATT wireless modem
The way ATT has you to accomplish the static IP - is to give you a particular APN - and if you enter that data APN in the modem and it matches your SIM card it will give you access to your static IP address.
So to test things - (when I remove it from the pfsense situation and connect it directly to my Mac) and then enter or use the specific APN - then I get the weird issues where none of my normally hosted servers can be reached (if by web browser it just has no response - just locks page ) -- XXX.mydomain.com (DNS by cloudfare and access to my network only on 443 with haproxy - no open ports)

If I remove the specific APN and just use the normal Broadband APN then I can access my sites just fine.
So I concluded that my main issue is with ATT

What I am trying to accomplish is to set up Site to Site Wireguard VPN's so that the work is mostly done before I travel 5hrs to deploy for my daughter.
With the ATT modem set in the normal mode I can put it in IP Passthrough mode - but it is acting weird (I guess CGNAT ?) in that IP that the WAN is getting by DHCP on the status page reports as 10.168.25.44 - , but if I do what is my ip from a web browser and if I set up DDNS on pfsense - it reports IP as - 166.199.150.73
I have VPN pass through checked on my wireless modem - but I assume that is just not going to work for trying to setup a Site to Site -- I never a get a handshake from EITHER side

Is there a workaround to test and setup a Site to Site when you only have one internet source (or the second one is a cellular modem that I assume wont work becasue of CGNAT) ??

ahole4sure

@patient0
@stephenw10
@SteveITS

I was initally able to do my modified Site to Site using Tailscale and things were accesible -- but that isn't as reliable or fast as setting up a Site to Site Wireguard

With that said I have 2 site to sites setup with 2 of our locations -- there ahouldn't be a problem with creating a third as long as I seperate all subnets -- correct?

SteveITS

Yeah CGNAT breaks the inbound. If the other end has a public IP you can connect out to that, though. Do you get IPv6?

You could use a third device as a VPN server and connect both ends to that.

Somewhere recently I thought I saw something saying customers could not use 10.0.0.0/8 internally, and I think it was from AT&T. 20+ years ago we had a T1 ISP that used that for their network but still routed public IPs to the customer.

ahole4sure

@SteveITS

So in my existing I have site to site from A to B

Could I set up my new test to connect to site B (server) and have access from site A to the new test ???

stephenw10

Unclear what you're asking. You should be able to establish the tunnel from the cell modem side as long as the other end has a known public IP it can listen on. Once the tunnel is up you can access things across it either way.

ahole4sure

@stephenw10
OK. thank you - I know stupid question - but why do a site to site in the first place then if all can be accomplished the other way?

stephenw10

What other way do you mean? Using Tailscale? No reason if that works for you. As you said it may be faster with a direct WG tunnel. But you may not need that additional bandwidth.

ahole4sure

@stephenw10
No I meant that I didn't know I could establish a peer to site with the peer being behind CGNAT (in pfsense). -- is that what you are saying that I should be able to do that ?
But does that allow access from the non CGNAT site into the network of the CGNAT site ??

stephenw10

It's still site to site it's just that the tunnel can only be established from the side behind CGNAT. But, yes, once the tunnel is established connections can be opened across it both ways.