Resolved: T-Mobile CellSpot connectivity issues
-
they said there is no response from the CellSpot when its behind the router as port 4500 is blocked.
Geniuses. Of course it's blocked to unsolicited inbound traffic. That's why it makes OUTBOUND connections on 4500 (ESP NAT-T).
I would let it sit. It looks like it has what it needs. Patience.
-
It has been sitting on it since last 24 hours :D
New connections are like this for a while.
VOIP udp 10.2.1.16:4500 -> 208.54.90.23:4500 NO_TRAFFIC:SINGLE 1 / 0 29 B / 0 B
-
Like Derelict said- get rid of the port forwards. I generally build a firewall rule on the WAN that shows source (AT&T network) and destination (range where we have all the cellspots) but its not necessary. I found it helps in the rare instance where the firewall was dropping the cellspot states for whatever reason. But no port forwards!
Is your cellspot assigned an address in your static DHCP table? If so change its address to something else and let it try again. Or if its not- assign it an address to grab that is not what it is now. See if that helps.
-
This is from one I have access to. It's on 2.3.1_5. The port 500 state is long gone.
LAN udp 208.54.66.205:4500 <- 172.24.128.198:4500 MULTIPLE:MULTIPLE 5.199536 M / 6.616718 M 1.00 GiB / 963.63 MiB
WAN udp OUTSIDE_IP:18887 (172.24.128.198:4500) -> 208.54.66.205:4500 MULTIPLE:MULTIPLE 5.199536 M / 6.616718 M 1.00 GiB / 963.63 MiBShe says it's working fine. It took a LONG time to finally sync up so if you have good, two-way traffic on UDP 4500 (indicated by MULTIPLE:MULTIPLE and increasing counters in both directions) I would let it do its thing for a while.
That's the issue. The counters do not increase. Some small bytes and then the connection is reset by the CellSpot as it tries a new server. I used to see the counters increase with good traffic on the older versions but haven't see them since moving to v.2.3.2
-
Like Derelict said- get rid of the port forwards. I generally build a firewall rule on the WAN that shows source (AT&T network) and destination (range where we have all the cellspots) but its not necessary. I found it helps in the rare instance where the firewall was dropping the cellspot states for whatever reason. But no port forwards!
Is your cellspot assigned an address in your static DHCP table? If so change its address to something else and let it try again. Or if its not- assign it an address to grab that is not what it is now. See if that helps.
Did all this many times since last week :)
Its on a fresh build with default settings. No changes and no NAT port forwards. I usually assign an IP address and have even moved it to different networks from VoIP network to LAN and let it acquire its own IP address. Even created an alias with T-Mobile network and gave it top preference on the rules. Saw the initial small bytes of communications and then nothing. None of this helped.
-
Dude. It's not the firewall unless you broke it trying to "fix" it. Maybe undo everything you have done and restart the cell spot. when I was at her house I got tired of waiting but the states looked good so I left. She called me a few hours later and said it was up.
-
Did you pay your bill?
-
Does it have a good GPS lock? My Verizon unit will not attempt a connection unless the GPS is working.
-
We had to be sure the GPS antenna was in a window too.
-
Dude. It's not the firewall unless you broke it trying to "fix" it. Maybe undo everything you have done and restart the cell spot. when I was at her house I got tired of waiting but the states looked good so I left. She called me a few hours later and said it was up.
As I said, it's on a fresh install with default settings. I did reset the CellSpot and when it worked I did see a large download on the CellSpot so I new it was doing it's thing. But that was an increase in counters and traffic graphs showed the data as well. Right now it hasn't moved more than 1.24MB of data since last 4 hours. Did multiple resets (rest to default and not just power reset) on the CellSpot. T-Mobile sent me a replacement which is behaving the exact same manner. Runs fine directly behind the modem but fails behind the router.
I have been suspecting DNS Resolver but it hasn't given issues to all my other network. May be T-Mobile is using some weird IP address that the DNS is not able to resolve even with DNS forwarding option. Just shooting in the dark as I am running out of options and its getting frustrating as I barely have any cell signal inside the house and this CellSpot is used by 4 lines in the family.
-
Does it have a good GPS lock? My Verizon unit will not attempt a connection unless the GPS is working.
Yeah its sitting right next to the window with the GPS connected. The power light (and that's the only light) keeps blinking and it doesn't even go the next step of acquiring a GPS signal. When behind the modem, I can see the network cable lights blink rapidly and it still acquires a GPS signal though after a good 15-20 mins) down in the basement. All lights turn solid green with 4G LTE blinking on data. But when behind the router just the power light blinks and blinks and blinks.. :D
-
Current states.. You can see its trying to connect to a different server. Also look at the Bytes.
-
Made some progress, I think.
I let it sit behind the modem to update itself. Everything looked good. Then I moved it behind the router with the updated software. This time the power light (green) and the Internet (orange) light blinks with a steady GPS light. Orange meaning an issue.
I suspect it's not able to talk to the outside world due to DNS issue. How do I troubleshoot the DNS resolver? Weird is that the cellphones use the same network on WiFi and don't have any issues. They all point to 10.2.1.1.
-
It is locating its destination address as shown by the fact it is actually connected to the server by what is shown in your state table. If you had DNS problems it wouldn't be able to locate the servers.
My guess is that something is screwing up the VPN packets somehow.
Your connected using a wired connection? Un-managed switch?
-
Yes wired connection. Managed switch as I have several VLANs. But no changes on the switch since last 4 years. Same VLAN used by 4 cellphones and 1 VoIP sip hone.
-
Looks like good two-way traffic to me.
-
Orange Internet light means it does not have an internet connection or unable to talk to the internet.
-
Orange Internet light means it does not have an internet connection or unable to talk to the internet.
They really need to re-write their guide.
Is the light flashing or solid?
-
Have you tried a different Ethernet cable? Solid light would seem to indicate no connection to your switch..
-
Orange Internet light means it does not have an internet connection or unable to talk to the internet.
They really need to re-write their guide.
Is the light flashing or solid?
Internet light is Flashing orange. When it has good internet connection its solid green.
If I turn off the cellspot for some time and then turn it back on the GPS light turns steady green as it was previously configured and acquires a signal. So it's blinking green power light and orange Internet light and steady green GPS light.
If I do a hard reset to factory defaults, just the green power light blinks as its trying to communicate to T-Mobile for a software download.
-
Have you tried a different Ethernet cable? Solid light would seem to indicate no connection to your switch..
Yeah ..lol. I can see it acquires an IP address in the DHCP leases table. I can ping it. Moved to LAN subnet and it works the same way. Same cable used to connect directly to the modem and it works.
-
https://support.t-mobile.com/docs/DOC-24271 I was just reading this doc..
Your cellspot is making a VPN connection to the Tmobile servers as proven by your state table.
Your cellspot is not making an connection inside the VPN as shown by your blinking orange light.
Something is messing up the VPN packets. Can you try connecting the cellspot direct to your pfsense box without the switch? Have an extra interface?
Just trying to rule things out.
-
Yep, just tried it. Plugged in the VoIP interface straight… no go :(
-
And to verify- No static DHCP setting for it is there?
Your not blocking DNS in any way? These devices unless you see different as it starts, tend to use their own DNS so your DNS should not matter
-
This seems like a 100% problem for t-mobile to solve. Too bad their support is so unhelpful and ignorant.
-
And to verify- No static DHCP setting for it is there?
Your not blocking DNS in any way? These devices unless you see different as it starts, tend to use their own DNS so your DNS should not matter
Wide open same as default LAN.
IPv4 * VOIP net * * * * none Default allow VoIP to any rule
-
This seems like a 100% problem for t-mobile to solve. Too bad their support is so unhelpful and ignorant.
Sounds like the device is not set for NAT operation.
Many people here including myself (for my office and two other customers) have these femtocell devices up and running behind our pfsense boxes with no issue. If this was an issue with pfSense there would be tons of posts with complaints of the same. But it just works for me and seemingly everyone else.
To test my NAT therory.. do you have a SOHO router hanging around? Your not double NATted are you?
-
There are generally zero configurable options on these things.
-
There are generally zero configurable options on these things.
Yea- has to be done by the carrier.
-
No double NAT. Just different subnets for LAN, VoIP and Video. I used a simple Asus SOHO router and it connected in matter of mins. Right now I have plugged it in directly behind the modem and that works as well. Just the moment I move it behind pfSense it hits a block. Firewall logs (when logging enabled for VoIP subnet rules) show a green check for the CellSpot IP.. means it passed through.
What I have not tried yet is force opening port 53 for the CellSpot even though the VoIP subnet (like the LAN) is wide open. I have even tried keeping on default LAN on a fresh plain install.. same issue.
What baffles me is that, nothing changed on my network except moving to v2.3.2. I may have to go back to v2.2 if this issue is not resolved.
-
What I have not tried yet is force opening port 53 for the CellSpot even though the VoIP subnet (like the LAN) is wide open. I have even tried keeping on default LAN on a fresh plain install.. same issue.
Your interface the cellspot is on needs full access to the internet. Basically it needs to be able to surf to anything it wants to. Rule= Voipnet any port any destination any port. Block nothing if you have not already. You really need nothing on WAN for it at all.
Other thing- Go to System/Advanced/Firewall&NAT TFTP Proxy. Enable that for your VOIP interface.
Another thing you could try is to make your devices port 4500 static in the outbound NAT page.
Im off to work in the field today so good luck!
-
Tried everything except for the TFTP Proxy. Will see if it does anything.
My VoIP Rule is like LAN.. [ IPv4 * VOIP net * * * * none Default allow VoIP to any rule ]
No change in data counters with port forwards for 4500, 500 and 123 directing to the CellSpot.
-
As I said before. Those port forwards might be screwing things up. They are 1000% not required for outbound connections.
I would get rid of all of that, get good two-way traffic on the IPsec session (500 and 4500 - whether it keeps increasing or not - look at all states on all interfaces filtering on whatever public IP you're connecting to) and call T-mobile and see whether they can see your unit connecting.
-
As I said before. Those port forwards might be screwing things up. They are 1000% not required for outbound connections.
I would get rid of all of that, get good two-way traffic on the IPsec session (500 and 4500 - whether it keeps increasing or not - look at all states on all interfaces filtering on whatever public IP you're connecting to) and call T-mobile and see whether they can see your unit connecting.
What I meant was port forwards didn't help. The system is on the default build state with no user changes.
Had called T-Mobile, but they said the CellSpot is not visible once behind the router and no connection requests are seen. Data counters are not showing any improvements. The CellSpot leaves the old connections and then tries with a new server every few mins.
-
If you have states to them and two-way traffic to them and no connection requests are seen by them, not sure what to tell you really.
You could packet capture but all you will be able to see is probably some DNS then the outside of an IKE/IPsec tunnel. You might do it and see if things seem to break down when packet sizes increase past about 1400 or something.
-
Tell us more about the hardware you are running pfSense on..
Especially the interface types.
-
What I have not tried yet is force opening port 53 for the CellSpot even though the VoIP subnet (like the LAN) is wide open. I have even tried keeping on default LAN on a fresh plain install.. same issue.
Your interface the cellspot is on needs full access to the internet. Basically it needs to be able to surf to anything it wants to. Rule= Voipnet any port any destination any port. Block nothing if you have not already. You really need nothing on WAN for it at all.
Other thing- Go to System/Advanced/Firewall&NAT TFTP Proxy. Enable that for your VOIP interface.
Another thing you could try is to make your devices port 4500 static in the outbound NAT page.
Im off to work in the field today so good luck!
FINALLY !!!! I can see the counters increasing steadily !!
It wasn't the TFTP Proxy.. but the MFS (Maximum Frame Size) on the network. Not sure why this happened now in pfSense v2.3.2 but I have the network config on 9216 (9000K) since 2012 with zero issues and the CellSpot worked just fine all these months. I installed an older v2.2 copy of pfSense last night on a backup server and the CellSpot connected fine once I restored the old config on it.
Definitely a frame size issue on v2.3.2. I confirmed this twice by changing the MFS to 9K and see the CellSpot disconnect. Tried putting a 9000 MTU on the interface but it didn't help.
-
ALL devices on a jumbo frame segment have to support, and be configured for, the jumbo frame size. Not quite sure how you are doing that on the cell spot with no config knobs.
-
Yeah, the CellSpot has 100Mbps port and not gigabit.
Along with changing the MFS to 1518 I had to enable (uncheck) the default set on the interface : Hardware TCP Segmentation Offloading & Hardware Large Receive Offloading.
Since I have intel 4 port NIC, I thought enabling it wouldn't hurt.
-
Yeah, the CellSpot has 100Mbps port and not gigabit.
Along with changing the MFS TO 1518 I had to enable (uncheck) the default set on the interface : Hardware TCP Segmentation Offloading & Hardware Large Receive Offloading.
Since I have intel 4 port NIC, I thought enabling it wouldn't hurt.
Intel drivers are generally just fine with those two checked. Is it em, igb or ?? Are you sure it's a real intel card and not a far-east knockoff?