Starlink - No Internet when "Reject leases from" Configured

stephenw10

Hmm, maybe I've lost track here then. What's the problem with not rejecting that then?

I thought that was required to prevent it ending up with a bad lease that could not connect.

Ramosel

@stephenw10 That's OK, I get lost and I'm in it....

If the reject address is there, it never gets the 192.168.100.100 address... so it just sits. Never connects.

If it's removed, it gets the 192.168.100.100, drops it and then you go into this mode where you need to reboot, unplug, replug... then it finally gets an address on 100.91.143.xx/10 and works. Then it will reboot and reconnect just fine.

But if there is a prolonged outage, or I plug in the Starlink Native router (to keep it up to date), then you start this whole dance over again.

What I'm hearing from the lady running Starlink as a Primary WAN and a Verizon 5g Home wireless gateway as a secondary in a failover is that if Starlink Fails over to Verizon, it never comes back unless she does the same reboot/unplug dance I'm doing on the Starlink only system

Hope that clears it up...

When it's working, it works great!

Ramosel

@ramosel Just FYI, the outage on Friday that started the latest round of reconnect fun was legit. Starlink had an expired cert on one of their ground stations. They were offline for half an hour or more. That was the cause of the lost connection.

stephenw10

Hmm, so it's as though it must pull that 192.168.100.100 lease in order to reach the second stage but it fails to retry the new lease without manual intervention?

I wonder if the FreeBSD dhclient is not respecting the given lease time? Does it eventually connect when the 192.168.100.100 lease tries to renew?
Does it connect if you just manually renew it from the Status > Interfaces page?

Steve

Gertjan

@stephenw10 said in Starlink - No Internet when "Reject leases from" Configured:

so it's as though it must pull that 192.168.100.100 lease in order to reach the second stage

Added to that question : if a PC was hooked up directly to the disk, would it establish a working connection ?
I understand that , normally, the disk (the 'thing' outside) is connected to a router inside.
All network devices are connected to this router, directly, or indirectly with a switch.

That a 'disk/modem/etc' comes up with a RFC1918 LAN network first upon boot : that's a known behavior.
As soon as the disk establishes its uplink - and this time the word uplink isn't an understatement, it really links upwards - it will obtain a 'real' WAN IP.
At that moment, it - the disk ! - should take it's LAN interface down, and then UP, so the DHCP client on the WAN interface of the spacelink router - or pfSense being the router here - asks again for an IP. This time a routable IP is obtained.

For what I know :
I never actually saw a SpaceX Starlink kit.
I understand that many Starlink don't 'need' the Starlink supplied router, as it is just a router with (probably) wifi capabilities.
I'm pretty sure 'thousands' removed the (startlink router) router and use their own router instead.
The real question is : @Ramosel why do you have an issue ? What's different with your setup ?
Are Starlink devices region depended ? Is this because you have a type A setup and all the others are using a type B setup ?
Like A = home and B is a pro setup ?
Why would Starlink make (your) live harder ?

@ramosel said in Starlink - No Internet when "Reject leases from" Configured:

Correct. It gets a lease on 192.168.100.1 for a short time, probably no more than 10 seconds (no subnet delineation) drops it and gets one on 100.91.143.1/10.

During that time :
What happens when you look here during the entire sequence :
Status > System Logs > DHCP
Status > System Logs > System General

Personally, myself, I don't bother looking at the gui.
I open two SSH connections.
In the first : tail -f /var/log/dhcpd.log
In the second : tail -f /var/log/system.log

I would be watching for the 'general' system log because that's the place interface DOWN and UP will take place. Upon WAN interface event, a whole lot of lines will be logged, as many processes will get restarted if something happens with WAN. The one of interest is called "dhcpd".
WAN Interface events of interest will use the interface device name, like 're0' or "em0' or 'igc0' where 0 can also be 1 etc.

The logs will help you to see what happens when. YOU can create all the scenarios, the logs will tell you what happens.
"DHCP" (IPv4) is a very mature process, and 'normally' no user interaction is needed to make it work.

In your case, where only a RFC1918 type is available upon 'disk' boot (and as soon as the DHCP server build in the disk start up right after it brought up electrically its 'LAN' port), pfSense will notice that it's WAN port became active (having electric signals) and kicks of a dhcp client on the WAN interface, and starts requests (broadcasting) "DHCPDISCOVER".
the disk DHCP server will receive these, and it has been set up like : give an RFC1918 in the 192.168.100.x range (himself probably being 192.168.100.1).
At that moment, even when the disk DHCP server is running, the other part of the disk is still aligning, looking for satellites, and doing whatever is needed to make all the satellite connection. You already get a RFC1918 IP 192.168.100.x IP so you can use that to connect yourself to the 'disk' GUI (the disk build in web server it probably has one) to see internal data like satellites found, position, connection passwords etc etc.

Satellite finding and connecting takes time, that's easy to understand.

So, at this moment, the WAN IP of pfSense is set : 192.168.100.x and a gateway is known (192.168.100.1 - but check this : is this the case ??)
The gateway IP is important as pfSense, by default, will ping (the so called 'dpinger' process) this IP 192.168.100.1 to check it's connection. If no ping then it will take action - and that means : resetting the WAN interface, so the pfSense DHCP client will kick in again etc.

After some time, several minutes probably, the disk obtains a working satellite link.
At that moment :
It will shut down its own local 'disk build in' DHCP server on the LAN port.
It will bridge its LAN port to the disk-internal-uplink-port (using radio waves, satellites, probably some fiber back on the ground, the whole thing).
It will bring up its LAN port. All traffic coming into the disk LAN port will now be send to the disk == send to the satellite == send to the ISP called 'spacex'.

pfSense doesn't know 'sh*t' about all this, all it senses its WAN being brought down : ok, link down so it does all the WAN down housekeeping stuff like destroying existing routes and all other things routers like to do.
And then its WAN link come sup again.
The pfSense DHCP client is started and sends as usual a first DHCPISCOVER. This request will not get intercepted by the DHXCP server in the disk - as it doesn't exist anymore. It will get send to the disk => stallite => etc etc deep into the spacex network.
Somewhere in the ISP spacex network (on the satellite, at Elon's home ? Dono) a SpaceX DHCP server will receive the DHCP request from the pfSense DHCP client and answer it.
This time, this DHCP server will propose a 'real' routable WAN IP, probably a DNS or two, a gateway IP (important - what is it ?) and maybe more.

The gateway IP is again used by pfSense, as a dpinger process is started :

ps ax | grep 'dpinger'

so it can check the WAN (up) link - and take actions (== reset interface) if the connection goes bad.

Why do I tell you all this ? Two reasons :

As soon as you understand what you see (the logs), you can find a way out, aka : the solution.
Understanding DHCP means DHCP will not give you issues anymore (I can't prove this scientifically, but somehow I know it applies)

Ramosel

@gertjan said in Starlink - No Internet when "Reject leases from" Configured:

Thanks for ALL that, gertjan... I notice avatar is a chevron. Military background? I was a US Navy Pilot. I have a brain injury. I'm still all here but can only bring my focus to bear for about 20min at time, so I'll try to digest all you sent in pieces over the next day or so.

Added to that question : if a PC was hooked up directly to the disk, would it establish a working connection ?
I understand that , normally, the disk (the 'thing' outside) is connected to a router inside.
All network devices are connected to this router, directly, or indirectly with a switch.

For what I know :
I never actually saw a SpaceX Starlink kit.
I understand that many Starlink don't 'need' the Starlink supplied router, as it is just a router with (probably) wifi capabilities.
I'm pretty sure 'thousands' removed the (startlink router) router and use their own router instead.

Lets take this part first. I have not tried a PC direct, but will today. Interesting thought.

Because of my IT background (not on the network side of the house, but I've certainly worked with them) and a family FAA connection to SpaceX, I was brought in on the Starlink Beta early. I have the V1 dish (round), there is now a V2 (rectangle).

V1 consists of a Dish, a power brick+ (it's more than just a power brick) and the wireless router - The wireless router does have a single RJ45 jack, for local ethernet past the router. You are right in that in this V1 configuration the wireless router can be removed from the setup.

V2 consists of a Dish and a wireless router. This router acts as a termimus for the dish and a power cord. The V2 router cannot be removed from the system and natively has no ethernet port. That is added via a "dongle" of sorts that strips off a physical port from the dish connection prior to the router. It has to be ordered and paid for as an extra.

The V1 power brick is not just a power source, there is some networking capability in this device which has an AC power input and two POE ports. One (black) goes up to the dish, the other (white) goes to the wireless router. In this case, I use the white connector to go directly to the WAN port on by SG-4860. All the rest of my network is then under pfSense control, LAN, OPTLANs, wireless backhauls, switching, ipPhones, alarm, etc.

If you want to DM me an email address, I'll gladly shoot you pics of all this if that would help. Or you can see most of it here: https://youtu.be/Zsuf7pdl_1M

So while I am certainly not unique, this Starlink/pfSense setup being used in small numbers. Most V1 users just use the supplied router. Likewise, most V2 users do bemoan the lack of native ethernet but some are ordering the dongle and I do know of just a few running pfSense off the dongle, such as the young lady running the failover with a 5G service that will never switch back. I personally don't know anyone using a plastic box router behind Starlink V1 or V2. I know a few who have tried, but gave up. But I'm sure they are out there and working.

Though Starlink is being used in every locale... Starlink is designed for rural users. Most rural users just want working internet, never unplug their Starlink router and are happy with wireless everything.

I absolutely agree with your 2 reasons at the end. It's why I'm here.

stephenw10

Mmm, it appears that either the Starlink router is doing something special with the dhcp client or the pfSense dhclient behaves differently to other routers/hosts.

Connecting a host directly would be an interesting test there.

The fact that Starlink don't exclude 3rd party routers entirely implies their own router is not doing something special like, for example, AT&T do.

Gertjan

@ramosel

If you have a moment : https://forum.netgate.com/topic/179959/starlink-and-pfsense/4?_=1683716772366

My question is :
Does the 'disk' pulls down it's LAN for a brief moment, when it has finished creating a stable uplink/downlink ?

Can you read what I've said over there ?
Anything to add ?
Take note : I'm not a 'starlink' user, so everything is pure speculation on my side.

On one side : I can't imagine that starlink (the disk) uses something utterly none-RFC compliant solution.
On the other side : If I was an ISP, I would force users to equipment I give them, so support is way easier (= read : way cheaper )

dadams69

I have a developer that lives in the sticks in Wyoming. His only option was line of sight wifi or Starlink. The current setup is similar to this one. The v2 dish to pfsense. the problem with starlink is it uses cIP address in the DHCP pools to hand out to all the dishes and do all the routing on a virtual layer. Software defined Networking at its finest. cIP addresses are Carrier grade IP addresses which are non route-able publicly. So in order to get his pfsense box to show it was online was to set the monitor ip in the wan gateway to 8.8.4.4. So it looks like this-

we also have 3 VPN site to site clients running fulltime on this box. Mainly do to this user not being a network person at all, So I have to remotely manage it via these tunnels.

I did not see anyone else ask this, but did you reboot the pfsense after rebooting the dish? because if you didn't it will not work. Or if you did not enter the Starlink as the hostname in the dhcp options screen.

stephenw10

Hmm, so simply setting a different monitoring IP allowed it to work?

dadams69

@stephenw10 That with adding the hostname here.

dadams69

@stephenw10 Starlink is Bi Polar and does play well with others!