[solved] Question regarding MIM: Controller not having a static *public* IP possible?

[Bob.Dig]

@stephenw10 Status is still unknown. But the second part is interesting. So when my public IP is finally changing, then I can not test this anymore because the registration data is not changing?

[stephenw10]

With status unknown like that the client has never connected back to the controller. So the traffic is probably blocked somewhere because it looks like it's trying to connect.

Currently the client connects back to the 'server' using only the IP address(es) passed to it in the registration data. So if the server side changes IP the connection will fail. Thus only static IPs are really supported for that side. But FQDN support is coming.

[Bob.Dig]

@stephenw10 But it is not blocked, you can see that in the last picture I posted. It can talk to the MIM Port on my (private) WAN-IP at home, just saying.

[Bob.Dig]

@stephenw10 Maybe it all is related to this?

Is this a timeout message? That VPS is horrible slow. Or there is an activation problem?

Log on Client

Spoiler

Nov 8 20:30:52 pfnet-controller 71929 797876 [/var/run/pfnet-controller.sock] GET /api/device/controller (DONE 129.606ms) OK: success
Nov 8 20:30:52 pfnet-controller 71929 797876 [/var/run/pfnet-controller.sock] GET /api/device/controller
Nov 8 20:30:48 pfnet-controller 71929 INFO Applying changes into config.xml
Nov 8 20:30:48 pfnet-controller 71929 711767 [/var/run/pfnet-controller.sock] DELETE /api/config/system/captiveportalbackup (DONE 5.991ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 698279 [/var/run/pfnet-controller.sock] DELETE /api/config/system/logsbackup (DONE 0.958ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 666789 [/var/run/pfnet-controller.sock] DELETE /api/config/system/dhcpbackup (DONE 1.080ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 657424 [/var/run/pfnet-controller.sock] DELETE /api/config/system/rrdbackup (DONE 1.353ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 401318 [/var/run/pfnet-controller.sock] DELETE /api/config/system/use_mfs_tmpvar (DONE 0.999ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 344427 [/var/run/pfnet-controller.sock] DELETE /api/config/system/dpinger_dont_add_static_routes (DONE 1.147ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 129633 [/var/run/pfnet-controller.sock] DELETE /api/config/system/keep_failover_states (DONE 1.041ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 112933 [/var/run/pfnet-controller.sock] DELETE /api/config/system/remove_failover_states_default (DONE 1.065ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 96084 [/var/run/pfnet-controller.sock] DELETE /api/config/system/schedule_states (DONE 1.204ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:48 pfnet-controller 71929 36961 [/var/run/pfnet-controller.sock] DELETE /api/config/system/pti_disabled (DONE 1.457ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 936446 [/var/run/pfnet-controller.sock] DELETE /api/config/system/thermal_hardware (DONE 1.333ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 926102 [/var/run/pfnet-controller.sock] DELETE /api/config/system/ipsec_mb (DONE 1.064ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 914451 [/var/run/pfnet-controller.sock] DELETE /api/config/system/crypto_hardware (DONE 1.378ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 566861 [/var/run/pfnet-controller.sock] DELETE /api/config/system/watchdogd_enable (DONE 1.040ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 460392 [/var/run/pfnet-controller.sock] DELETE /api/config/system/powerd_enable (DONE 1.268ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 451874 [/var/run/pfnet-controller.sock] DELETE /api/config/system/block_external_services (DONE 1.173ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 437248 [/var/run/pfnet-controller.sock] DELETE /api/config/system/do_not_send_uniqueid (DONE 1.890ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 416491 [/var/run/pfnet-controller.sock] DELETE /api/config/system/lb_use_sticky (DONE 1.015ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 407598 [/var/run/pfnet-controller.sock] DELETE /api/config/system/proxypass (DONE 1.380ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 402921 [/var/run/pfnet-controller.sock] DELETE /api/config/system/proxyuser (DONE 1.004ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 397833 [/var/run/pfnet-controller.sock] DELETE /api/config/system/proxyport (DONE 0.944ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 391868 [/var/run/pfnet-controller.sock] DELETE /api/config/system/proxyurl (DONE 1.263ms) ERROR: (*urlhandler.HandlerError) 400 not found
Nov 8 20:30:47 pfnet-controller 71929 328168 [/var/run/pfnet-controller.sock] DELETE /api/config/system/harddiskstandby (DONE 23.195ms) ERROR: (*urlhandler.HandlerError) 400 not found

[stephenw10]

Hmm, so the client is connecting out to the controller at the correct public IP but it never arrives at the controller?

[Bob.Dig]

@stephenw10 It is working now! Most probably the VM needed more RAM than I thought. It had only 512 MB before. I will mark this solved.

[Bob.Dig]

Today it is not working anymore but also there was the first IP change on my side. Doing some port magic didn't helped. So if I am able to, I will host the controller on a second instance in the cloud (with a static IP).

[Bob.Dig]

Maybe some last thought about this in this thread. I now had the chance to use a third Plus-Installation, it is on a VPS (2) too.
It worked right away to register it to the controller at home, while the other VPS (1) still didn't worked.

I then removed all peers and toggled the controller on all instances and registered all peers to the controller on VPS 2. This worked out of the box. There is not much difference on those two VPS other than that VPS 1 is very slow and that VPN connections towards the pfSense at home differs.

One problem I noticed with that new setup (on VPS 2) is that VPS 1 is still trying to connect at every second towards Home over one WireGuard tunnel. And I can't stop it other than blocking it. I will open another thread about this.

[Bob.Dig]

OT but didn't deserve its own thread I think: On 24.11-RC I noticed that on an installation (with 1 Gig RAM) the pfnet-controller Service wasn't running. I did a reboot, saw the same thing again. Starting it manually works. That controller is the actual controller of two other instances.
Only thing I noticed:

Spoiler

No problem with my main-machine though. The service is running but it is not the controller of other instances.

[marcosm]

Are the logs any different if you increase the log level?

[Bob.Dig]

@marcosm Not really.

Nov 14 17:07:10 	kernel 		controltun0: link state changed to DOWN 

Nov 14 17:07:09 	pfnet-controller 	57413 	controltun0 packet from 185.*.*.*:*
Nov 14 17:07:09 	pfnet-controller 	57413 	Unknown peerId in received packet from peer 4ae627
Nov 14 17:07:09 	pfnet-controller 	57413 	NG System busy, will challenge peer
Nov 14 17:07:09 	pfnet-controller 	57413 	NG Handshake received from 185.*.*.*:*
Nov 14 17:07:09 	pfnet-controller 	57413 	controltun0 packet from 185.*.*.*:*
Nov 14 17:07:09 	pfnet-controller 	57413 	Unknown peerId in received packet from peer 4ae627 

Nothing after that for the MIM log.

That IP is my very slow instance...

[marcosm]

Would you please reproduce the issue, generate diagnostic data while it's still stopped, and upload it here?

Also, are there any interesting logs from the client(s) when that happens?

[Bob.Dig]

@marcosm said in [solved] Question regarding MIM: Controller not having a static *public* IP possible?:

Would you please reproduce the issue

Done. This time even more services hadn't started... It is a VPS in the oracle cloud with just 1 GB RAM.

Edit2: Now it stopped again, I can't even get it working... While I see it now is supporting DDNS, I haven't used that and it is way less stable. Maybe I am supposed to make a new config(db) for it? No, latter doesn't change anything.

And on my main machine at home I have a WireGuard problem it seems, service isn't running although most of the tunnels do... And I see this, not the first time.

Spoiler

So I guess I am looking forward to an all new Install-Image to start over... At least CrystalDiskInfo isn't showing any problems.

Edit: A Reboot fixed this for my main machine, still, I have never seen this before on it. Also did a RAM-check, no problems found.

[stephenw10]

You should be able to install the RC directly with Net Installer if that's an option for you.

[Bob.Dig]

@stephenw10 said in [solved] Question regarding MIM: Controller not having a static *public* IP possible?:

You should be able to install the RC directly with Net Installer if that's an option for you.

I will do this with the final release. And reboot fixed all problems with my home installation for now. It might just be a problem with FreeBSD on ZFS on Hyper-V on NTFS.

[marcosm]

The failing SCSI commands are telling. I've experienced similar issues before with VMs running on NFS shares. IMO the issue there is storage. I wouldn't trust it even with reinstalls.

[Bob.Dig]

@marcosm Something seems odd to me with me the RC. I did an "offline" disk check in Windows, no problems found. After rebooting the host, the pfSense VM didn't boot fully, or to be more precise, not all services where loaded and a message said, that boot verification hasn't completed. And because it didn't, I disabled MIM and rebooted the Host again. This time everything went well and no problems at all.
So this behavior I only have seen with the RC, now at home and before in the oracle cloud... I will disable MIM on all machines and have a look if these problems are related to it...

@marcosm said in [solved] Question regarding MIM: Controller not having a static *public* IP possible?:

I wouldn't trust it even with reinstalls.

I will let you know if you where right.

[marcosm]

If you're referring to the message:

Automatic boot verification is still running - wait a moment for boot to complete.

You'll need to wait for the boot process to finish. The GUI becomes available before all services are ready.

[Bob.Dig]

@marcosm said in [solved] Question regarding MIM: Controller not having a static *public* IP possible?:

You'll need to wait for the boot process to finish. The GUI becomes available before all services are ready.

Yeah, I did but it didn't.

[stephenw10]

Does it show 'bootup complete' at the console? If not how far does it get?