PfSense FreeBSD on OVH Public Instance - no internet connection

maar · Jul 23, 2021, 9:07 AM

I have no internet connectivity in freshly installed PfSense system.
Steps:

I've created OVH instance with Ubuntu and I mounted mfsbsd image.
After booting to mfsbsd there was also internet problem. Couldn't download anything or ping at all.
I copied PfSense image through SSH using local network.
I installed PfSense with UFS BIOS option.
Using OVH console I can see that PfSense booted properly after restart. I configured also reverse proxy on nginx using other server in the same local network (http://192.168.10.22/ --> https://pfsense.domain.com/). After that I can reach GUI through pfsense.domain.com URL, but there's an error after logging in and I can't do anything in UI:

An HTTP_REFERER was detected other than what is defined in System >
Advanced (https://pfsense.domain.com/). If not needed, this check can
be disabled in System > Advanced > Admin Access.

After logging in to PfSense server using console I noticed that there's also network connectivity problem.
Some outputs:

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: ifconfig
    vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
            ether xx:xx:xx:xx:xx:xx
            inet6 xxxx::xxxx:xxxx:xxxx:xxxx%vtnet0 prefixlen 64 scopeid 0x1
            inet 123.123.123.123 netmask 0xffffffff broadcast 123.123.123.123
            media: Ethernet 10Gbase-T <full-duplex>
            status: active
            nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
            ether xx:xx:xx:xx:xx:xx
            inet6 xxxx::xxxx:xxxx:xxxx:xxxx%vtnet1 prefixlen 64 scopeid 0x2
            inet 192.168.10.22 netmask 0xffffff00 broadcast 192.168.10.255
            media: Ethernet 10Gbase-T <full-duplex>
            status: active
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    enc0: flags=0<> metric 0 mtu 1536
            groups: enc
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
            options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
            inet 127.0.0.1 netmask 0xff000000
            groups: lo
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    pflog0: flags=100<PROMISC> metric 0 mtu 33160
            groups: pflog
    pfsync0: flags=0<> metric 0 mtu 1500
            groups: pfsync

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
123.123.123.123    link#1             UHS         lo0
123.123.123.123/32 link#1             U        vtnet0
127.0.0.1          link#4             UH          lo0
192.168.10.0/24    link#2             U        vtnet1
192.168.10.22      link#2             UHS         lo0
213.186.33.99      xx:xx:xx:xx:xx:xx  UHS      vtnet0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#4                        UH          lo0
fe80::%vtnet0/64                  link#1                        U        vtnet0
fe80::xxxx:xxxx:xxxx:xxxx%vtnet0  link#1                        UHS         lo0
fe80::%vtnet1/64                  link#2                        U        vtnet1
fe80::xxxx:xxxx:xxxx:xxxx%vtnet1  link#2                        UHS         lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: ping serverfault.com
ping: cannot resolve serverfault.com: Host name lookup failure

I checked /etc/defaults/rc.conf and options defaultrouter and gateway_enable were set to NO. I changed them to YES, but It didn't help after rebooting.

DaddyGo · Jul 24, 2021, 7:19 PM

@maar said in PfSense FreeBSD on OVH Public Instance - no internet connection:

I've created OVH instance with Ubuntu and I mounted mfsbsd image.

Hi,

You are brave and I appreciate it, okhayyyy.... here comes the black soup

Well, that's quite a mule solution, scratch your left ear with your right hand above your head.
Choose a provider that supports FreeBSD or pfS by default...
(like: f.e.: https://aws.amazon.com/marketplace/pp/prodview-gzywopzvznrr4)

Or rather OVH support question is this (mfsbsd image), btw. we just moved from them because of the fire.... good luck

+++edit:

we did not have much with them(?), which is now trashed....... only 35 TB NAS backup data
(25 years of complete music database)

+++edit2:

Ahhh, I am not disappointed
(I used to trust in something now I will never do it again)

maar · Jul 26, 2021, 10:56 AM

@daddygo Yeah, I know, but It's more complicated. I have about 10 database servers and backups in different locations. Also I have the AWS with Kubernetes and I need to connect the OVH network with the AWS VPC.

Anyway, I figured it out. I connected to temporary created OpenVPN tunnel in OVH, I opened the Pfsense WebUI using private IP address and I was able to disable HTTP_REFERER. It's not cool that I can't change it from CLI, but nevermind. Now, I can get there through DNS name under https. I added OVH Gateway IP address to WAN interface using the Pfsense WebUI and the connectivity started working. It's interesting that I was trying to do the same by adding routing from FreeBSD cli and it didn't want to work.

For now I've established the VPN connection using AWS Site-2-site VPC service and I'll see if it is stable cause I've already tried the same with Strongswan instead of PFSense and the connection was randomly disconnecting from time to time. I couldn't find any logs why this was happening.

Also, I've created the EC2 with PfSense image from AWS Marketplace and I was trying to establish connection using Wireguard from AWS-PFSense to OVH-PFSense without luck. :( If the AWS S2S service will fail with stability - probably I'll create another topic here about the Wireguard.

DaddyGo · Jul 26, 2021, 11:07 AM

@maar said in PfSense FreeBSD on OVH Public Instance - no internet connection:

Wireguard from AWS-PFSense to OVH-PFSense

Nice save

I suggest you try IPSec instead of WG

BTW:

We have moved from OVH to SSDNodes in Amsterdam and Dallas, this is more than enough for a DB server(s) full KVM, full root access you can do whatever you want...

the cheapest on the market, massive telecom power "w" Tier1 DC, 10Gig connectivity, ultra fast customer service + NVMe

we use these plans:

maar · Jul 28, 2021, 11:47 AM

@daddygo Yeah, but like I said - I had some problems with IPSec before using Strongswan - now I'm testing stability with PFSense.

When it comes to provider the price is nice, but you're probably paying those $106 one or three years and after that $2880, and that price isn't as attractive anymore. You can pay it or migrate to another provider again.

DaddyGo · Jul 28, 2021, 12:05 PM

@maar said in PfSense FreeBSD on OVH Public Instance - no internet connection:

but you're probably paying those $106 one or three years

That's what I thought at first, but it's not, if you keep the VPSs (permanently), it gives you a perpetual guarantee on the price.

Not to mention the perpetual free daily full snapshot backup.

It's unbelievable, but I've moved 12 VPSs and they have great customer service, they're nice, fast...

Yes in this world it is hard to believe that there is such a thing.

I can only attest to the real VPS performance, in a KVM environment, without any continuations problem, yeah so great.

BTW:
Note that, I am not their man :), never heard of them until 5 months ago, a colleague suggested them when they launched EU DCs.

+++edit:

stephenw10 · Jul 28, 2021, 4:29 PM

You can disable the http referer check from the console using the php shell. It's one the available playback scripts there.

You can't set the gateway or default route as you were because pfSense does not use the FreeBSD RC system. Anything you set there is ignored.

Steve