loopback address being blocked?

johnpoz

And still not the commands I actually asked for.. Lets see the output of netstat or sockstat actually showing something bound to loopback.

Sockstat prob better because would actually show "what" is bound to that port.

Again - there is no firewall rule that could block pfsense from talking to itself.

You could have some race condition where something else is binding to 443 on loopback?? The gui will try and bind to everything.. So maybe you have something trying to bound to 443, before the gui can bind to it on loopback? So lets see sockstat " grep :443 showing what process if anything is bound to the loopback on that port.

ludejim

My bad. I just saw your screen shot and thought that’s what you were referring to. Here is a sockstat | grep :443

ludejim

@johnpoz Now that we know multiple things are bound to loopback, is there a command that could help me determine what those things are. I can see they are Nginx servers, but I don’t know what they are associated with.

I do have OpenVPN and HAProxy running. My OpenVPN instance is using the default port of 1194. HAProxy is bound to port 80 for a web server.

Unless those two items do something in the background I am unaware of, I’m not sure what the other items are that are binding to port 443 on the loopback.

johnpoz

nginx is the gui..

Those are the pids of the processes, 2 workers and the main one.. on mine for example

[2.4.5-RELEASE][admin@sg4860.local.lan]/root: ps -ax | grep 82772
82772  -  Is        0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
38899  1  S+        0:00.00 grep 82772

Again there is no firewall rule that would prevent access to itself.. Lets see the output of your curl with the -v so we can see exactly what is going on.

[2.4.5-RELEASE][admin@sg4860.local.lan]/root: curl -v https://127.0.0.1:8443
*   Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

Also sniff on the localhost while you try and do your test.. So you can actually see the connection attempt, and the source IP, etc..

So here example you see me doing that above test while doing a packet capture on localhost for my port 8443

For grins check your loopback rules

[2.4.5-RELEASE][admin@sg4860.local.lan]/root: pfctl -sr | grep loopback
pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
pass in on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback"
pass out on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback"

You do show your lo up and on 127.0.0.1? with an ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Also so we are sure.. Is this baremetal or is pfsense on some VM?

ludejim

curl command and response:
curlCommand

Packet capture during curl command captured nothing:
PacketCaptureDuringCurlCommand

loopback rules:
loopbackRules

ifconfig lo0 is up:
lo0up

Great question, my pfsense is being hosted on eSXI.

johnpoz

do you have a route for loopback?

example

[2.4.5-RELEASE][admin@sg4860.local.lan]/root: netstat -rn | grep 127
127.0.0.1          link#7             UH          lo0
[2.4.5-RELEASE][admin@sg4860.local.lan]/root:

There is mine..

That error in curl
Immediate connect fail for 127.0.0.1: Can't assign requested address

Can you ping localhost?

[2.4.5-RELEASE][admin@sg4860.local.lan]/root: ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.128 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.063 ms

ludejim

I can’t ping localhost:
Cantping

Netstat results:
Netstat

kiokoman

it's in the wrong interface
127.0.0.1 link#4 UH lo0

should be lo0, not vmx0
idk how you ended up with something like this

johnpoz

Yeah that is clearly Borked!!

ludejim

Borked it is! I can see how to assign all other interfaces. Anyway to reassign the loopback?

johnpoz

Never seen such a thing.. Some weirdness with interfaces being re-arranged when interfaces added in esxi maybe?

I don't know how you would re-assign that to be honest.

I would start over with new VM..

ludejim

That’s what I figured. I just backed up all my configurations and began setting up the new VM. It’ll have to wait for tonight because work from home and schooling from home. We need the internet for the day. I’ll let you guys know how it works out. Thanks for the help!

johnpoz

I would double check your lo0 assignment before you get to far into the config ;)

ludejim

Are you taking about on the new VM? If so, that was my going to be step one.

johnpoz

Yeah on the new vm ;)

Do you have any other VMs running on this host that you could look to see how the lo0 is assigned?

ludejim

I simply rebooted the pfsense VM and the loopback was reassigned to lo0. What a headache for apparently no reason!

johnpoz

I would of assumed you had done that already ;) When it wasn't working the first time?

ludejim

Unfortunately no. I rarely if ever need to restart that VM. It’s uptime was 172 days prior to my restart.