Unable to add PPA to Ubuntu server when connected to HAProxy.

strongthany · Apr 22, 2020, 5:20 PM

Hello! I'll start off with the network setup I have, as the title gives the overview of the question:

HAProxy with a Let's encrypt via the ACME plugin. HAProxy routes traffic to the host of question, though the server running into this issue isn't one of the publicly available ones. You can see more about this configuration on this post I had made previously about a different issue. Within the LAN I have an Ubuntu box I am intending to make into a Landscape server so I can manage my other server through a central location.

The issue at hand is when I add the ppa for the server I run into some issues. Below is the output when I try to update:

admin@landscape:~$ sudo apt update 
Ign:1 https://ppa.launchpad.net/landscape/19.10/ubuntu bionic InRelease
Err:2 https://ppa.launchpad.net/landscape/19.10/ubuntu bionic Release
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. The name in the certificate does not match the expected.  Could not handshake: Error in the certificate verification. [IP: 91.189.95.83 443]
Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:6 http://us.archive.ubuntu.com/ubuntu bionic-security InRelease
Reading package lists... Done                     
E: The repository 'http://ppa.launchpad.net/landscape/19.10/ubuntu bionic Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
admin@landscape:~$

When I try to curl the url it would be pulling from, I get the following(tried two urls)

admin@landscape:~$ curl http://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/Release
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
admin@landscape:~$ curl http://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/InRelease
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

I believe this would have to do with my reverse proxy, which as side note does have a redirect for incoming traffic to go from 80 to 443, but disabling that didn't fix this issue. Is there a way for me to whitelist a url or host(either on the wan or lan) that would allow that traffic to bypass the proxy? Let me know if I'm missing any important information. Thank you!

PiBa · Apr 22, 2020, 8:34 PM

@strongthany said in Unable to add PPA to Ubuntu server when connected to HAProxy.:

Err:2 https://ppa.launchpad.net/landscape/19.10/ubuntu bionic Release

Why are you using 'http' for the curl test while 'apt' is using 'https'?

What does curl https://ppa. ... result in? And if it complains about the cert try with a -k to ignore cert errors, and also add a -v to show what headers are exchanged ?

strongthany · Apr 22, 2020, 9:11 PM

I am using http as that's what it defaults to when you add the PPA. I have ran the following conditions you met and shared the outputs down below:

admin@landscape:~$ curl https://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/Release

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

admin@landscape:~$ curl https://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/InRelease

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

admin@landscape:~$ curl -k -v https://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/Release

*   Trying 91.189.95.83...
* TCP_NODELAY set
* Connected to ppa.launchpad.net (91.189.95.83) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=pfSense webConfigurator Self-Signed Certificate; CN=pfSense-5e4ae44202003
*  start date: Feb 17 19:06:42 2020 GMT
*  expire date: Aug  9 19:06:42 2025 GMT
*  issuer: O=pfSense webConfigurator Self-Signed Certificate; CN=pfSense-5e4ae44202003
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56120ac791e0)
> GET /landscape/19.10/ubuntu/dists/bionic/Release HTTP/2
> Host: ppa.launchpad.net
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404 
< server: nginx
< date: Wed, 22 Apr 2020 21:18:35 GMT
< content-type: text/html
< content-length: 146
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host ppa.launchpad.net left intact

admin@landscape:~$ curl -k -v https://ppa.launchpad.net/landscape/19.10/ubuntu/dists/bionic/InRelease

*   Trying 91.189.95.83...
* TCP_NODELAY set
* Connected to ppa.launchpad.net (91.189.95.83) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=pfSense webConfigurator Self-Signed Certificate; CN=pfSense-5e4ae44202003
*  start date: Feb 17 19:06:42 2020 GMT
*  expire date: Aug  9 19:06:42 2025 GMT
*  issuer: O=pfSense webConfigurator Self-Signed Certificate; CN=pfSense-5e4ae44202003
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555a3c5161e0)
> GET /landscape/19.10/ubuntu/dists/bionic/InRelease HTTP/2
> Host: ppa.launchpad.net
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404 
< server: nginx
< date: Wed, 22 Apr 2020 21:20:31 GMT
< content-type: text/html
< content-length: 146
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host ppa.launchpad.net left intact

I'm not entirely sure why I'm getting these errors, but I see that it says it's hitting the webconfigurator cert. I have pasted the ssl config for my two frontends below for my https frontend and the http redirect respectively

Does this information help give any direction to what I'm doing wrong? Thank you!

PiBa · Apr 22, 2020, 9:36 PM

@strongthany
I cannot tell whats wrong from this, perhaps its easier to copy the actual haproxy.cfg from bottom of settings tab.? more complete and easier to read (for me anyhow).

But if curl is getting a webgui certificate of pfSense, then its likely not connecting to haproxy. Can you move the pfSense-webgui to a different port.? i like usually configure :444 for my pfSense-webgui.. so if a browser or any other client is pointing to pfSense:443 i know for sure it will either connect to haproxy or not at all.. Perhaps check that no port-forward settings are redirecting the traffic..?

strongthany · Apr 23, 2020, 9:25 PM

I will goahead and try the webconfig port thing once I've got a bit more precautions in place in case I lock myself out In the meantime, Here's what I have for a raw config:

# Automaticaly generated, dont edit manually.
# Generated on: 2020-04-22 16:27
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend SharedFrontend-merged
	bind			My.Public.IP:443 name My.Public.IP:443   ssl crt-list /var/etc/haproxy/SharedFrontend.crt_list crt-ignore-err all  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			aclcrt_SharedFrontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.mydomain\.tld(:([0-9]){1,5})?$
	acl			ACL1	var(txn.txnhost) -m str -i chat.mydomain.tld
	acl			ACL2	var(txn.txnhost) -m str -i blog.mydomain.tld
	acl			ACL3	var(txn.txnhost) -m str -i cloud.mydomain.tld
	http-request set-var(txn.txnhost) hdr(host)
	use_backend chat.mydomain.tld_ipvANY  if  ACL1 
	use_backend blog.mydomain.tld_ipv4  if  ACL2 
	use_backend cloud.mydomain.tld_ipv4  if  ACL3 
	default_backend chat.mydomain.tld_ipvANY
	default_backend blog.mydomain.tld_ipv4
	default_backend cloud.mydomain.tld_ipv4

frontend http-to-https
	bind			My.Public.IP:80 name My.Public.IP:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	http-request redirect scheme https 

backend chat.mydomain.tld_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			chat.mydomain.tld 10.10.10.252:443 id 101 ssl check inter 1000  verify none 

backend blog.mydomain.tld_ipv4
	mode			http
	id			10102
	log			global
	option			log-health-checks
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	option			httpchk OPTIONS / HTTP/1.1\r\nHost:\ blog.mydomain.tld
	server			blog.mydomain.tld 10.10.10.251:443 id 10103 ssl check inter 1000  verify none 

backend cloud.mydomain.tld_ipv4
	mode			http
	id			10104
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	option			httpchk OPTIONS / HTTP/1.1\r\nHost:\ cloud.mydomain.tld
	server			cloud.mydomain.tld 10.10.10.250:443 id 10105 ssl check inter 1000  verify none

Does this lead you anywhere?

PiBa · Apr 23, 2020, 9:32 PM

@strongthany
So i see 3 default_backends.. seems a bit much, 1 default should be enough, or none might be even better.. then at least you wont end up on the wrong webserver, but just get a 503 back..

None of the acl's match the ppa.launchpad.net domain..? How does that relate to chat/blog/cloud ?

strongthany · Apr 24, 2020, 5:17 PM

@PiBa the 3 backends are for the individual servers. I don't have a backend for the landscape server, that one isn't publicly reachable because it doesn't need to be.

the ppa server the one I need to be able to get through the reverse proxy to the landscape server. How do I have that whitelisted so that request doesn't go through the http to https redirect?

PiBa · Apr 24, 2020, 5:17 PM

@strongthany
Sorry but im not following..

Your sending a request to haproxy for a server it doesn't have configured.?? I'm missing something in my (mental) picture here.. Perhaps you can draw a little picture (doesnt need to be nice), about what ip's all the mentioned servers and pfSense use and from where to where you want the connection to work.?

Perhaps you can try and disable the 'transparent-client-ip' feature on any 'involved' server.?. if one is.. it causes reply packets to not get routed for that server:port combination..

strongthany · Apr 24, 2020, 7:49 PM

I have a server called Landscape that lives in my LAN. As shown in the config I posted, there is not a back-end for Landscape on the HAProxy config. I want to install Landscape on the server called Landscape. I cannot do so because HAProxy is filtering traffic coming into my network. HAProxy is filtering traffic coming into my network by routing things over http to https. When I try to install Landscape on the server called Landscape it fails to do so. It fails to do so because HAProxy moves the http traffic to https. I want to tell HAProxy not to filter that traffic Landscape.

How do I achieve this? Which of these options would be a better course of action, and how would I preform the one you deem better:

Exclude the server called Landscape from having it's traffic filtered

or

have the address for where I install Landscape from excluded form being filtered

Does this make sense?

PiBa · Apr 24, 2020, 9:45 PM

@strongthany
Either i'm totally not understanding your setup or the road we are on is totally unrelated to the problems your experiencing..

It seems to me that you have a new 'Landscape server' on which you want to install some software with 'apt', from a repository on 'the internet'..
This should have nothing to do with haproxy.. As the launchpad servers are not going to connect to your public ip where haproxy is listening, and you do not have the landscape server or dns records configured to point to your own wan-ip/haproxy-ip..
As such traffic should have absolutlely nothing to do with haproxy and also not with a pfSense-webgui certificate..
So it seems to me like your either running a other forward-proxy like squid, which could be filtering traffic, or something like snort/pfBlocker assuming the issue lies in a package... haproxy normally is not involved in traffic going from the LAN to the-internet.

But now i think its most likely that there is a wrong port-forward rule in the pfSense nat rules. Can you check/disable those.? Haproxy is listening on the wan-ip already, so doesnt need any nat rules.. only a firewallrule to allow traffic from clients on the internet to the wan-ip.

strongthany · Apr 25, 2020, 9:37 PM

I have no NAT rules. After spending some time talking to some people on the Ubuntu IRC forum I'm not so sure it's even HAProxy anymore.

When I run wget -qSO /dev/null http://ppa.launchpad.net/wireguard/wireguard/ubuntu 2>&1 | less from a device connected to my network I always get the following:

  HTTP/1.1 301 Moved Permanently
  Server: nginx
  Date: Fri, 24 Apr 2020 22:13:04 GMT
  Content-Type: text/html
  Content-Length: 162
  Connection: keep-alive
  Location: https://ppa.launchpad.net/wireguard/wireguard/ubuntu
  X-Frame-Options: SAMEORIGIN

Running the same command outside the network gets a 200 code from an apache server. Someone noted that the reason I'm seeing nginx is because of this note. However I have no idea why I'm not able to get out and to be quite honest I'm losing my mind over this.

PiBa · Apr 25, 2020, 9:37 PM

@strongthany
If that 'note' applies to your situation that would mean you have configured a services/captive-portal.? Do you have that.?

strongthany · Apr 26, 2020, 7:26 PM

@PiBa No captive portal, only thing that would be intercepting traffic like that would be the reverse proxy which I've shown by disabling it that is not the issue. Would there be something else in the way that would be breaking the connection?

PiBa · Apr 27, 2020, 2:59 PM

@strongthany
Can you show the output of these commands (maybe replace your public ip, if it shows up..)?:

pfctl -sn
ipfw show

strongthany · Apr 27, 2020, 8:37 PM

I have ran the command(doing remote cli, pfctl -sn didn't work but ipfw show did) and got the following:

00010    487555     70366491 fwd 127.0.0.1 tcp from 10.10.10.251 443 to any in recv re0
00011    651701    255737457 fwd 127.0.0.1 tcp from 10.10.10.250 443 to any in recv re0
65535 254477642 172839414961 allow ip from any to any

Not sure what this means..

PiBa · Apr 28, 2020, 5:51 PM

@strongthany
Traffic from the webserver running at '10.10.10.251 443' is redirected to pfSense's localhost, no matter where the client-return-address was, but this should not affect outgoing https requests from such a server. As they would target google:443 for example, but come from that server on a random client-port.. And certainly shouldn't affect traffic from a other new server..

I'm running out of ideas what to check without some 'hands on' checking.. But i think we are active in different time-zones.. 'CET-evenings'..

strongthany · Apr 28, 2020, 8:51 PM

@PiBa said in Unable to add PPA to Ubuntu server when connected to HAProxy.:

I'm running out of ideas what to check without some 'hands on' checking.. But i think we are active in different time-zones.. 'CET-evenings'..

That's okay, I understand. I think I'm going to make a new question in a different topic that might have more folks who can help. I really appreciate you helping me out here.