[Solved] Squid 3.5 Reverse Proxy and Exchange 2010 - can't send e-mail

victorlclopes

Hi there,

I recently came across a very specific issue when configuring pfSense 2.3.2-RELEASE-p1 with Squid 0.4.23_1 (3.5.19_1) as a reverse proxy for the Microsoft Exchange Server 2010 and some other web applications.

The post is a bit long, I wanna apologize. And want to take this opportunity to thank all pfSense developers and supporters.

After having the certificates, server address and all mappings in place, everything looked great. The webmail (OWA) was fine and all communications including autodiscover, RPC over HTTP were working OK.

But then when I decided to move this router to production, my users started to complain that they were unable to send. Outlook would hang during the "Sending message" and e-mails would be stuck in the Outbox.

It so happens that if the e-mail messages were really small (just test messages, for example), they would be sent normally, but if they were a little bit bigger or if they had attachments, Outlook would stay a while trying to send, showing the status "Sending message 1 of 1" with the progress bar and then it would just timeout with no error messages.

In this scenario you may see Outlook error logs such as "Sending done, Error code = 0x80040115".
Or event logs such as "Rpc call (EcDoConnectEx) on transport (ncacn_ip_tcp) to server failed with error code (6ba)", but generally you don't get much useful information at first.

Another behavior is that when you try to use the webmail (OWA) to send the same e-mails you notice that you can't attach files. If you try attaching a very small file (<10KB) it completes the process, but anything bigger the page will appear to be loading until it times out too.

The Exchange Server itself doesn't show anything about this on Windows's default event logs as well.

Outlook troubleshooting logs may look like this:

2016.11.07 17:16:42 Sending one message
2016.11.07 17:16:42 Progress: Sending message 'Teste' (size 73.65 KBytes)
2016.11.07 17:16:46 MAPI Status: (IN – ---/OUT fl act)
2016.11.07 17:19:05 MAPI Status: (IN -- ---/OUT fl ---)
2016.11.07 17:19:07 MAPI XP Call: SubmitMessage, hr = 0x80040115
2016.11.07 17:19:07 FINISHED MAPI TASK
2016.11.07 17:19:07 suporte@domain.com.br: ReportStatus: RSF_COMPLETED, hr = 0x80040115
2016.11.07 17:19:07 suporte@domain.com.br: Synch operation completed
2016.11.07 17:19:07 Sending done, Error code = 0x80040115
2016.11.07 17:19:07 MAPI Status: (-- -- ---/--- fl ---)
2016.11.07 17:20:31 MAPI Status: (IN -- ---/OUT fl ---)
2016.11.07 17:20:31 MAPI Status: (IN -- ---/OUT -- ---)
2016.11.07 17:20:31 Finishing the Spooling Cycle, Error code = 0x80040115

Solution

After a lot of research, I found this thread (from Dec, 2015):

http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-reverse-proxy-infront-of-exchange-2010-td4674947.html

Where in response, dweimer (Dean E. Weimer) suggests that the problem is on the cipher algorithm and SSL version being used internally between squid and the Exchange server:

"On our Reverse proxy I ran into an issue uploading attachments to
Exchange back end, a while back, turned out the solution was to lock it
down so that the proxy only used ssl version 3 to connect to the
Exchange server. This however did recently break after a windows update
in Novemeber. Further investigation led to the particular cipher that
was in use. After discovering this I was able to use the same cipher
with TLSv1.0

Currently I am using TLSv1.0 with RC4-SHA cipher to talk to the Exchange
server.

cache_peer 10.20.10.161 parent 443 0 ssl no-query proxy-only no-digest
originserver
name=owa2010_parent sslcapath=/usr/local/share/certs
sslflags=DONT_VERIFY_PEER
login=PASSTHRU front-end-https=on connection-auth=on sslcipher=RC4-SHA
sslversion=4"

And that's true, although I don't know what the issue specifically is.
If you remove encryption on the LAN side, telling squid to offload and not use HTTPS to connect to your internal Exchange CAS, the problem is gone.
But then, not using HTTPS to hit the Exchange server is not ideal, both because the traffic would go unecrypted on you network and because OWA acts strangely.

Yes, RC4 is definitely not a good choice, but since this will be just internally (pfSense/squid at this current version will enforce only TLS1.0, 1.1 or 1.2 for external connections), is not a big deal. And if you have time to test other ciphers you may find a better one that works as well.

The fact is, if you tell squid to use TLS 1.0 with RC4 to connect internally with your Exchange, the issue will be solved.

OK, so the challenge now is that the pfSense GUI for Squid does not currently support input for such parameters like cipher and SSL version. You can manually edit the squid.conf file but it would be overwriten on every setting change.

Edit these two files located at /usr/local/pkg (you can use pfSense's "Diagnostics > Edit File" menu), to include the modifications highlighted:

squid_reverse_peer.xml

…

<adddeleteeditpagefields><columnitem><fielddescr>Status</fielddescr>
<fieldname>enable</fieldname></columnitem>
<columnitem><fielddescr>Alias</fielddescr>
<fieldname>name</fieldname></columnitem>
<columnitem><fielddescr>IP Address</fielddescr>
<fieldname>ip</fieldname></columnitem>
<columnitem><fielddescr>Port</fielddescr>
<fieldname>port</fieldname></columnitem>
<columnitem><fielddescr>Protocol</fielddescr>
<fieldname>Protocol</fieldname></columnitem>
<columnitem><fielddescr>Cipher</fielddescr>
<fieldname>cipher</fieldname></columnitem>
<columnitem><fielddescr>SSL-TLS Version</fielddescr>
<fieldname>sslversion</fieldname></columnitem>
<columnitem><fielddescr>Description</fielddescr>
<fieldname>description</fieldname></columnitem></adddeleteeditpagefields>

…

<field><fielddescr>Peer Protocol</fielddescr>
<fieldname>protocol</fieldname>
<description>Select protocol listening on this peer port.</description>
<type>select</type>
<options><option><name>HTTP</name> <value>HTTP</value></option>
<option><name>HTTPS</name> <value>HTTPS</value></option></options></field>
<field><fielddescr>Cipher</fielddescr>
<fieldname>cipher</fieldname>
<description>The cipher to use between squid and the internal host (Optional)</description>
<type>input</type>
<size>60</size></field>
<field><fielddescr>SSL-TLS Version</fielddescr>
<fieldname>sslversion</fieldname>
<description>The SSL version to use between squid and the internal host (Optional).</description>
<type>select</type>
<options><option><name>Default (auto)</name><value></value></option>
<option><name>2.0</name> <value>2</value></option>
<option><name>3.0</name> <value>3</value></option>
<option><name>4 (TLS1.0)</name> <value>4</value></option>
<option><name>5 (TLS1.1)</name> <value>5</value></option>
<option><name>6 (TLS1.2)</name> <value>6</value></option></options></field>
<field><fielddescr>Peer Description</fielddescr>
<fieldname>description</fieldname>
<description>Peer Description (Optional)</description>
<type>input</type>
<size>60</size></field>
…

squid_reverse.inc

…

$active_peers = array();
if (is_array($reverse_peers)) {
foreach ($reverse_peers as $rp) {
if ($rp['enable'] == "on" && $rp['name'] != "" && $rp['ip'] != "" && $rp['port'] != "") {
$conf_peer = "#{$rp['description']}\n";
$conf_peer .= "cache_peer {$rp['ip']} parent {$rp['port']} 0 proxy-only no-query no-digest originserver login=PASSTHRU connection-auth=on round-robin ";
if ($rp['protocol'] == 'HTTPS') {
$conf_peer .= "ssl {$sslflags_cache_peer} front-end-https=auto ";
if ($rp['cipher'] != "") {
$conf_peer .= "sslcipher={$rp['cipher']} ";
}
if ($rp['sslversion'] != "") {
$conf_peer .= "sslversion={$rp['sslversion']} ";
}
}
$conf_peer .= "name=rvp_{$rp['name']}\n\n";
…

Now you can go and change your "Web Server" and configure these values. Here is how mine look like (login to see):

And these are my mappings/URIs for this setup, if someone needs a sample:

^https://webmail.yourdomain.com/owa/?.$
^https://webmail.yourdomain.com/public/.$
^https://webmail.yourdomain.com/exchange/.$
^https://webmail.yourdomain.com/ecp/.$
^https://webmail.yourdomain.com/rpc/.$
^https://webmail.yourdomain.com/oab/.$
^https://webmail.yourdomain.com/ews/.$
^https://autodiscover.yourdomain.com/autodiscover/?.$
^https://webmail.yourdomain.com/Microsoft-Server-ActiveSync.*$
^https://webmail.yourdomain.com/default.aspx$
^https://webmail.yourdomain.com/?$

Last two are for redirecting to /OWA, you may not need it.

![01 - cipher and sslv.jpg](/public/imported_attachments/1/01 - cipher and sslv.jpg)
![01 - cipher and sslv.jpg_thumb](/public/imported_attachments/1/01 - cipher and sslv.jpg_thumb)
![02 - web servers list.jpg](/public/imported_attachments/1/02 - web servers list.jpg)
![02 - web servers list.jpg_thumb](/public/imported_attachments/1/02 - web servers list.jpg_thumb)

victorlclopes

I hope this helps somebody :)

May be someone has a better idea of a cipher to use… or I think this can be marked as closed or something.
Remember that any GUI customization will probably be lost on package upgrade.

tim_co

This has been very helpful. Thanks for the post.

With the caveat that I clearly have a pretty basic understanding of pfSense, Squid and Ciphers, I also currently have the flu and I’m home with a sick kid so I’m extra stupid.

I’m still having problems and would appreciate any assistance. My environment:

2.3.3-RELEASE-p1 (amd64)
0.4.36_1
Exchange 2007/SBS 2008

I implemented the noted changes and I can see the Cipher and SSL-TLS options under now.

After making all the changes, I'm still having problems sending attachments. I suspect it's related to the Cipher I've defined under Squid Revers Peer Mappings Web Servers.

Per best practices, I used the IIS Crypto 2.0 tool to disable all but the following protocols on the Exchange server:
https://www.nartac.com/Products/IISCrypto

Protocols: TLS 1.0
Ciphers: Triple DES 168, AES 128/128, AES 256/256
Hashes: MD5, SHA
Key Exchanges: Diffie-Hellman, PKCS, ECDH

SSL-TLS version is defined as “4 (TLS1.0)”. I've tried defining the Cipher as 3DES, AES, AES-SHA, RC4-SHA, AES256, etc. After making the changes, Squid restarts correctly but attachments still don't flow.

Can somebody help me understand what cipher I should define for my environment? More specifically, exactly what should I type into the Cipher field?

tim_co

I think I figured it out using the web site and info below. I defined the cipher as "RSA+AES256". Attachments are flowing again. To quote victorlclopes - I hope this helps someone! :)

http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit

The results of a NMAP ssl-enum-ciphers -p 443 and the cipher error message from Chrome.

*Chrome - The connection to this site uses an obsolete protocol (TLS 1.0), a strong key exchange (ECDHE_RSA with P-384), and an obsolete cipher (AES_256_CBC with HMAC-SHA1).

Here are the options I tried:

sha256RSA-sha256
sha256RSA
sha256RSA-SHA1
sha256-sha1
AES_256_CBC-HMAC-SHA1
AES_256_CBC
AES_256_CBC-SHA1
AES_256-SHA1
aes_256_cbc with hmac-sha1
TLS_ECDHE_RSA_WITH_AES_256_CBC-SHA384_P521
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
AES_256_CBC_SHA
EECDH+RSA+AES256

stevo1664

I had the same problem, this fixed it.

Thank you very much for sharing.

Steve