HA Proxy pass client IP for autoban

[pppd]

I setup HAProxy to proxy my IMAP server which in turn has autoban on. But this causes HAProxy to be banned if an unauthorized user attempts a brute force login which in turn bans all authorized logins.

Any ideas how can I pass the client IP to my IMAP server for autoban to work effectively?

[kiokoman]

backeend / Advanced Settings
Transparent ClientIP
Use Client-IP to connect to backend servers.
read the warning

[pppd]

I did select this option but all it did was prevent IMAP clients from connecting to the back-end. Could SSL on the back-end cause this?