OK Here's what's happening:
I created a FW rule to allow traffic from wifi AP to LAN on port 1812. That definitely was missing! Good catch ;) At least now I see something in the system logs…
After that I followed your instructions, that is, created a new cert under "System > Cert. manager > Certificates" (named pfsense-RADIUS) then exported the root CA (under System > Cert. manager > CAs) to my Android phone.
Then I modified the wifi connection on the phone and assigned the newly imported root CA to it. Then I modified RADIUS's config and selected the certificate "pfSense-RADIUS" (under EAP tab > SSL Server Certificate).
Finally I tried connecting to the Wifi network, but the phone still says "Authentication problem" and pfsense logs show:
Oct 16 18:54:11 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:54:11 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:54:11 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:54:11 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:54:11 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:53:31 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:53:31 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:53:31 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:53:31 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:53:31 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:53:09 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:53:09 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:53:09 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:53:09 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:53:09 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:51:34 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:51:34 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:51:34 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:51:34 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:51:34 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:50:52 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:50:52 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:50:52 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:50:52 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:50:52 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:50:29 radiusd 22391 Login incorrect (TLS Alert read:fatal:unsupported certificate): [wifiuser] (from client unifi-ap-ac-lite port 0 cli XX-XX-XX-XX-XX-XX)
Oct 16 18:50:29 radiusd 22391 SSL: SSL_read failed inside of TLS (-1), TLS session fails.
Oct 16 18:50:29 radiusd 22391 rlm_eap: SSL error error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
Oct 16 18:50:29 radiusd 22391 TLS_accept: failed in SSLv3 read client certificate A
Oct 16 18:50:29 radiusd 22391 TLS Alert read:fatal:unsupported certificate
Oct 16 18:50:16 radiusd 22391 Ready to process requests.
Oct 16 18:50:16 radiusd 22010 Loaded virtual server <default></default>
Thats weird no?