Ok, so i got it sorted.
TL;DR : A captive portal was enable and behave strangely on the SSID it was enabled on, wasn't showing when connecting, and cause the NG-2100 to drop packets.
I first tried with a new NG-2100 in a lab, same configuration, and everything worked perfectly, so I suspected an issue with the LAN in production.
I tried multiple thing, got some packet captures, and saw that UDP was working fine, everything in local was working fine, and TCP SYN packets were going out, but SA were block going back
I tried to make an untagged port on the L2 switch with VLAN tag 2U to try without WLAN, and everything worked fine.
I then tried to switch the VLAN on the SSID that was not working to go from 2T as 1U and figured out that there was a captive portal enabled on this SSID, probably from an old config that wasn't causing issue with old router. For whatever reason, the captive portal wasn't showing on this SSID when VLAN tagging was enabled, but was acting weird with TCP requests I guess.
Removed captive portal, everything worked fine.