According to this document, you need to open UDP 1024 - 65535 between WLC (source) and APs (dest).
And it's correct - I can see these sessions getting dropped on my firewall since I didn't open those ports.
I only opened CAPWAP 5246-5247 from AP to WLC, and it seems to be working just fine. I can see my APs on my WLC, and configuration changes work.
So my question is : what is the WLC trying to send to my APs with those new UDP sessions ? I cannot find any information anywhere, and cannot run a packet capture.