You might have a somewhat hidden or layered packet size problem, probably most prominent in the case of 5G+VPN.
If true, the performance impact observed might come from fragmentation happening somewhere along the path, and the system ressources wasted for the effort of fragmentation and reassembly.
Be sure to determine end-to-end PathMTU for the involved ...
I found the solution to this eventually, with some help.
The issue was that on the ASR side, the tunnel was in its default configuration which uses GRE, which the ASA doesn't support.
The solution was to change the tunnel mode to ipsec ipv4 like so:
tunnel mode ipsec ipv4
With this done, bidirectional traffic was possible over the tunnel.