You might have a somewhat hidden or layered packet size problem, probably most prominent in the case of 5G+VPN.
If true, the performance impact observed might come from fragmentation happening somewhere along the path, and the system ressources wasted for the effort of fragmentation and reassembly.
Be sure to determine end-to-end PathMTU for the involved participants (end systems and intermediate neworking gear) in all 4 cases for both directions, up- and downstream. Path MTU is an unidirectional thing, might be different in each direction.
If this is a site-to-site VPN, it might be worthwile to first check/measure/verify PathMTU for the external connectiviy separately (i.e. from VPN endpoint to VPN endpoint, across the "public" network).
Then, be sure to understand how much packet overhead your chosen variety of VPN will generate. Up to 100bytes per packet are not uncommon, depending on crypto standards and tunneling/encapsulation modes.
Once you know the external PathMTU, you can derive the given PathMTU for the "inner" connections from that.
Then, for TCP based applications, proceed to check if the/all involved networking gear does TCP MSS clamping correctly (must be done pre-encryption if with VPN) so that the end systems get tricked into generating TCP payload chunks small enough to fit into a single packet-with-VPN-Headers.
For non-TCP applications, consider application layer configuration to send sufficiently small datagrams, and also check/verify that PathMTUdiscovery is properly supported by the networking gear, the end system's IP stacks and their security settings (dropping incoming IMP "packet too large" messages kills PathMTUdiscovery).
Also worth considering/reviewing: Avoid TCP-over-TCP in VPN scenarios. In general, you get a more robust solution when running the "outer" connection/tunnel with a connectionless protocol (IPSec with ESP, IPSec NAT-T on udp/4500, possibly DTLS on udp/443 or something entirely custom on UDP).
When using TCP as both outer transport and inner end-to-end protocol, you end up with two largely independent layers of TCP flow control and congestion avoidance, which can be really hard to get right.