Most communication networks are designed to be centrally controlled and scheduled. CAN, OTN, SDH, FR, GSM etc.
In the cell phone network central clocks are distributed (via network typically) to every network element. All traffic is time switched on/off the network. This allows the network wide latency to be fixed and predictable.
The alternative is Ethernet/Wifi where there is no central scheduling. In this model everyone waits for the medium to go quiet then transmits. In this situation packets can collide (two stations talking at the same time). In ethernet there is Collision Detection and in Wifi there is Collision Avoidance.
On centrally scheduled networks ATM QOS comes in 5 variants.
The two other variants are Real Time and Non Real Time. That's how frequently the use is calculated.
QOS in Ethernet is typically implemented as WRED - Weighted Random Early Detection and Discard https://en.wikipedia.org/wiki/Weighted_random_early_detection
Packets are put into 3 or more queues. The highest priority queue is drained first then other ques in order of priority. If any queue looks like it's going to overflow packets are dropped early to prevent overrun.
To answer your specific question:
The things that cause variability in latency are:
QOS queuing, CSMA/CD and retransmission, Router load (as the router gets more load it will slow down at 75% of capacity), Packet size (lots of small packets reduces overall bandwidth), Overhead signaling (Wifi management frames), Overall router load (some events on routers control plane can affect latency). L1/L2 activity (ARP), Fragmentation (if the router needs to fragment you will get a significant performance hit), Re-assembly, Re-ordering packets (specifically bad with IPSec) and probably more that I have forgotten.