I am trying to ping between two devices A [Server] and B[Router]. From A to B the average ping response is about 1ms and from B to A is 0.5ms which is half. As a result, if I scp a file to B, by logging into A, it transfers very slow but if I try to scp the same file from A to B, by logging into B, the transfer is much faster. What is happening here and how can I debug this?
ping uses ICMP echo requests which are no exact method to measure anything except that the path is working and that there's an upper limit to the round-trip time.
ICMP messages have a very low processing priority on networking devices. While that doesn't matter much for a host - which does pretty much all processing in software and has a fast CPU - network hardware is normally much slower to respond: on the one hand, ICMP isn't offloaded by hardware (like the routing processing) and needs to be processed by the CPU and on the other hand, the CPUs on those appliances are usually much slower.
Your experience with SCP in each direction may have similar reasons but it's impossible to judge without details.