A failure in the return loss measurement means that you have excess reflection energy from the cable configuration due to impedance mismatch. This failure means that a standards compliant transceiver is not guaranteed to work at IEEE BER requirements - but it is not guaranteed to fail.
The reflection energy can limit performance since some aspects of efficient physical layer chip design (i.e. chip area/power/cost) depend upon predictable cable system behavior, including acceptable return loss performance (e.g. number of taps in equalizers and echo cancellers).
However, depending upon how much link margin you have due to other factors - e.g. noise, crosstalk, insertion loss - the link can still operate okay without errors even with a RL failure, though in some cases it may result in very infrequent errors that will not show up in short tests.
Note that there is some margin built into IEEE compliant systems - many systems companies won't accept physical layer systems that only just meet the standard requirements. Therefore you may not see any bit errors with marginal cable test failures but it is safer not to rely on this fact.