Let's consider only the Root Bridge and the switch not participating in RSTP.
The root bridge (which runs RSTP), puts one of its port connected to the non-RSTP switch, to Backup role (Discarding state) and the other port continues in Designated role (Forwarding state).
Reason: Initially both ports of the Root Bridge sends BPDUs. The peer non-xSTP node, floods the BPDU (doesnot forward BPDU since Dest MAC is a multicast), so the BPDU is received by another port of the ROOT Bridge.
- Then comparison takes place with the information in BPDU and the information in the receiving port.
- Here all the parameter matches (Root BridgeID, Root Path Cost,... ) except the PortID parameter.
- Hence among the port with lower (Better) PortID will be chosen as Designated, and other will be chosen as Backup.
[In some switches, BPDU dest MAC frames, will be dropped, which might lead to loops]
Kindly confirm whether the answer clears your doubt.