How does STP/RSTP determine which bridge is the root bridge, and how are paths to the root bridge selected?
Not sure how much you know about switching and spanning tree but basically when starting out all switches claim that they are the root. All switches send BPDUs (Bridge Protocol Data Unit) which contain a priority and the BID (Bridge ID).
The BID is 8 bytes long. 6 bytes is used for the MAC address of the bridge. 12 bits is used to indicate the VLAN, this is called extended system ID. 4 bits are used to set the priority. Lower priority means it is preferred compared to a higher. The priority is set in multiples of 4096.
If there is a tie in priority then the lowest MAC address will determine which bridge becomes the root.
To select the path to the root the cost to the root is calculated. As the BPDU travels from the root downstream the cost is increased INBOUND. 802.1D-1998 (legacy STP) had a cost of 19 for a FastEthernet interface. The newer standard 802.1D-2004 defines a cost of 200000 for FastEthernet.
If there is a tie in cost then choose the BPDU that came from the switch with lowest BID. If that is a tie as well (multiple links to same switch) port ID comes into play. The port ID is from the upstream switch as well and consists of a port priority and port ID which identifies the interface. The default port priority is 128.
There is a lot to spanning tree but these are the basic steps.
You're asking about STP theory:
But the basic idea is that each switch has a Bridge ID which is a combination of its priority and its MAC address. By default, all switches use 32768 as their priority so by default the switch with the lowest MAC will act as the Root Bridge. You can manipulate this by adjusting the priorities, since the default Root Bridge is likely not the one you want to act as Root. You would decide which switch to use as root based on your layer 2 designs and where you anticipate layer 2 traffic to travel most.
The cost of the path to root is the sum of the link costs between the port in question and the relevant port on the root bridge. The faster the link, the lower the cost. So ten hops to root over 1GBPS links would give a port cost of 40 (4+4+4+4+4+4+4+4+4+4), while a single 10MBPS link would be 100. So the root port would be the one that connects to the 1GBPS links.
You can also manipulate the STP paths and roots per vlan. The link above will get you started. It can be as complex as you want it to be.
So as to not repeat the answers here and to present an alternative view toward a best practice, the question should be "How do I predetermine the root bridge so there's a deterministic view of the network ahead of troubleshooting?"
Set your aggregation or distribution switches as root by splitting your vlans between a primary and secondary switch. Usually odd number vlans get set on one switch as root and even numbered vlans as secondary, then the opposite on the other switch.
Sw1: spanning-tree vlan 1,3,5,7 root primary spanning-tree vlan 2,4,6,8 root secondary Sw2: spanning-tree vlan 1,3,5,7 root secondary spanning-tree vlan 2,4,6,8 root primary