Using MSTP with RSTP

Question

I have two layer-3 switches connected with two redundant links that I'm trying to run PTP OSPF sessions across, a la:

                (OSPF)
              VLAN 1002
      - - - - - - - - - - - -
   /                           \
CORE (RSTP)                   ACCESS
   \                           /
      - - - - - - - - - - - - 
              VLAN 2002
                (OSPF)

The access switch was running RSTP, which obviously will just block one of the ports here so the second OSPF adjacency won't come up until RSTP reconverges. PVSTP solves that problem beautifully (along with some other STP weirdness I can't quite explain while running RSTP since it doesn't seem physically possible for any other loop to exist), except that it's limited to per-VLAN STP (not RSTP) on these switches.

I'd like to not change anything on the core switch stack to avoid bringing the whole network down, but when I try to use MSTP on the access switch instead of PVSTP, it's still blocking one of those redundant ports (and I have the other inexplicable STP issues). My thought was that by putting VLANs 1002 and 2002 in different MSTP instances on that access switch, it would work the same as with PVSTP (there are no other VLANs on either of those ports).

Am I missing something? Wouldn't the RSTP switch just see a different BPDU on each port, and the MSTP switch be keeping the BPDUs from the RSTP switch in separate spanning-trees? Does running two MSTP instances on one switch and RSTP on the other switch not work?

Also, it appears that after adding the VLANs to their respective MSTP instances, the ports they're on still reside in the CIST as well, but I can't seem to find a way to remove them. Am I missing something here? Here's the relevant documentation for the switch: https://img-en.fs.com/file/user_manual/s5900-24s4t2q-stp-configuration.pdf

Answer 1

My thought was that by putting VLANs 1002 and 2002 in different MSTP instances on that access switch, it would work the same as with PVSTP (there are no other VLANs on either of those ports).

That's how MSTP instances work. However, all the switches need to be in the same region, and have exactly the same configuration. (check configuration digest.) By default, every VLAN is in the common instance. (IST/CIST/instance 0/whatever your switch calls it) With only one instance, MSTP behaves just like (R)STP. With multiple instances, VLANs should be pruned instead of entire links being blocked, but some tweaking of priorities/etc. may be necessary.

A last resort may be to simply turn STP off on those two ports. But that's mess for another day.

Answer 2

One of the important concepts in MSTP is the MSTP region, which is a set of directly connected MSTP switches with the same configuration (digest). MSTP rules apply only to the region and intra-regional links. Inter-regional links are fully backward compatible with RSTP.

If you are running exactly one MSTP switch, MSTP rules apply on exactly zero links. You are always wasting your time configuring multiple MSTP instances on only one switch.

Answer 3

RSTP, which obviously will just block one of the ports here so the second OSPF adjacency won't come up until RSTP reconverges

With RSTP, one of the link will always be blocked. RSTP is completely VLAN-agnostic.

by putting VLANs 1002 and 2002 in different MSTP instances on that access switch, it would work the same as with PVSTP

Not by itself. With MSTP you need to use different root bridges in each instance to get different spanning trees, or use different port priorities to move the active link to the other port pair. Note that RPVST+ also requires tweaking to utilize different links for different VLANs.

Wouldn't the RSTP switch just see a different BPDU on each port, and the MSTP switch be keeping the BPDUs from the RSTP switch in separate spanning-trees?

RSTP interoperates only with the MSTP common instance.

Does running two MSTP instances on one switch and RSTP on the other switch not work?

The common denominator is RSTP which always blocks (all but) one link.

after adding the VLANs to their respective MSTP instances, the ports they're on still reside in the CIST as well, but I can't seem to find a way to remove them

As Ricky has already pointed out, MSTP instances require identical configurations on all region switches in order to work. Likely, there's a fallback to RSTP, check the log.

Since you seem to be running those VLANs only in order to get routed links, you should consider turning off MSTP/RSTP on these ports. If the switches permit, routed ports would also work as they don't participate in xSTP.

MSTP instances would work but require MSTP in both sides. Limited to RSTP, there's not much else you can do.

(LACP) LAG would also work between two switches, but it would connect each VLAN over each physical link, removing your control over the physical distribution.