I have two layer-3 switches connected with two redundant links that I'm trying to run PTP OSPF sessions across, a la:
(OSPF) VLAN 1002 - - - - - - - - - - - - / \ CORE (RSTP) ACCESS \ / - - - - - - - - - - - - VLAN 2002 (OSPF)
The access switch was running RSTP, which obviously will just block one of the ports here so the second OSPF adjacency won't come up until RSTP reconverges. PVSTP solves that problem beautifully (along with some other STP weirdness I can't quite explain while running RSTP since it doesn't seem physically possible for any other loop to exist), except that it's limited to per-VLAN STP (not RSTP) on these switches.
I'd like to not change anything on the core switch stack to avoid bringing the whole network down, but when I try to use MSTP on the access switch instead of PVSTP, it's still blocking one of those redundant ports (and I have the other inexplicable STP issues). My thought was that by putting VLANs 1002 and 2002 in different MSTP instances on that access switch, it would work the same as with PVSTP (there are no other VLANs on either of those ports).
Am I missing something? Wouldn't the RSTP switch just see a different BPDU on each port, and the MSTP switch be keeping the BPDUs from the RSTP switch in separate spanning-trees? Does running two MSTP instances on one switch and RSTP on the other switch not work?
Also, it appears that after adding the VLANs to their respective MSTP instances, the ports they're on still reside in the CIST as well, but I can't seem to find a way to remove them. Am I missing something here? Here's the relevant documentation for the switch: https://img-en.fs.com/file/user_manual/s5900-24s4t2q-stp-configuration.pdf