I'll be trying an answer here, but it is a difficult subject.
First I think we should note that 802.11ac traffic is not necessarily in MU-MIMO, as your title might suggest. Today this technology is still under test at the WiFi Alliance and there is no client device that implement it. Also, in 802.11ac, MU-MIMO is used only in a downlink context, thus it is really named DL-MU-MIMO.
Channel sounding and beamforming
The AP sounds the channel the same way it does in SU-MIMO, using Null Data Packets sounding PPDUs (NDP for short).
As IEEE Std 802.11ac-2013, 22.3.11 says it, the AP uses DL-MU-MIMO beamforming to perform the transmission using different subsets of the same space-time streams:
With SU-MIMO beamforming all space-time streams in the transmitted signal are intended for reception at a single STA.
With DL-MU-MIMO beamforming, disjoint subsets of the space-time streams are intended for reception at different STAs.
As in SU-MIMO beamforming, DL-MU-MIMO beamforming uses a steering matrix determined by beamformees feedback matrices that are sent to the beamformer (usually the AP). The feedback matrix consist in a local mesure of the channel using "training symbols" previously transmitted by the beamformer. The beamformee sends it in a compressed state to the beamformer, which continues the process by sending a "Beamforming Report Poll" to the next beamformee.
The steering matrix then provides the informations used to create the space-time streams in a way that it suppress crosstalk between participating beamformees. And the beamformee recreates its feedback matrix on each NDP sounding PPDU received.
Once that is done:
The AP can create up to four A-MPDUs, each carrying MPDUs destined for an associated MU-capable STA. The AP then transmits the A-MPDUs simultaneously in separate space-time streams such that each recipient STA is able to demodulate the space-time streams carrying its A-MPDU.
Channel acquisition
There is no relation between MU-MIMO and RTS/CTS. The acquisition of the channel works the same way.
In theory, the AP uses contention windows and if it fails and leads to retries, it uses RTS/CTS to secure the channel transmission as a way to ensure that all frames will be transmitted this time. In a practical context, the implementation is dependent of the chipset and might differ from the theory.
Regardless and concerning your question, RTS/CTS is not station-dependent. When the AP wants to transmit a frame over the full 80MHz of its channel, it sends a Non-HT (meaning a 802.11a) duplicated RTS frame across all four 20MHz channels, indicating that it would like to acquire the whole channel. Then, receivers of the RTS (every STA in the channels) do their own Clear Channel Assessments and, for each free channel, send CTS frames. Then the AP is able to take a decision based on the results.
In our context, that would mean both stations would send CTS frames. Then, the regular dynamic bandwidth process would be used to determinate the best bandwith available. If one of the stations cannot send any CTS frames (on any of the 4 20MHz channels), for example because of some interferences on the whole 80MHz bandwidth, the AP would defer its transmission until the channel is newly free (or it might change channel if a real-time auto channel selection mechanism is implemented, allowing the transmission to be carried on). In the end the dynamic bandwidth process will select the best bandwith by taking in account all CTS frames received, but the transmission decision stays implementation-dependent.
One last note: in the case of your network, there would likely be near to no point in using DL-MU-MIMO as your stations are almost out of range. MU-MIMO would be best used at low-range, eventually mid-range.
Sources:
IEEE Std 802.11ac-2013
The pictures and some sentences are directly taken from 802.11ac: A Survival Guide, Matthew S.Gast, O'Reilly ed.. I invite you to buy it as it really better than the standard to understand how .11ac works.
