I have network connecting with 500 users and 50 servers connecting different layer3 switches and layer2 switch. Can i run rstp in few switches and mstp on remaining switches.
RSTP and MSTP were developed from the ground up to make them work together. We can see MST as being a extension over RSTP. The MST BPDU is in fact and RSTP BPDU + some few additional things required to support several instance. So basically, from the perspective of an MST bridge, an RSTP peer looks like a different MST bridge in a different region. We don't find any issue running RSTP and MST both except of course that you will only be able to run a single instance (instance 0, the CIST) at the boundary between RSTP/MST. Hence hop count MST uses is only within a region. Between regions, MST uses the message age, just like RSTP.
Cisco will not support a real RSTP as defined in the IEEE. We only have Rapid-PVST, which is a proprietary version running one instance of RSTP per VLAN. The trick is that we don't need to support IEEE RSTP because MST is a superset of RSTP. Running MST with no region configuraiton is basically equivalent to running RSTP