I'm going to answer your questions out of order.
What's the minimum possible "hello time" for Rapid Spanning Tree (RSTP)?
Per standard, the range is 1-10 seconds, with a recommended default of 2 seconds.
Is the minimum dictated by the RSTP standard, or would it be switch-dependent?
Min/Max ranges are dictated per standard, but the default value is recommended (as mentioned above).
Actual implementation would technically be vendor specific, though they'd be unlikely to deviate from something so fundamental. For example, both Juniper and Cisco adhere to the 1-10 second range, and 2 second default values.
Is it possible to reduce the hello time to e.g. 0.1 seconds?
In theory? Sure. In practicality, not really as you'd have to find a vendor that supports it.
Or would it be a situation-dependent practical minimum due to increased BDPU traffic or some other issue?
This would be up to the network's operator and the specific reason for any variance in configuration of the hello timer would be pure speculation.