Can you make an HA pair out of an actual Nexus 1110 appliance vsm and a 1000V VM based vsm in one pair? If so can anyone point to documentation that allows it?
I found only one reference to this, and it was the software configuration guide for the Nexus 1010 appliance. Read through and you'll see the following statement:
Once you have created the virtual service on the primary Cisco Nexus 1010, an HA pair forms between the secondary VSM on the ESX and the primary virtual service on the Cisco Nexus 1010.
This implies that - while meant to only be a transitory state, I'm sure - it is at least possible to form an HA pair between a 1000v VSB on a 1010 appliance and the ESXi-hosted VSM.
Since you're using the newer 1110 model, I'd refer to the new guide, but the aforementioned statement has been changed to only reflect migrations from the 1010 to the 1110, and does not include a migration directly from the hypervisor to the 1110:
Once you have created the virtual service on the primary Cisco Nexus 1110 Series, an HA pair forms between the secondary VSB on the Cisco Nexus 1010 series and the primary virtual service on the Cisco Nexus 1110 Series.
EDIT: I neglected to read the section titled "Migrating a VSM". I instead linked to "Migrating a VSB" which of course is an appliance-only concept. Refer here and you'll see a very promising, similar statement to before:
Once you have created the virtual service on the primary Cisco Nexus Virtual Services Appliance, an HA pair forms between the secondary VSM on the ESX and the primary virtual service on the Cisco Nexus Virtual Services Appliance.
I will say this much - I wouldn't recommend mixing the two permanently (i.e. not as part of a transitory migration state). Technical issues aside, you're undoubtedly going to get an earful every time you call TAC. If there's budgetary constraints that prevent you from getting a second physical appliance, just go all-virtual.