I want to make sure there isn't a switch in between.
It's not usually possible to see if there's a switch between A and B, unless you have a candidate switch: then you can disable/reenable sockets and see what happens to link status.
"show arp" ... showing the mac address of the directly connected layer 2 device right?
"Directly connected layer 2 device" means "host ignoring any switches" then you are correct. It is direct at layer 2, you put the frame on the wire and it's supposed to arrive at the targe. It's possibly not direct at layer 1, the cable.
not the mac address of the next hop router correct?
Correct, unless you have devices doing proxy ARP, in which case you'll have the ethernet address of the proxy, not the actual host.
If there's a next hop router in the normal routed situation, there's no ARP entry for the target.