For IPv4, see RFC 791:
To fragment a long internet datagram, an internet protocol module
(for example, in a gateway), creates two new internet datagrams and
copies the contents of the internet header fields from the long
datagram into both new internet headers. The data of the long
datagram is divided into two portions on a 8 octet (64 bit) boundary
(the second portion might not be an integral multiple of 8 octets,
but the first must be).
When more than two fragments are created, all but the very last one need to have an octet size divisible by eight.
Technically, there is no way to specify a more granular fragment offset, as detailed in Ron's answer.
Is it mentioned in any RFCs that when one is splitting a Datagram into fragments, the bytes in the first fragment should be "the closest multiple of 8" or is it just a standard implementation.
No. "The closest multiple of 8" is an incorrect interpretation by a third party. As Ron has indicated, the largest multiple of 8 that fits the MTU is the most reasonable choice. The "closest" multiple may be larger than the MTU which isn't possible.
For completeness, for IPv6 the (end-to-end) fragmentation header isn't standard but an option. RFC 2460 defines for the field
Fragment Offset
13-bit unsigned integer. The offset, in 8-octet units, of the data following this header, relative to the start of the Fragmentable Part of the original packet.
So it's the exact same situation.