My question has to do with layering of unicast over multicast and has two parts.
Part 1. IP unicast over MAC multicast
First part of the question is whether there is a standard that prohibits use of IPv4 unicast addresses with multicast MAC address in an ethernet frame ?
I found the following discussion on the topic from 2005. https://forums.tomshardware.com/threads/multicast-mac-and-unicast-ip-address.951449/
And my understanding of the issue after reading through that email chain is that it this not explicitly prohibited. (at-least till 2005) and there are multiple vendors whose clustering solutions rely on this.
I am guessing that as far as standards compliance goes this is still the case. What I would like to know is from practical standpoint of commercially available network equipment and network stacks, how such packets are handled. For example a focus on security in recent years may have resulted in stacks that do not tolerate anything that is not a typical use case.
Part 2. TCP over IP Multicast
This may sound a tiny bit crazy or pointless, please bear with me.
Can stream (TCP) sockets work with multicast IP addresses ?
This basically boils down to two things from a socket application's perspective. Would calling "connect" API specifying a sockaddr that specifies a multicast destination address work ? (Assume that the destination node is designed to reply to ARPs for specific unicast IPv4 addresses with a multicast MAC address)
Also can I expect setsockopt(sd, IPPROTO_IP, IP_ADD_MEMBERSHIP, group address .... ) on a stream socket that is bound and listening on a TCP port (say 0.0.0.0:port) to succeed ?
I had a quick look at search at linux network stack and did not see a specific check that would prevent this, https://elixir.bootlin.com/linux/latest/source/net/ipv4/ip_sockglue.c
In my limited experience working with a network stack in a small RTOS, I would be surprised if the above combination worked out of the box. For example, this would mean the stack received a multicast ethernet frame (having configured any MAC layer hardware filter to allow this) and then looking at protocol field in IPv4 header, associate the payload with a TCP port object without blinking.
Some background. I am not doing this to implement 1:N or N:M TCP sessions. Quite boringly, all I am interested are standard point to point TCP sessions.
Unfortunately, unless I come up with concrete reasons why the above mentioned combination will not work or is undesirable, I will have to deal with layer two hardware that will most probably refuse to forward unicast ethernet frames (based on statically defined forwarding tables for handling TSN streams) and implement application layer protocols some of which use TCP over this network.
One would have thought a straightforward response such as "it will not work!" would be good enough... apparently not any more in this brave new world. and as they say "with enough thrust..."