Assume 2 multicast IP addresses IP1 and IP2 whose mac addresses are same due to collision
Let's say a host's NIC has joined multicast stream IP1 and another host on the original host's subnet joined multicast stream IP2
Now how can we make sure the original host's NIC only takes packets from IP1 and not IP2
Multicast is really sent everywhere on the same LAN. Only hosts that have subscribed to the multicast group will listen to a multicast packet for that group.
Multicast has its own set of destination addresses. For IPv4, the range of multicast MAC addresses is 01:00:5e:00:00:00 to 01:00:5e:7f:ff:ff. There are 28 bits of IPv4 multicast group addresses, but there are only 23 bits of MAC multicast addresses. That means that every multicast MAC address represents 32 different IPv4 multicast addresses (28 - 23 = 5 and 2^5 = 32). The difference is handled at layer-3.
(IPv6 has a much, much larger multicast address range than IPv4, 120 bits vs. 28 bits, and its MAC multicast address range is 33:33:00:00:00:00 to 33:33:ff:ff:ff:ff, which is larger than the IPv4 MAC multicast address range, but again there is overlap, and IPv6 multicast has flags and scopes, and it is much more complex than IPv4 multicast.)
At layer-2, the layer-3 multicast address is converted to the MAC multicast addressing for the destination MAC address. A host subscribed to a multicast group will also listen for traffic with the multicast MAC address for the multicast IP address, and it will pass that traffic up to layer-3 to determine if the layer-2 multicast address is indeed for the layer-3 multicast group subscribed to by that host.
Host1 and host2 might get both streams - it is the NIC/OS job to do the final filtering.
Moreover, you don't even need multicast MACs collision to get extra traffic - as switches use hash of these addresses (many popular ones are VERY limited - e.g. I'm using D-Link devices and they apparently use ony 4 bits, every 32nd multicast group is received!) the real traffic might be much larger than expected.
While my OS receives 450 Mb/s of multicast streams, the raw port statistics reveal 700 Mb/s of actual traffic. After flipping "allmulticast" flag on the NIC I can easily see all the extra groups with tcpdump. And no, it's not related to duplicated multicast IP group (with different ports), only to the hash collision of non-collided multicast MAC addresses.
