What happens if traffic is sent to a multicast group from a host which is not a member of that multicast group ? Does the router keep any information for a valid Multicast source and discard this traffic ? I could not get a clear picture of (*, G) vs (S, G) entry. Will the packet be forwarded since there is a (*, G) entry ?
Multicast senders are not generally part of the multicast group they are sending to, since they already have the information they are sending to the group.
Depending on your multicast configuration the router could take the new multicast packet and forward to a known rendezvous point and the RP will forward towards interested receivers. This would create an (S,G) entry, if you want to control who is sending to your multicast groups you need to limit the sender source address.
Other configurations could have different forwarding decisions.
Multicast sources register with the RP (rendezvous point) through their local first-hop routers (DR - designated router) and the receivers "join" the shared tree rooted to the RP via their own local first hop DR. Now (in sparse mode) the receivers join the shared tree (*,G) via IGMP messages indicating they wish to receive the multicast, and then the (S,G) (shortest path) state is built from RP towards source's DR. Once this connection is realized, the multicast traffic is sent down the shared tree. However, once the receiver's DR sees the multicast's source IP in the first few packets down the shared tree (*,G) , it switches over to use the shortest path (S,G) and the shortest path tree means it uses the unicast routing shortest path back to the source.
To answer the first part of your question "What happens if traffic is sent to a multicast group from a host which is not a member of that multicast group ?" If you do not have any security built into your multicast configurations, traffic can be delivered to a multicast group from a rouge source.
The second part of your question "Does the router keep any information for a valid Multicast source and discard this traffic ?" Again, if you had no security, yes the DR routers accepts this as a valid source and retains it's (S,G) state, which will also send a PIM source message to the RP, which maps it to the source.
And the third part of your question "I could not get a clear picture of (*, G) vs (S, G) entry. Will the packet be forwarded since there is a (*, G) entry ?" Again, if you had no security, the rogue multicast source can and would send traffic to all requesting receivers. The rogue builds a valid, but unauthorized shared tree (*,G) to the receivers and once the receiver's DR receives the first few multicast packets from the unauthorized source, the DR will switch to the unicast routing distribution scheme, meaning it will use the shortest path to the source (S,G).
So in your case, for "rogue source security", which is a very common best practice, you can use the "pim accept-register" on the RP and all DR's and then this is tied to an ACL permitting all valid multicast sources. "ip pim accept register list valid_sources" "ip access-list extended valid_sources" "permit ip 126.96.36.199 0.0.0.15 239.x.0.0 0.0.255.255