Each IP address must be unique, no matter what its local subnet mask is or what prefix size is used on a route.
Let us say, one address in our subnet is 10.0.0.1 (subnet mask /8)
If you use 10.0.0.0/8 on any interface you can use addresses from that subnet only on that segment and nowhere else in your network.
Now, we decide to create a subnet further, and use 3 bits from the remaining 24 to form its network ID, thus creating a /11 network.
You mean you replace the /8 subnet with a /11 subnet that's been split off from the original prefix? OK.
Thus we have two host address that are same, albeit with the different network mask.
No. If you use a 10.0.0.0/11 subnet you cannot use 10.0.0.0/8 for a subnet anywhere else in your network any more. Doing so makes routing ambiguous and you don't want that.
My question is, does there exist any system to prevent this?
Not by default. It's your task as a network admin to prevent this (usually there's an address plan or IPAM in place). And usually you've got some kind of general logic that simplifies things. For instance, you could use /24 for all productive subnets, and you could structure subnets in supernets by location, purpose, security zone, ... like this:
- 10.0.0.0/8 global scope
- 10.1.0.0/16 location A
- 10.2.0.0/16 location B
- 10.3.0.0/16 location C
- 10.x.0.0/24 client subnet on each location
- 10.x.4.0/24 server subnet on each location
- 10.x.8.0/24 IP phones on each location
- 10.x.12.0/24 utitity devices/IoT
- 10.x.16.0/24 security appliances (alarm, doors)
- 10.x.20.0/24 SAN
- 10.255.0.0/24 P2P link segment subnets /31