In brief, because a vlan should roughly equate to a subnet. If a destination address is in the same subnet as the source address, it wouldn't want to route it. You switch within subnets but route between them.
For your example to work routers 3 and 4 can't be on the same subnet.
If instead router 3 was 192.168.10.1 and router 4 was 192.168.20.1, they'd be in different subnets and would need routing to be able to communicate. You could put 2 sub interfaces of router 6 - 192.168.10.254 and 192.168.20.254 for example and then add a static default route to each of router 3 and 4 to point to the corresponding subinterface on router 6. That default route says "to get to anything not in my subnet, go through my default gateway at 192.168.10.254" (or 192.168.20.254 for router 4)