In terms of design principles, the most common implementation is to align your VLANs with your organizational structure, ie Engineering folks in one VLAN, Marketing in another, IP phones in another, etc. Other designs include utilizing VLAN's as "transport" of separate network functions across one (or more) cores. Layer 3 termination of VLANs ('SVI' in Cisco parlance, 'VE' in Brocade, etc) is also possible on some devices, which eliminates the need of a separate piece of hardware to do inter-VLAN communication when applicable.
VLANs become cumbersome to manage and maintain at scale, as you've probably seen cases of already on NESE. In the service provider realm, there's PB (Provider Bridging - commonly known as "QinQ", double tagging, stacked tag, etc), PBB (Provider Backbone Bridging - "MAC-in-MAC") and PBB-TE, which have been designed to try to mitigate the limitation of the number of VLAN ID's that were available. PBB-TE more aims to eliminate the need for dynamic learning, flooding, and spanning tree. There's only 12 bits available for use as a VLAN ID in a C-TAG/S-TAG (0x000 and 0xFFF are reserved) which is where the 4,094 limitation comes from.
VPLS or PBB can be used to eliminate the traditional scaling ceilings involved with PB.