Your description and Heading contradicts each other.In description you mentioned that you want to achieve it without using VLAN and QOS.However,In description you have mentioned that "enable either VLAN X if its a IPTV Box connected or VLAN Y if its a SOHO Router connected".
Given that,the best method is to use QOS and VLAN to achieve your requirement.There is also another method do it using a different way. Iptables + tc in a linux based router
Identifier for Device:
You can use the Option 60 in DHCP request.Vendor Class identifier to understand the type of device connected.
Bandwidth rate limiting Logic
You can use both iptables policies and tc to limit the bandwidth for particular devices with different Vendor class identifier.
Since your query doesn't have more details about the bandwidth rate or name of vendor class identifiers.I will just give some sample commands for your reference.
Sample TC rule for 100mbps
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
I'm sure if you explore both tc and iptables,you should be able to come up with a script to achieve your requirement.It is good to use an openW.R.T based router with tc support for your setup.
Note: The best solution is to use VLAN + QOS solution.
EDITED based on the Comment
A good router/Gateway/CPE at the customer end can solve this problem. I belive what you are trying to achieve is to provide seamless services to your end customers.
Faster connectivity to End Customer and Priority for the SetTop Box Traffic in your network.Constraint: STB doesn't support QOS.
SOHO router packets need to be considered with lower priority and treated in a differnt VLAN.
Do you have access/own the Gateway/Router at the Customer Premise.If yes,then Scheduling and Prioritizing the packets with different queues can provide a solution for this.
Option #1: Using vendor class idetifier
Assuming that an STB can be identified using DHCP option 60 in the network,You can tag all the packets from STB with Higher priority with (eg. "CS5" for Broadcast Video).Similarly you can tag all the SIP/RTP packets with "EF" priority and rest of the packets in Best effort queue. This will help you to maintain a good QOS for customers.
Why don't you create and establish different PPPOE from WAN (at Router)based on the service.
PPP_IPTV for STB services at 184.108.40.206 IP range.
PPP_Data for Default data at 220.127.116.11
PPP_VOIP for VOIP services at 18.104.22.168
IP address mentioned above is arbitrary.