From what I can tell, each process could get its own IP address, with loads of IP's to spare.
What would the drawbacks be?
Essentially, your host would become a router and each process is a host (in the current system).
IPv6 does not have a NAT standard as IPv4 does (NAT breaks the end-to-end premise of IP, and IPv6 was designed to restore that). There is an experimental RFC for IPv6 NAT, but it is a one-to-one NAT at the network layer, rather than something like the IPv4 NAPT that also translates port addresses, and, in fact, the experimental IPv6 NAT RFC expressly forbids that.
If you think about the various transport protocols, TCP and UDP use ports, which are really addresses for those transport protocols. Other transport protocols may use other addressing, and some use no addresses.
Your idea would possibly work with either TCP or UDP, but only one, and probably not with other transport protocols. IPv6 is connectionless, like UDP, so it may work with UDP, but TCP is connection-oriented, and it performs a lot of work that would otherwise need to be performed by the application.
There are requirements for both connectionless and connection-oriented transport protocols. The predecessor to IPv4 actually had the equivalent of IP and TCP as a single protocol, but it became necessary to split them because some transport protocols need to be connectionless.
One drawback is that the upper layers would need to be aware of IP addresses, which sort of violates the layering principle. What would then happen if you switched to IPv4? Or something else?
How would the upper layers tell the network layer that it should start responding to an IP address? Suppose a new application starts up. How does the network layer know there's a new address to respond to?
IPv6 could theoretically eliminate NAT, but there are still reasons why an organization might want to hide its internal IP addresses.
The need for NATs as a way to share public IP adresses will be gone with IPv6, but NATs may still have some uses for security reasons.
Removing ports might be possible in theory, but it would be a huge undertaking. Even if you reserve the last 16 bits of the IP adress for port number + 1 bit for TCP/UDP flag, you would need to rework all the software that unpacks the pactet from the 3rd layer into the 4th one to use the bits from IP adress for port number and protocol, instead of data in the packet.
And that's the "easy" solution when you just put the port nubmers in the IP adress, if you wanted to get rid or port numbers altogether, you would need to change every software and library that uses ports, be it servers, clients, firewalls, etc etc. It would be basicly impossible, every software that says "open server at port X" which would work fine when just going from IPv4 to IPv6 would break instantly if you remove ports altogether.
It is infinetely more easier to just keep the ports.
It wouldn't be IPv6. Such a system is of course feasible, but it would be an variation of the IP protocol. Today, IP's contain two parts: a host identifier and a service identifier.
With your scheme, it would essentially be a service descriptor. This would require changes to DNS (e.g. how do you differentiate between the SMTP server for example.com and the www-server for example.com?)
I can't see how it would be beneficial over todays system with host:service descriptors. It would not achieve anything not possible today.
As others have said, having port numbers is a requirement, even if they end up being 'encoded' in the IP address of the service.
Some applications need more than one port number. For example, most web servers need port 80 and 443 (and maybe another port for websockets or whatever). Such a system would need three IP addresses with encoded ports. Whilst we've got plenty of IPs, it would (at present) mean system-level configuration would be required to support those three port numbers. Using a separate port number means the apps can just start up and use the ports without needing the sysadmin to do some config work first. As others have said - it's just easier to use a port number.
As for NAT, yes, IPv6 should remove the need for NAT in almost all cases. Some cases remain though, so it probably won't disappear entirely, but wide-spread use is likely to diminish significantly.
IP Addresses and Ports, though usually seen together, perform very distinct tasks. IP Addresses are used to route traffic to your computer specifically and port numbers are used by applications.
A very good analogy is sending mail to an apartment building. The address of the building would be equivalent to an IP address and the unit numbers would be analogous to a port number. For this analogy we'll say the building's address is 127 Home Ave. Say you wanted to send a letter to Henry T.T. Peterson, so you send a letter to the building, he doesn't receive it because it needs further direction to his apartment, apartment 80.
In theory, IPv6 address could take the place of both IPv4 addresses and port numbers, however this violates well-established precedents of networking and there are some major drawbacks:
As far as NAT goes, this was created for 2 reasons, there were already concerns that the world was running out of (or would run out of) public IP addresses, which was massively alleviated by allowing for organizations and even home networks an incredible amount of IP addresses that can be used internally, and allows for important machines to be "hidden" from the world (such as your database with sensitive information on it, or the CEO's laptop)
I hope this explanation was helpful; you see, both NAT and port numbers are a good thing, we wouldn't want to replace them with IPv6 because that is a huge step backwards, if anything we'd want to actually expand them, or replace them with something even better