IP is the common ground that ties the Internet together. IP can run on top of many different underlying network technologies and many different higher level protocols can run on top of IP. Parts of the Internet certainly do run on top of ATM networks. Ethernet is becoming the dominent technology but it's by no means the only technology.
Being "on the Internet" means being able to reach all or at least the vast majority of other hosts "on the Internet". That means interconnecting with other networks.
There are two main types of relationship, peering and transit (and also various intermediate possibilities).
In a transit relationship, a customer pays a provider to deliver traffic to and from the "Internet in general". The provider will almost certainly charge the customer for this service. It will almost certainly run over a private link.
In a peering relationship, two networks interconnect so that each network and it's customers can talk to the other nework and it's customers. The agreement may be "settlement free" or it may be paid depending on the relative negotiating positions of the two proviers.
At the top of the pile are transit free providers (sometimes called "teir 1", but the traditional definition of "teir 1" required settlement free peering while it's possible to become transit free using paid peering). These all peer with each other.
P.S. It is exactly because IP is the common ground that the process of upgrading from IPv4 to IPv6 is so painful. Essentially there are now two seperate Internets, the IPv4 internet and the IPv6 internet. They run on top of the same infrastructure and there are some limited transition mechanisms but they are essentially seperate networks.