I'm curious about why 10Base-T and then 100Base-T Ethernet networks used cables which had four pairs if they only needed two? Were there some cables that only had two pairs?
If we wanted to increase the speed of Ethernet further in the future, could adding more pairs of cables help?
10BASE-T saw first light as StarLAN that made use of the already existing twisted-pair category 3 telephone cabling (instead of the dedicated coax that 10BASE5/2 required). That multi-purpose cabling standard carries four pairs to each wall jack. StarLAN (10) and subsequently 10BASE-T had no use for more than two pairs, so they ignored the other two. 100BASE-TX borrowed FDDI's copper design ("CDDI") with also just two pairs, so it's the same situation.
The already existing cabling is also the reason for the use of straight-through cables as standard (with the crossover, MDI vs MDI-X logic on the device port side), and crossover cables only in special cases. Fiber uses crossovers throughout.
Cables with only 1-2 & 3-6 pairs are sub-standard, yet functional with 10BASE-T and 100BASE-TX. Of course, there are (or at least were) many installations and adapters that would split a four-pair cable into two independent 10/100 links (for cost, cabling restrictions, ...). That won't work with 1000BASE-T and faster though - the faster Ethernet variants split traffic into four full-duplex lanes to reduce the cable's frequency requirements to a quarter, so they require all four lanes.
For completeness, the 100BASE-T4 and 100BaseVG standards that initially competed with 100BASE-TX both used all four pairs of voice-grade category 3 cabling, non-intuitively both were inherently half duplex. In spite of the extra cost of upgrading to category 5 cable (but with lower port cost), 100BASE-TX quickly won the race and became ubiquitous.
If we wanted to increase the speed of ethernet further in future could adding more pairs of cables help?
It could, in theory, use more than the current four lanes. But nobody's going to redeploy their whole cable plant. By various standards (e.g. TIA-568), horizontal/tertiary cabling is commonly deployed using twisted-pair cabling (good for up to 10 Gbit/s, depending on category and length), but vertical/secondary cabling and upwards uses fiber anyway.
40GBASE-T and 25GBASE-T using category 8 cabling (30 m max) are already having a hard time competing with fiber and likely we won't get faster TP standards ever. Fiber is already at 400 Gbit/s and ready for much more using WDM or multi-lane fiber which is much more practical than with copper, so that's the future.
EDIT As has been pointed out in the comments, you could also add pairs for more speed using another cable. That is actually common practice called link aggregation (LAG).
However, LAG performance can differ substantially from a single, equally fast multi-lane link. Multiple lanes across a single cable are irrelevant to performance because data distribution across the lanes is very finely grained - usually on a multi-bit level, depending on the PCS line code.
An aggregated link distributes data coarsely on the frame level. You really have to avoid changing the frame order and considerably hurting overall performance, so traffic isn't distributed dynamically (by link utilization) but statically - by L2 addresses, L3 addresses, L4 ports, ..., depending on the switches and their configuration. The higher the layer, the better, usually.
At best, you distribute traffic on the L4 connection level - but still, no single flow can ever exceed the bandwidth of a single link. With L3-based distribution, that limit applies to any host-to-host connection. Much worse even, L2-based traffic distribution puts that limit on the sum of all flows between any two routers, hosts or mix. Accordingly, interconnecting 10G switches with aggregated 1G links is a bad idea mostly.
Aggregating multiple links is generally limited as well. The most common LAG protocol LACP allows a maximum of eight active links, and all switches I've seen limit the number to eight (or less) for static LAG as well. Ethernet's speeds usually increase tenfold between grades, so that's more than you can gain by LAG. Correspondingly, link aggregation is often just a stop-gap measure before upgrading the actual link speeds.
There's nothing to stop a new twisted-pair standard using a new connector in the design, with more wires.
If a physical site had to choose between using existing 8-wire / 4-pair and recabling everything to use a new standard, then the additional costs would push most sites to stay with existing wiring for as long as it fills the need.
One half-way point might be to use two RJ45 plugs, with the expectation that forward-planning sites have more than one socket in every desk location. Downside is that this will halve the effective port density in racks and switches, requiring more space in server/distribution rooms.
Ultimately, technologies survive for long after a replacement arrives. Many old timers have likely seen 10base2 still in use in the 2000's, far after cat5 cable "replaced" it. I had a 10base2 link from a router to a firewall for years, because the link was 2 Mbit and 10 Mbit was ample bandwidth, while the LAN was 100 Mbit and gigabit.
I've also seen a POC/demo of 100 Gbit fibre that used ten separate 10 Gbit links combined. However the cost was significant.
To answer the part of your question "Were there some cables that only had two pairs?":
I worked at a very large multi-building site that had 8wire/4pr twisted pair in the walls terminating with and split between two RJ13* jacks at each wallplate. (2 pairs going to each) One RJ13 was used for POTS phone, and the other was later used for 10baseT (and later 100baseT even though that's not supposed to work on Cat3) with the two services sharing the single UTP cable in the wall.
On the other end, in the telecom closet, the network parts of the circuit was split out to a large patch panel of RJ13s. Custom 4wire/2pr RJ13 to RJ45 patch cables were used on both ends. We used that successfully for almost 15 years until each building was rewired with new CAT5/6 throughout.
So yes, the fact that only 2prs were needed until gigabit came along allowed us to use cables that only provided that many (as patch cables, and to repurpose of the unused parts of the phone cabling)
EDIT: As has been pointed out, this is not an approved standard
*RJ13: I'm am more of a networking person than a telecom person, so you'll have to forgive me if I've used the incorrect RJ designation for 6P4C
