I have been told that Ethernet magnetic transformers are used for base-t Ethernet when transmission is sent over a lengthy cable.
They are always used, not just when "sent over a lengthly cable"
What is the purpose of the transformer?
The primary purpose is isolation. Typically they are also used as part of the signal conditioning, turning a pair of single ended drives into a differential signal on transmit and establishing the correct common mode voltage for the receiver on receive. For this reason the device-side of the transformers is usually center-tapped.
Isolation is a very good idea on communications systems that are linking lots of hardware over a wide area. You don't want fault current/voltages in from faults in the mains wiring or devices to spread through your communications wiring.
There are basically two options for isolation, opto and transformer. Transformer isolation has a couple of major advantages. Firstly the signal power passes through the transformer which means you don't need to get a power supply to the "isolated" side of the barrier. Secondly transformers are very good at generating and receiving differential signals while providing high common mode rejection, this makes them a good combination with twisted pair wiring. Thirdly it is easier to design transformers for high frequency (aka high speed) than optocouplers.
Transformer coupling does have some downsides, transformers don't work at DC and small transformers that work well at high frequencies don't work well at low frequencies but this is easilly dealt with through line coding schemes that avoid low frequencies.
P.S. I am unsure about how data is transmitted over Ethernet. Is it that all differential pairs (DA: pins 1 & 2, DB: pins 3 & 6, DC: pins 4 & 5, DD: pins 8 & 7) are busses where data can be transmitted both ways (unlike the UART where RX has to be connected to TX)? and in case I am just using 2 pairs, would I be connecting only DA and DB?
It depends on the version of Ethernet. 10BASE-T and 100BASE-TX used one pair in each direction. On older hardware you had to manually ensure that transmitter was connected to reciever (using crossover cables if needed) but more recent hardware usually has AUTO-MDIX which figures it out automatically.
1000BASE-T uses all four pairs in both directions at the same time using echo cancellation techniques to seperate received data from transmitted data.
I think 10GBASE-T also uses echo cancellation techniquest but i'm not postive on that.
Is there also an issue with connecting a 4 wire device to a network that uses 8 wires?
Most devices suport lower speed modes but not all. In particular twisted par to fiber media converters usually only support a single speed on the twisted pair side. Devices that support 10GBASE-T usually also support 1000BASE-T but only sometimes support 100BASE-TX and afaict never support 10BASE-T.