How do mismatches occur; what is the impact of a mismatch on network connectivity; is resolution of mismatches really worth the effort; what are some ways to detect mismatches on a large scale?
To address your questions in order:
- Mismatches occur when auto-negotiation or manual configurations fails. This may be caused by physical layer issues or because manual configurations do not agree. Physical issues could be something like an inline hub or physical cable (missing/broken pairs) issues.
- Duplex mismatches can cause a significant impact to your network performance. Collisions, particularly late ones, can take a long time (relatively speaking) for the TCP stack to recover from. Collisions with UDP means the data may never get there or require an application layer recovery mechanism.
- Resolving mismatches is absolutely worth your time resolving. It may also hint at other issues since duplex/speed should be an assumed thing these days.
- Some of the centralized management tools (Solarwinds, Spiceworks, HP OpenView & others) probably have reports for interface errors which should include these types of issues.
To fully understand why duplex mismatches occur, you need to understand how the technology evolved.
Originally, all Ethernet was half-duplex. When full-duplex entered the picture, someone wisely decided that devices (especially half-duplex and full-duplex devices) should be able to agree between themselves on how they would communicate and auto-negotiation entered as well.
However, none of those older half-duplex devices were designed to auto-negotiate, so when the standard was written, it is to be assumed by the auto-negotiating device that if the other side did not participate in negotiation, it was to run in half-duplex mode because the device on the other side must be capable of only half-duplex.
As others have pointed out, auto-negotiation didn't always work well early on, so many devices were configured with static speed and duplex settings (often 100/full), and when a negotiating device is connected to such a device, a duplex mismatch occurs.
As to the problem, a duplex mismatch can be much worse than running in half-duplex mode. This is because one side (full-duplex) thinks it can transmit at any time even if it is current receiving. The half-duplex side will see this as a collision and back off, while the full-duplex side will keep transmitting.
If the full-duplex side tends to transmit a lot of data, this can "starve" the half-duplex side as it is waiting for the medium to clear before transmitting, causing frames to be queued and eventually dropped.
All in all, a bad situation to be in and one you should fix.
When it comes to detecting mismatches, you can look for errors. On the full-duplex side, you will generally see many runts and often CRC errors (vendors may use different terms at times). On the half-duplex side, you will often see collisions and buffer failures. Any decent management system should be able to provide you a list of interfaces that are generating a higher than expected number of errors.
These days the most common causes are links where one system (either network or end device) is manually configured, and the other automatic.
In the early days of auto-negotiation (full duplex 10Mb and Fast Ethernet) it was not uncommon for devices to fail to negotiate correctly.
Due to this (and other inertia related reasons) many large corporate and SP networks required manual configuration of some or all links.
These days there is no justification for doing this, and in fact on Gigabit Ethernet (copper at least) auto-negotiation is required, and well behaved devices will not allow it to be disabled. In some cases this may not be clear, for example on some Cisco kit "disabling" auto-negotiation on gig links simply restricts the values acceptable in the auto-negotiation process (which can be valuable if you don't alarm on unexpected interface speed & duplex).
Mismatches most often occur when one side of a link is explicitly configured and the other side is set to auto-negotiate. When devices are under separate management, the parties may fail to communicate and verify the settings. The impact to network connectivity ranges from unnoticed on light usage links to severe on heavily loaded links. It is generally worth the effort to resolve mismatches wherever possible. On Cisco switches, an interface reliability indicator of less than 255 is a good way to spot mismatches. This value can be polled with SNMP to detect mismatches on a large scale.