Why Do We Need MAC Addresses?

Every NIC on the wire saw every packet; the destination mac address indicated whether the packet was interesting enough to each node to be worth putting a copy in host memory and bothering the CPU to look at.

This is fundamentally incorrect. IPv4 for LAN, sure - but "bootstrap" it is not.

There's plenty of L2 that have no MAC: PPP, ATM, Frame Relay, etc. Also, MACs aren't needed for IPv6. ND works without MAC in v6.

And, finally... There's no MAC for loopbacks.

Until you consider broadcast/multicast traffic.

You can put other things than IP packets inside Ethernet frames. You can put IP packets into non-Ethernet carriers.

The short version is that small LANs route frames by MAC addresses and don’t know about IP addresses, and the Internet cares only about IP addresses and not MAC addresses.

Notably, none of this implies IP. IP was the expensive DARPA project that the military was building out. Ethernet is just "I am machine X, I want to talk to machine Y, here's some data". It just so happens that Ethernet was very popular, and people wanted to route IP over Ethernet. But you could route all sorts of other protocols over Ethernet as well and it would more or less work. So now we need two different classes of addressing: one for just the local network, and one for the IP network running on top of it.

And to make matters even worse, we don't use Ethernet in its original configuration anymore. First we replaced the loop of coax and vampire taps with 8P8C wire and passive hubs, then switches got cheap enough that we stopped making hubs. Ethernet framing also wound up being used for Wi-Fi, except like 99% of all Wi-Fi connections are in Infrastructure mode running IP anyway.

If Ethernet had been built to run IP exclusively, we wouldn't need Ethernet's header or MAC addresses at all. It would just need to encode IP packets and their boundaries. Even IPv6 wouldn't need a separate header as IP has its own version field.

[0] MAC addresses are supposed to be unique, but let's just say it's "LAN-scoped" and leave it at that.

The headers of IP packets wouldn’t be affected by MAC addresses, Ethernet packets carry IP packets as their payload and would have their own headers as appropriate.

A router’s job is essentially to “pass on” or relay IP packets based on its routing tables, they are what connect networks together.

(Also, to have been a fly on the while when you worked out that whole congestion thing.)

This isn't entirely true. In ye-olden days, Ethernet was a proper bus architecture, where you would have many devices talking on a single line.

As networks got larger, we wanted to have more devices than we could get on a single cable, so we introduced cheap booster hubs that would physically connect multiple cables. Since these hubs didn't have any fancy switching logic, it continued to look to devices like everyone was on a single bus (which they basically were).

Eventually, compute became cheaper, and the limitations of having a massive bus more apparent, so we started replacing some of the Ethernet hubs with Ethernet switches to reduce contention. It still looks as if everyone is on a single bus; but now a node only sees traffic involving a node on its side of the nearest switch. Notably, these switches were drop in replacements for hubs, so they did not rewrite MAC addresses.

Nowadays, actual Ethernet hubs are a dying breed (although I still keep one lying around to avoid buying an expensive managed switch with a monitor port). Almost every Ethernet cable you run into today is a genuine point-to-point connection connecting two specific nodes, and carrying only the data that needs to be sent between those nodes.

Now that modern Ethernet is a full featured packet switched protocol, it has to deal with all the routing issues that come with that. This led to the introduction of the Spanning Tree Protocol, to allow Ethernet switches to construct Ethernet routing tables. Notably, nothing inside of these Ethernet networks replaces MAC addresses. When computer A sends to computer B on the same Ethernet network, the destination MAC address A uses is B's actual MAC address, regardless of how many switches are between them.

Ethernet can't scale like IP because it's a non-hierarchical address structure.

They kinda are already, your phone probably uses a random MAC address for each network it connects to.