The Single Byte That Kills Your Exploit: Understanding Endianness

Learning this initially was confusing for me too, aren't we arranging bits?

Little endian does appear strange at first, but if you consider the motivation it makes a lot of sense.

Little endian's most valuable property is that an integer stored at an address has common layout no matter the width of the integer. If I store an i32 at 0x100, and then load an i16 at 0x100, that's the same as casting (with wrapping) an i32 to an i16 because the "ones digit" (more accurately the "ones byte") is stored at the same place for both integers.

Since bits aren't addressable, they don't really have an order in memory. The only way to access bits is by loading them into a register, and registers don't meaningfully have an endianness.

You can't address individual bits. There is no way of telling if the LSBit is "left" or "right" of the MSBit. So endianness can't be about that.

For bytes, you can distinguish them, as you can look at the individual bytes produced from a larger-than-byte store.

The concept of order can only matter down to the units that are addressable.

Bits are typically not addressable, therefore do not have endiannness.

Bits are manipulated by special instructions, and those instructions are tied to arithmetic identities, due to the bits being interpreted as a binary number: like that a shift left is multiplication by 2.

In many instruction sets, the shift is a positive amount, and whether it is left or right is a different instruction. If it were the case that shifting one way is positive and the other way negative, then you have a kind of endiannness in that one machine uses positive for multiplication by powers of two, whereas another one for division. That would not result in an incompatible storage format though.

When data is transferred between machines as a sequence of bytes, there is a bit order in question, but it is taken care of by the compatibility of the data links.

Classic Ethernet is little endian at the bit level: the baseband pulses that represent the bit of a byte are sent into coax cable least-significant-bit first. RS-232 serial communication is the same: least significant bit first.

I think I²C is an example of a data link / physical protocol that is most-significant-bit first. So if you somehow hooked up an RS-232 end to I²C and got the communication to work, the bytes would be reversed.

We rarely, if ever, see bit endian effects because nobody does that --- transmit bytes between incompatible data links. If won't work for other reasons, like different framing bits, signaling conventions, voltages, speeds, synchronization methods, checksums, ...

Endianness of bits shows up in some data formats which pack individual bitfields of variable length.

Bitfields in C structures reveal bit endianness to some extent. What typically happens is that on a big endian target, bit fields are packed into the most significant bit of the underlying "cell" first. E.g.

   struct { unsigned a : 1, b : 1 };

the underlying cell might be the size of an int, like 32 bits. So where in the cell do "a" and "b" go? What you see under GCC is that on a big endian target, b will go to the most significant bit of the underlying storage cell, and b to the second most significant one. Whereas on little endian, a goes to the least significant bit, and b to the second least. In both cases, the bits map to the first byte, at the lowest address.

So in a certain sense, the allocation of members in C, as such, is little endian: the earlier struct members go to the lowest address, regardless of machine endian. It is probably because of that the bit order follows. Since putting bitfield a at the lowest address, as mandated by C field layout order, means that it has to go into the first byte, and that first byte is the most significant byte under big endian, it makes sense that the bit goes into the most significant bit position, for consistency.

That way we only have two possibilities to deal with for, say, a memory mapped status register:

    struct port_status_word {
  #if HAVE_BIG_ENDIAN
      unsigned transmit_ready : 1;
      unsigned data_received : 1;
      unsigned carrier_present : 1;
      // [ ... 29 more]
  #else
      // [ ... 29 more]
      unsigned carrier_present : 1;
      unsigned data_received : 1;
      unsigned transmit_ready : 1;
  #endif
   };

If we had separate byte and bit order, we would need two levels of #if nesting and four possibilities, which is even more ugly.

it does seem like the audience the article is explicitly targeted for is an edge case. people who understand enough to be writing an exploit but are somehow unaware of their target architecture works.

but i guess the real target audience is probably people that are just starting out on CTFs and just trying to string stuff together without a proper understanding of the fundamentals. everyone has to start somewhere and i guess if people are just using packers and tools to generate exploit code then its quite easy to use the wrong flags and not know what is going on.

People learn things in different orders. For many people, low-level CTF challenges are their introduction to computer architecture (a good way to learn if you ask me!) If so, endianness is a novel concept to them.

While I personally learned about endianness before writing my first exploit, I've definitely made endianness-related mistakes before.