How do you inspect Avro/Protobuf records during debugging?

Question

Not a data engineer, but I’ve worked with Avro a tiny bit and it quickly became obvious that manually inspecting payloads would not be quick and easy w/o some custom tooling. I'm curious how DEs actually do this in the real world? For instance, say you’ve got an Avro or Protobuf payload and you’re not sure which schema version it used, how do you inspect the actual record data?

HackerNews · Accepted Answer

Inspecting Avro or Protobuf records during debugging can be challenging without the right tools. Experienced data engineers often use a combination of command-line tools and custom scripts to inspect payloads. For Avro, avro-tools is a popular choice, allowing users to dump and inspect Avro files. Similarly, for Protobuf, protoc is used to decode messages. Many teams also develop internal tools or scripts tailored to their specific use cases and schema versions.

How do you inspect Avro/Protobuf records during debugging?

Resources