How to Decode Kafka Messages: JSON, Base64, and the Avro Wire Format
You run kafka-console-consumer, and instead of JSON you get something like \x00\x00\x00\x01\x86H\x02.... The message is not corrupted — it is almost certainly binary-serialized, and the first bytes tell you exactly how.
Step 1: recognize the format from the first bytes
Starts with a zero byte (\x00)? That is the Confluent Schema Registry wire format: one magic byte (always 0), a 4-byte big-endian schema ID, then the Avro (or Protobuf/JSON Schema) encoded payload.
byte 0 : magic byte (0x00)
bytes 1-4 : schema ID (big-endian int)
bytes 5... : Avro binary payloadStarts with {?Plain JSON — your consumer's deserializer is the problem, not the payload. Looks like eyJ...? That is Base64-encoded JSON (common when a producer double-encodes). Decode the Base64 first.
Step 2: extract the schema ID
With the wire format identified, the schema ID is bytes 1–4:
# Python
raw = message_bytes
assert raw[0] == 0, "not Confluent wire format"
schema_id = int.from_bytes(raw[1:5], "big")
# Then fetch the writer schema:
# GET http://schema-registry:8081/schemas/ids/<schema_id>A surprising number of "deserialization failure" incidents end here: the consumer was configured for the wrong subject, and the schema ID in the message proves it.
Step 3: decode the payload
With the writer schema in hand, any Avro library can decode the remaining bytes. If you just need to see the payload structure quickly — schema ID, field names, values — paste the raw message into the Hexabench Kafka Message Decoder. It detects JSON, Base64, hex, and the Confluent wire format, extracts the schema ID, and shows the payload structure — locally in your browser, which matters because Kafka messages routinely contain production data.
Common decoding failures and what they mean
"Unknown magic byte" — the consumer expected wire format but the message is plain JSON (or vice versa). Usually one producer among many is misconfigured; check by partition and producer.
Payload decodes but fields are shifted/wrong — decoding with the wrong schema version. Avro requires the writer's schema; the reader schema can differ only within compatibility rules.
Message is valid Base64 but decodes to binary — likely Avro inside Base64 (common when messages transit HTTP). Decode Base64 first, then apply the wire-format steps above.