Reverse engineering the storage format for an undocumented database

during developing Kafka search tool, it have to support many kind of decode formats ( such as Avro + schemaRegistry, protobuf, ConnectJson(json with schema), and so on ). it recommend a type of deserializer by looking at the leading bytes. After user select final decision, it is ok if error occurs during first data deserialize, But many case it’s not occurred error, just pass through with plausibly wrong parsed data. So my solution was display 10 pre decoded sample to user, and make user select one deserializer by eye.

ConnectJson is reliable, because it has data and schema embedded in every message ( but large ). In my case using schema-registry, server1 has schema no 10, and server2 has schema no 10 but it’s different. I make a mistake select wrong server 2, data parsing is progressed but plausibly wrong data It took me a long time to find and correct some thing wrong.