Ordering abstract JSON in Go

In this article I am sharing the way I extended OrderedMap lib to handle arbitrary JSON to preserve the original structure and order.

Huge bookshelf next to the window full of books and newspapers.

Earlier this year I have created and published go-omap library. It is meant to create an abstraction over maps and slices in Go to keep read and write fast, at the same time preserving the specific order of the keys. I have shared more details on it in this article:

I’ve been facing use cases during creation of API services, where response was expected to be ordered. It isn’t hard to keep a second array (slice) to keep all the keys ordered, using array for iterating. But I made an extra step to create a solution once, to re-use it.

New problem

I found out I need to create more and more submit forms. There are many services and integration that enables accepting a bunch of fields and store it somewhere. But I’m developer myself, I love writing code, why wouldn’t I deep dive into relevant problem?

The problem is: imaging you got HTML form with arbitrary fields. It means key can be anything, these might change, added or removed from the set. And I wanted all of it to be handled by Go server.

In Go JSON is literally a slice of bytes. Even unprocessed JSON.RawMessage is wrapper type over []byte. There are just a couple ways of converting it to something more useful: decoding it into a struct or into a map[string]any. To solve this problem struct won’t work well, due to the unstructured (dynamic) nature of the payload.

One more requirement, that makes it relevant to go-omap library: it would be more convenient to handle data in the specific order. And the most straightforward solution would be to preserve the order of the fields. However, as soon as JSON is decoded to a map it is not possible to preserve the order of the keys.

Custom decoder

This is the place where json.Decoder become extremely useful. The basic functionality of the decoder is to get values one by one read from the stream. That’s the most of it.

Next step is to decide what type of value is that. For JSON there are 3 major types: object, array and value. Value being the simplest case: it’s either key (string) or any plain value, like string, number, bool.

Objects always start with { so it’s easy to deal with them recursively, because entire JSON is an object. Get inside it, parse the tokens and see what’s the value. Based on the value - decide how to process it.

Arrays don’t differ much, being a collection of plain values. But that’s important to remember it might go deeper than 1 level.

Here is the complete logic to make custom structure JSON decodable: https://github.com/eli-l/go-omap/blob/master/decode.go

Putting it all together

With the json.Parser and custom decoder I could easily swap default map[string]any with OrderedMap[string]any and use proper API methods. This builds proper OrderedMap, with keys being stored into all the underlying structures.

Further usage of OrderedMap’s iterator grants ordered access to the underlying data. As well as it provides additional methods to sort based on the key value or key-value pair, as described.

Definitely I wouldn’t deep dive into such a solution from the ground, but since I had OrderedMap already published I naturally followed the curiosity to learn more about parsers and usage of custom structs for working with JSON.