Go JSON Tricks: JSON Arrays as Go Structs

December 19, 2019

Most JSON you find in the wild uses objects when different types of data are required. A contrived, but believable example:

{
    "status": 404,
    "result": "error",
    "reason": "Not found"
}

But sometimes you’ll find this exact same data expressed in a different way. Particularly if the producer of the JSON is a loosely-typed language such as JavaScript or Python:

[ 404, "error", "Not Found" ]

I could probably write an entire post about why I dislike this, but that’s not what this post is about.

This post is about how to deal with such (arguably backwards) JSON in Go, without going insane.

The problem in Go

In the first example, the normal approach to using this in Go would be quite straight forward:

type Result struct {
    Status int    `json:"status"`
    Result string `json:"result"`
    Reason string `json:"reason"`
}

No further explanation needed.

For the second example, though, we’re a bit stuck. The naïve solution is to use a slice. But we have different types of data in our result, so we’re forced to use a slice of interface{}, which is ugly and annoying.

type Result []interface{}

But this is annoying for a number of reasons, not least of which is the required type assertions to get at the underlying data, not to mention the implicit knowledge that there are 3 elements expected in the slice, and their position matters.

var data = []byte{`[ 404, "error", "Not Found" ]`}
var r Result
if err := json.Unmarshal(data, &r); err != nil {
    log.Fatal(err)
}
// Ugh, so ugly and fragile!
fmt.Println("Status code %d, reason: %s\n", r[0].(float64), r[2].(string))

json.Unmarshaler to the rescue

The solution to this situation is a custom unmarshaler. The standard library’s json package lets us define a method named UnmarshalJSON on any custom type, to handle JSON unmarshaling for us. We still need to unmarshal to a slice, but this becomes an intermediate step. This is similar to the approach I talk about in Go JSON Tricks: “Slightly” Custom Marshaling, but with a completely distinct local type. Let me demonstrate:

type Result struct {
    Status int
    Result string
    Reason string
}

func (r *Result) UnmarshalJSON(p []byte) error {
    var tmp []interface{}
    if err := json.Unmarshal(p, &tmp); err != nil {
        return err
    }
    r.Status = int(tmp[0].(float64))
    r.Result = tmp[1].(string)
    r.Reason = tmp[2].(string)
    return r
}

With this new function in place, it is now possible to unmarshal the JSON array “directly” into a Go struct.

var data = []byte{`[ 404, "error", "Not Found" ]`}
var r Result
if err := json.Unmarshal(data, &r); err != nil {
    log.Fatal(err)
}
// Not ugly! Not fragile!
fmt.Println("Status code %d, reason: %s\n", r.Status, r.Reason)

This is great, but there’s still room for improvement. You may have noticed that we’ve just moved the ugly and fragile code into the custom UnmarshalJSON method. This is still a worthy improvement–at least that special knowledge is now contained in one place, rather than spread around your code base.

But we can do better.

Eliminating type assertions

The first thing I’d like to eliminate is the various type assertions. These are fragile, because if we ever receive an unexpected input, our code is liable to panic, rather than to return a proper error.

One option would be to use the two-value type assertion format, and this would improve the safety of the operation:

    r.Status, ok = int(tmp[0].(float64))
    if !ok {
        return errors.New("not a float64!")
    }

To be sure, this is an improvement, but we can still do much better. I, for one, would rather return a standard JSON error, rather than a custom error of my own invention, in a case like this. This is easily done with just a couple small tweaks.

First, we’ll use the json.RawMessage type in our slice, rather than the empty interface. Second, we’ll unmarshal each element of the slice directly into our target array. This will allow the standard json package to do standard error handling for us.

func (r *Result) UnmarshalJSON(p []byte) error {
    var tmp []json.RawMessage
    if err := json.Unmarshal(p, &tmp); err != nil {
        return err
    }
    if err := json.Unmarshal(tmp[0], &r.Status); err != nil {
        return err
    }
    if err := json.Unmarshal(tmp[1], &r.Result); err != nil {
        return err
    }
    if err := json.Unmarshal(tmp[2], &r.Reason); err != nil {
        return err
    }
    return r
}

It’s a bit longer, and a bit more repetitive, but so much more robust!

Reversing the process

Doing the reverse is, of course, also possible using essentially the same technique. That is, marshaling a Go struct into a JSON array is done by defining a MarshalJSON method. This method is a bit simpler, and doesn’t require nearly as much error checking:

func (r *Result) MarshalJSON() ([]byte, error) {
    return json.MarshalJSON([]interface{}{r.Status, r.Result, r.Reason})
}

The key here is to ensure that our elements are put in the correct order in the slice of interface{} (there’s no reason to use json.RawMessage here).

Arrays of varying length

In some cases, a JSON array may have a varying number of elements. Suppose our result array has only two elements for success:

[ 200, "success" ]

Our current code will still panic with “index out of range” in such a case, when it tries to assign to r.Reason:

    if err := json.Unmarshal(tmp[2], &r.Reason); err != nil {
        return err
    }

It’s easy enough to handle this case:

func (r *Result) UnmarshalJSON(p []byte) error {
    var tmp []json.RawMessage
    if err := json.Unmarshal(p, &tmp); err != nil {
        return err
    }
    if err := json.Unmarshal(tmp[0], &r.Status); err != nil {
        return err
    }
    if err := json.Unmarshal(tmp[1], &r.Result); err != nil {
        return err
    }
    if len(tmp) > 2 {
        if err := json.Unmarshal(tmp[2], &r.Reason); err != nil {
            return err
        }
    }
    return r
}

And don’t forget to produce valid outputs, too:

func (r *Result) MarshalJSON() ([]byte, error) {
    if r.Reason == {
        return json.MarshalJSON([]interface{}{r.Status, r.Result, r.Reason})
    }
    return json.MarshalJSON([]interface{}{r.Status, r.Result})
}

Other variations

You may also have arrays whose order significance changes. Perhaps you’ll get either two elements:

[ 200, "success" ]

Or three, with extra timing information:

[ 0.0987, 200, "success" ]

You’ll want to detect the length of the unmarshaled intermediate slice, and if it is 3, put the first value into r.Timing (for example). I leave this exact exercise for the reader. But I’m sure you get the idea by now.

Conclusion

Go’s JSON support is very powerful, and with a little cleverness, a lot of flexibility is possible.

If you’re facing a Go JSON challenge, I’d love to hear from you in comments, and I’ll try to follow-up with a similar trick for your situation.


comments powered by Disqus