Go JSON Tricks: The Self-Referencing Marshaler

December 12, 2020

For more content like this, buy my in-progress eBook, Data Serialization in Go⁠, and get updates immediately as they are added!

The content in this post is included in my in-progress eBook, Data Serialization in Go, available on LeanPub.

I’ve done a lot of JSON handling in Go. In the process, I’ve learned a number of tricks to solve specific problems. But one pattern in particular I find myself repeating ad infinitum. It’s evolved over the years, and now I want to share it, in hopes that others may benefit. Or perhaps you can tell me that I’m a moron for doing things this way, and then I can learn from your superior experience!

The problem

The general problem I aim to solve here, is a struct (or any data type, really), that implements a custom MarshalJSON method, but that method needs to access the non-custom JSON representation of itself. (The same general problem exists for UnmarshalJSON, too, but I’ll focus on just marshaling for demonstration purposes.)

But why?

Why would you ever want this? If you’ve not run into the problem, it may seem like insanity. If you have run into this problem, you’ve probably pulled out a little hair trying to find a good solution (I know I would have, if I had any hair to begin with…)

There are many reasons you may want to do this, but a few off the top of my (bald) head:

  • You want to special-case some value (such as an empty struct) to a specific output (such as null)
  • You need to add to, or modify the output before returning it
  • You need to unmarshal the data more than once (obviously only applies to the UnmarshalJSON case)

To demonstrate, let me use a simple example: That of rendering an empty struct as null.

Default behavior

To demonstrate, let’s imagine a data type that contains a blog post ID, and a list of tags. Not all blog posts have tags, so sometimes this struct should marshal as null.

type Metadata struct {
	ID   string   `json:"id"`
	Tags []string `json:"tags"`
}

If we marshal an empty instance of this with the default behavior, we get the following output:

{"id":"","tags":null}

A first attempt

To achieve our stated goal of producing null for an empty value, we need a custom marshaler:

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}
	// TODO
	return nil, nil
}

Now an empty instances of Metadata marshals as we desire:

null

However, we still need to solve the general case of marshaling the non-empty case. The naïve approach is simply to add a call to json.Marshal:

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}
	return json.Marshal(m)
}

If the problem with this approach isn’t obvious, it will be as soon as you run this code, and discover the infinite loop caused by MarshalJSON (indirectly) calling itself.

Breaking the cycle

To break this infinite loop, we need to call json.Marshal on a different type–one that doesn’t have the same MarshalJSON method. This is easy enough to accomplish, by defining a copy of the type. We could declare the copy at the package level, but I like to keep my package-level symbols as tidy as possible, and since this type should only ever be used in the MarshalJSON method, I like to define it within the method itself.

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}

	type metadataCopy struct {
		ID   string   `json:"id"`
		Tags []string `json:"tags"`
	}

	myCopy := metadataCopy{
		ID: m.ID,
		Tags: m.Tags,
	}

	return json.Marshal(myCopy)
}

This works nicely! But it’s very verbose. Especially for a struct with many fields. We can do better.

If our original type, and the local “copy” have the same fields and types, we can do a simple conversion, rather than assigning each field individually:

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}
	
	type metadataCopy struct {
		ID   string   `json:"id"`
		Tags []string `json:"tags"`
	}
	
	myCopy := metadataCopy(m)
	
	return json.Marshal(myCopy)
}

More improvements

Now we’re getting somewhere. But we can still do better. There’s actually no need to list all of the fields in our definition of metadataCopy, if we use Metadata as the TypeSpec in our type declaration:

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}
	
	type metadataCopy Metadata
	
	myCopy := metadataCopy(m)
	
	return json.Marshal(myCopy)
}

This version is not only shorter, it’s also much more future-proof, as now any change made to the Metadata type will automatically be supported by our MarshalJSON method.

Note that this is not an alias declaration. A type alias would not serve our purpose, as it’s just a new name for an existing type, meaning the infinite loop would still be a problem.

A final revision

With one final edit to shorten things a bit, and add some test cases, we have a final version (See it on the Go Playground)

package main

import (
	"encoding/json"
	"fmt"
)

type Metadata struct {
	ID   string   `json:"id"`
	Tags []string `json:"tags"`
}

func (m Metadata) MarshalJSON() ([]byte, error) {
	if m.ID == "" && len(m.Tags) == 0 {
		return []byte("null"), nil
	}

	type metadataCopy Metadata

	return json.Marshal(metadataCopy(m))
}

func main() {
	empty := Metadata{}
	emptyMarshaled, err := json.Marshal(empty)
	if err != nil {
		panic(err)
	}
	fmt.Println("empty:", string(emptyMarshaled))

	populated := Metadata{ID: "abc", Tags: []string{"def", "hij"}}
	populatedMarshaled, err := json.Marshal(populated)
	if err != nil {
		panic(err)
	}
	fmt.Println("populated:", string(populatedMarshaled))
}

The output produced is:

empty: null
populated: {"id":"abc","tags":["def","hij"]}

A json.Marshaler example

For the sake of completeness, let me briefly show how to do the reverse with a self-referencing json.Unmarshaler implementation on the same type:

func (m *Metadata) UnmarshalJSON(p []byte) error {
	if string(p) == "null" {
		*m = Metadata{}
		return nil
	}

	type metadataCopy Metadata
	
	var result metadataCopy
	err := json.Unmarshal(p, &result)
	*m = Metadata(result)
	return err
}

Conclusion

This basic pattern can be extended and adapted to a large variety of situations. I’ll be writing about more of these here, so be sure to subscribe to be notified of new posts.

Questions?

Let me know what challenges you face with JSON in Go in the comments section, and I’ll try to answer in an upcoming post.


comments powered by Disqus