Find bugs caused by unexpected input in Go with Fuzzing

2023-12-17 golang

Fuzzing is a test method that uses random data as input, and is supported since Go 1.18.

Let’s test the following FizzBuzz function.

func FizzBuzz(n uint64) string {
	if n%3 == 0 {
		if n%5 == 0 {
			return "FizzBuzz"
		}
		return "Fizz"
	}

	if n%5 == 0 {
		return "Buzz"
	}

	return strconv.Itoa(int(n))
}

First, we do a table-driven test.

func TestFizzBuzz(t *testing.T) {
	testcases := []struct {
		title string
		in    uint64
		out   string
	}{
		{"A multiple of 3 returns Fizz", 3, "Fizz"},
		{"A multiple of 5 returns Buzz", 5, "Buzz"},
		{"A multiple of 3 and 5 returns FizzBuzz", 15, "FizzBuzz"},
		{"Otherwise returns number string", 1, "1"},
	}

	for _, testcase := range testcases {
		t.Run(testcase.title, func(t *testing.T) {
			ret := FizzBuzz(testcase.in)
			if ret != testcase.out {
				t.Errorf("FizzBuzz(%d) = %s, want %s", testcase.in, ret, testcase.out)
			}
		})
	}
}

When we run it, all test cases pass, so is it okay to say that there is no problem? In fact, this function has a bug.

$ go test
PASS
ok      github.com/sambaiz/gofuzztest   0.093s

In the first place, bugs caused by unexpected inputs are often omitted from test cases, and if there are many input patterns, it may not be possible to cover them completely. In such cases, Fuzzing may help to find bugs. Besides, it does not support types such as map and struct yet.

func FuzzFizzBuzz(f *testing.F) {
	// f.Add(9223372036854775807)
	f.Fuzz(func(t *testing.T, n uint64) {
		ret := FizzBuzz(n)
		if ret != strconv.FormatUint(n, 10) &&
			ret != "Fizz" &&
			ret != "Buzz" &&
			ret != "FizzBuzz" {
			t.Errorf("FizzBuzz(%d) = %s, want %s, Fizz, Buzz, or FizzBuzz",
				n, ret, strconv.FormatUint(n, 10))
		}
	})
}

When we run it, it starts searching for interesting inputs that increase the coverage. As a result, we found that unexpected values are returned when the input is very large.

$ go test -fuzz=FuzzFizzBuzz -fuzztime 10s
fuzz: elapsed: 0s, gathering baseline coverage: 0/8 completed
fuzz: elapsed: 0s, gathering baseline coverage: 8/8 completed, now fuzzing with 8 workers
fuzz: elapsed: 0s, execs: 16 (143/sec), new interesting: 0 (total: 8)
--- FAIL: FuzzFizzBuzz (0.11s)
    --- FAIL: FuzzFizzBuzz (0.00s)
        main_test.go:39: FizzBuzz(9223372036854775873) = -9223372036854775743, want 9223372036854775873, Fizz, Buzz, or FizzBuzz
    
    Failing input written to testdata/fuzz/FuzzFizzBuzz/b0764ff8575cbc59
    To re-run:
    go test -run=FuzzFizzBuzz/b0764ff8575cbc59
FAIL
exit status 1
FAIL    github.com/sambaiz/gofuzztest   0.323s

This is caused by converting uint64 to int(64). After fixing it, we can test it again with the corpus file of the failed input.

$ cat testdata/fuzz/FuzzFizzBuzz/b0764ff8575cbc59
go test fuzz v1
uint64(9223372036854775873)

$ go test -run=FuzzFizzBuzz/b0764ff8575cbc59

Corpus files like this and f.Add() provides seed corpus, and they can be used to increase the baseline coverage and make the search more efficient.