ezpkg.io/conveyz: Understanding the Implementation of FConvey

2024-12-20

· go

The FConvey function is a standout feature of ezpkg.io/conveyz, designed to simplify focused testing in Go. Unlike the original smartystreets/goconvey framework, which requires developers to set FocusConvey at every nested block level, FConvey is smarter. It ensures that focusing on a single block propagates to all relevant nested and parent blocks, streamlining the debugging process.

In this article, we dive deep into the implementation of FConvey, why the original FocusConvey does require manual setting at every level, and how FConvey solves the problem.

The motivation behind ezpkg.io/conveyz

Before we dive into the implementation details, let’s first understand the motivation behind the creation of ezpkg.io/conveyz.

Ginkgo and Gomega are great, but verbose

At work, we use gomega and ginkgo for writing behaviour tests. They are great, but sometimes too verbose. We have to write a lot of boilerplate code to set up the test environment, and it’s hard to focus on the test itself.

Each package has a suite_test.go file to set up the test environment, with RegisterFailHandler(Fail) and RunSpecs().
Each file starts with Describe block then var block, then Context blocks, then more var blocks, then more Context blocks, with BeforeEach and AfterEach blocks inside, then end up with a lot of It blocks at the leaf.
Lists of variables are declared at the beginning of the Describe and Context blocks, then initialized in BeforeEach blocks, before being used in It blocks, or JustBeforeEach blocks.
And even JustBeforeEach blocks to keep the execution logic separated.
So the execution is outer blocks to inner blocks: Describe then BeforeEach then Context then BeforeEach then It.
Oh, no! It needs to jump to JustBeforeEach before It. Finally, AfterEach at the last.
And we have to repeat the same setup for each test file.

Well, you got it!

An example of profile_suite_test.go:

package profile_test

import (
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
)

func TestProfile(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Profile Suite")
}

and user_test.go:

package user_test

import (
    "context"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "go.uber.org/mock/gomock"

    profilepb "connectly.ai/go/pb/profile"
    "connectly.ai/go/services/profile"
    "connectly.ai/go/services/profile/mocks"
)

var _ = Describe("UserService", func() {
    var (
        ctx      = context.Background()
        ctrl     *gomock.Controller
        mockRepo *mocks.MockProfileRepository
        service  profilepb.UserServiceServer
    )
    BeforeEach(func() {
        var err error
        ctrl = gomock.NewController(GinkgoT())
        mockRepo = mocks.NewMockProfileRepository(ctrl)
        service = profile.NewUserService(mockRepo)
    })
    AfterEach(func() {
        ctrl.Finish()
    })
    Describe("SignUp", func() {
        var (
            req *profilepb.SignUpRequest
            res *profilepb.SignUpResponse
            err error
        )
        JustBeforeEach(func() {
            res, err = service.SignUp(username, password)
        })
        BeforeEach(func() {
            req = &profilepb.SignUpRequest{
                Username: "admin",
                Password: "password123",
            }
        })
        Context("valid username & password", func() {
            BeforeEach(func() {
                mockRepo.EXPECT().
                    CreateUser(ctx, req.Username, req.Password).
                    Return(/*...*/)
            })
            It("should success", func() {
                Ω(err).ToNot(HaveOccurred())
                Ω(res.UserID).ToNot(BeEmpty())
            })
        })
        Context("invalid username", func() {
            BeforeEach(func() {
                req.Password = "password123"
            })
            It("should error", func() {
                Ω(err).To(HaveOccurred())
                Ω(err.Error()).To(ContainSubstring("invalid username"))
            })
        })
        Context("invalid password: too short", func() {
            BeforeEach(func() {
                req.Password = "****"
            })
            It("should error", func() {
                Ω(err).To(HaveOccurred())
                Ω(err.Error()).To(ContainSubstring("password too short"))
            })
        })
    })
})

Convey: Focus on the test, not the setup

Compare the above code with the following from smartystreets/goconvey. It’s much simpler and easier to focus on the test itself.

No need to set up the test environment in a separate file.
You only need to use Convey and So to write the test.
The execution is straightforward: Convey then Convey then Convey then defer().
Each run, it executes from the root block to each leaf block:
UserService → SignUp → valid username & password → should success → defer()
UserService → SignUp → invalid username → should error → defer()

package profile_test

import (
    "context"
    "testing"

    . "github.com/smartystreets/goconvey/convey"
    "go.uber.org/mock/gomock"

    profilepb "connectly.ai/go/pb/profile"
    "connectly.ai/go/services/profile"
    "connectly.ai/go/services/profile/mocks"
)

func TestUserSignUp(t *testing.T) {
    Convey("UserService", t, func() {
        ctx      := context.Background()
        ctrl     := gomock.NewController(t)
        mockRepo := mocks.NewMockProfileRepository(ctrl)
        service  := profile.NewSignUpService(mockRepo)
        defer ctrl.Finish()

        Convey("SignUp", func() {
            req = &profilepb.SignUpRequest{
                Username: "admin",
                Password: "password123",
            }
            Convey("valid username & password", func() {
                mockRepo.EXPECT().
                    CreateUser(ctx, req.Username, req.Password).
                    Return(/*...*/)
                
                res, err = service.SignUp(ctx, req)

                Convey("should success", func() {
                    So(err, ShouldBeNil)
                    So(res.UserID, ShouldNotBeEmpty)
                })
            })
            Convey("invalid username", func() {
                req.Username = "admin@"
                
                res, err = service.SignUp(ctx, req)

                Convey("should error", func() {
                    So(err, ShouldNotBeNil)
                    So(err.Error(), ShouldContainSubstring, "invalid username")
                })
            })
            Convey("invalid password: too short", func() {
                req.Password = "****"
                
                res, err = service.SignUp(ctx, req)

                Convey("should error", func() {
                    So(err, ShouldNotBeNil)
                    So(err.Error(), ShouldContainSubstring, "invalid password")
                })
            })
        })
    })
}

Problems: Incompatible assertions and repeated `FocusConvey`

While the code looks cleaner, there are still some problems:

With ginkgo, you can focus on a single block by setting FIt or FContext. Basically, add a single F and it will only run that block (and children), skipping others.
But in goconvey, you have to set FocusConvey at every level. It’s not smart enough to propagate the focus to all relevant blocks.
And the assertions are not compatible. You have to change Ω to So, To to Should, ToNot to ShouldNot, ToNotBeEmpty to ShouldNotBeEmpty, To(HaveOccurred()) to ShouldNotBeNil, To(ContainSubstring("...")) to ShouldContainSubstring("..."), and so on.
We don’t want to maintain two sets of assertions, and it’s hard to convince the team to switch to goconvey or even doing the rewrite.

Meet ezpkg.io/conveyz

So, I decided to create a new package, ezpkg.io/conveyz, to solve the above problems. It’s a wrapper around goconvey, with the following features:

Ω to replace So and To in goconvey, so you can use the same assertions as gomega.
FConvey to focus on a single block and propagate to all relevant blocks while skipping others.

The above code can be rewritten as:

package profile_test

import (
    "context"
    "testing"

    . "ezpkg.io/conveyz"
    "go.uber.org/mock/gomock"

    profilepb "connectly.ai/go/pb/profile"
    "connectly.ai/go/services/profile"
    "connectly.ai/go/services/profile/mocks"
)

func TestUserSignUp(t *testing.T) {
    Convey("UserService", t, func() {
        ctx := context.Background()
        ctrl := gomock.NewController(t)
        mockRepo := mocks.NewMockProfileRepository(ctrl)
        service := profile.NewSignUpService(mockRepo)
        defer ctrl.Finish()

        Convey("SignUp", func() {
            req = &profilepb.SignUpRequest{
                Username: "admin",
                Password: "password123",
            }
            Convey("valid username & password", func() {
                mockRepo.EXPECT().
                    CreateUser(ctx, req.Username, req.Password).
                    Return( /*...*/ )

                res, err = service.SignUp(ctx, req)

                Convey("should success", func() {
                    Ω(err).ToNot(HaveOccurred)
                    Ω(res.UserID).ToNot(BeEmpty)
                })
            })
            Convey("invalid username", func() {
                req.Username = "admin@"
                res, err = service.SignUp(ctx, req)

                Convey("should error", func() {
                    Ω(err).To(HaveOccurred())
                    Ω(err.Error()).To(ContainSubstring, "invalid username")
                })
            })
            Convey("invalid password: too short", func() {
                req.Password = "****"
                res, err = service.SignUp(ctx, req)

                Convey("should error", func() {
                    Ω(err).To(HaveOccurred())
                    Ω(err.Error()).To(ContainSubstring("password too short"))
                })
            })
        })
    })
}

Solution: Make “gomega” `Ω` work with “goconvey”

We can make Ω work with goconvey by creating an adapter that implements gomega.Assertion:

The adapter will call convey.So() with the actual value and the matcher.
It returns empty string if the matcher passes, or the error message if the matcher fails.

package conveyz

import (
    "github.com/onsi/gomega"
    gomegatypes "github.com/onsi/gomega/types"
    "github.com/smartystreets/goconvey/convey"
)

// Ω is a reimplemented version of gomega.Expect to call goconvey.So under the hood.
func Ω(actual any, extra ...any) gomega.Assertion {
    assertion := gomega.Expect(actual, extra...)
    return gomegaAssertion{actual: actual, assertion: assertion}
}
type gomegaAssertion struct {
    actual    any
    offset    int
    assertion gomega.Assertion
}
func (a gomegaAssertion) To(matcher gomegatypes.GomegaMatcher, optionalDescription ...any) bool {
    convey.So(a.actual, func(_ any, _ ...any) string {
        success, err := matcher.Match(a.actual)
        if err != nil {
            stack := stacktracez.StackTraceSkip(4)
            return formatMsg(optionalDescription, stack, "%vUNEXPECTED: %v%v", colorz.Red, err, colorz.Yellow)
        }
        if !success {
            stack := stacktracez.StackTraceSkip(4)
            msg := matcher.FailureMessage(a.actual)
            return formatMsg(optionalDescription, stack, "%s", msg)
        }
        return ""
    })
    return true
}
func (a gomegaAssertion) ToNot(matcher gomegatypes.GomegaMatcher, optionalDescription ...any) bool {
    convey.So(a.actual, func(_ any, _ ...any) string {
        success, err := matcher.Match(a.actual)
        if err != nil {
            stack := stacktracez.StackTraceSkip(4)
            return formatMsg(optionalDescription, stack, "%vUNEXPECTED: %v%v", colorz.Red, err, colorz.Yellow)
        }
        if success {
            stack := stacktracez.StackTraceSkip(4)
            msg := matcher.NegatedFailureMessage(a.actual)
            return formatMsg(optionalDescription, stack, "%s", msg)
        }
        return ""
    })
    return true
}

When failure, the output will look like this, with a nice error message and a focused stacktrace:

Failures:

  * /Users/i/conn/go/services/profile/user_test.go
  Line 160:
  Expected
      <bool>: true
  to be false

  go/sdks/errors/api/error_test.go:160 · api.TestUser.func1.2.1.1
  go/sdks/errors/api/error_test.go:155 · api.TestUser.func1.2.1
  go/sdks/errors/api/error_test.go:94 · api.TestUser.func1.2
  go/sdks/errors/api/error_test.go:63 · api.TestUser.func1
  go/sdks/errors/api/error_test.go:36 · api.TestUser


120 total assertions

--- FAIL: TestUser (0.01s)

And to make it easier for our engineers, we can re-export all the assertions from gomega:

func BeNil() types.GomegaMatcher {
    return gomega.BeNil()
}
func BeTrue() types.GomegaMatcher {
    return gomega.BeTrue()
}
func BeFalse() types.GomegaMatcher {
    return gomega.BeFalse()
}

So to migrate tests from goconvey to ezpkg.io/conveyz, you don’t need to change the assertions, just do some cleanup:

Replace the gomega/ginkgo imports with . "ezpkg.io/conveyz".
Replace all Describe|Context|It with Convey.
Remove BeforeEach and merge it with the var block to simplify the setup code.
Replace AfterEach with idiomatic Go defer.
For JustBeforeEach, you can simply inline the code right before the assertion. If it needs more logic, you can extract to a function and call it right before the assertion.

This way, we can enjoy the best of both worlds: the simplicity of goconvey with the power of gomega!

But there is still one problem: to focus on a single Convey block, you have to set FocusConvey at every level, from the outermost Convey to the innermost Convey. Then delete each one after finishing your tests… 😮‍💨

Solution: Make `FConvey` smarter

To understand how “ezpkg.io/conveyz” FConvey works, let’s first look at how “ginkgo” FIt works and why “goconvey” FocusConvey requires manual setting at every level.

ginkgo: How a single `FIt` can focus on a test

In ginkgo, inside each Describe|Context block, you can set FIt or FContext to focus on that block.

For example, with the following code: When you set FIt("should error (invalid username)"), it will only exec that test and skip others.

Specifically, it execs in this order (🟢 for exec and 🟡 for skip):

Start with 🟢 Describe("UserService") → 🟢 BeforeEach → 🟢 Context("SignUp") → 🟢 BeforeEach
→ 🟡 skip Context("valid username & password") → 🟡 skip BeforeEach inside → 🟡 skip all Its inside
→ 🟢 exec Context("invalid username") → 🟢 BeforeEach
- → 🟡 skip It("should record failure")
- → ✅ exec FIt("should error (invalid username)")
→ 🟡 skip all remaining non-focus Contexts

It can be able to do that, because when run the Context("valid username & password"), it calls BeforeEach, but that BeforeEach does not actually run any logic inside - it only registers the logic to execute later.
Then there are 2 scenarios:

If ginkgo does not encounter any FIt (or PIt for pending),
it will 🟢 execute all the registered logic.
If ginkgo encounters an FIt, it will only 🟢 execute that It (and all logic to that path),
but 🟡 skip all other registered logic outside the execution path.

// 🟢 start with Describe
Describe("UserService", func() {
    // 🟢 execute this BeforeEach block
    BeforeEach(func() {
        ctrl = gomock.NewController(GinkgoT())
        mockRepo = mocks.NewMockProfileRepository(ctrl)
    })
    // 🟢 execute this Context
    Context("SignUp", func() {
        var req *SignUpRequest
        // 🟢 execute this BeforeEach block
        BeforeEach(func() {
            req = &SignUpRequest{
                Username: "admin",
                Password: "password123",
            }
        })
        // 🟡 all non-focused Context are skipped
        Context("valid username & password", func() {
            // 🟡 skip this BeforeEach block
            BeforeEach(func() {
                mockRepo.EXPECT().
                    CreateUser(ctx, req.Username, req.Password).
                    Return(/*...*/)
            })
            // 🟡 skip this It
            It("should success", func() {
                // ...
            })
        })
        Context("invalid username", func() {
            // 🟢 execute this BeforeEach block
            BeforeEach(func() {
                req.Username = "admin@"
            })
            // 🟡 skip this It
            It("should record failure", func() {
                // ...
            })
            // ✅👉 put FIt here to focus on the test 👈
            FIt("should error (invalid username)", func() {
                // ...    
            })
        })
        // 🟡 all non-focused Context are skipped
        Context("invalid password: too short", func() {
            BeforeEach(func() {
                req.Password = "****"
            })
            It("should error (password too short)", func() {
                // ...    
            })
        })
    })
    // 🟡 all non-focused Context are skipped
    Context("Login", func() {
        Context("valid login", func() {
            // ...
        })
    })
})

But “goconvey” `FocusConvey` must be set at every level

Because if you only set FocusConvey at the leaf level without setting it at the parent level, it will not skip the parent level.

When it runs a parent Convey block, it will run all the code inside, including the before-each logic (the BeforeEach blocks in previous “ginkgo” example).
Unknown “ginkgo”, which registers the before-each logic inside BeforeEach blocks without actually execute it, “goconvey” does not have that privilege:
It has to run the Convey block to know if there is any FocusConvey inside!

So you get idea: No way to know if there is any FocusConvey inside without actually running the Convey block, including the before-each logic!

// 🟢 start with Convey
Convey("UserService", func() {
    // 🟢 execute this "BeforeEach" block
    ctrl = gomock.NewController(GinkgoT())
    mockRepo = mocks.NewMockProfileRepository(ctrl)
    
    // 🟢 execute this Context
    Convey("SignUp", func() {
        // 🟢 execute this BeforeEach block
        req := &SignUpRequest{
            Username: "admin",
            Password: "password123",
        }
        
        // 🟡 all non-focused Convey are skipped
        Convey("valid username & password", func() {
            // 🟡 skip this "BeforeEach" block
            // ❌ but how? goconvey does not have BeforeEach to 
            //    register the logic beforehand without actually execute it ❌
            // ❌ the Convey still needs to run to know if there is any FocusConvey inside ❌
            mockRepo.EXPECT().
                CreateUser(ctx, req.Username, req.Password).
                Return(/*...*/)
            
            // 🟡 skip this Convey
            Convey("should success", func() {
                // ...
            })
        })
        Convey("invalid username", func() {
            // 🟢 run this "BeforeEach" block
            req.Username = "admin@"
            
            // 🟡 skip this Convey
            // ❌ but how? it still needs to run it to know if there is any FocusConvey inside ❌
            Convey("should record failure", func() {
                // ...
            })
            // ✅👉 put FocusConvey here to focus on the test 👈
            FocusConvey("should error (invalid username)", func() {
                // ...    
            })
        })
        // 🟡 all non-focused Convey are skipped
        Convey("invalid password: too short", func() {
            req.Password = "****"
            
            Convey("should error (password too short)", func() {
                // ...    
            })
        })
    })
    // 🟡 all non-focused Convey are skipped
    Convey("Login", func() {
        Convey("valid login", func() {
            // ...
        })
    })
})

ezpkg.io/conveyz: Set a single `FConvey` to focus on a test

Okay, now we know why “goconvey” FocusConvey requires manual setting at every level. But how does “ezpkg.io/conveyz” FConvey workaround that limitation? What’s the magic behind it?

Turns out, it’s quite simple: Instead of running the Convey blocks, we can first read the source code and find all the FConvey (or SConvey) inside! When we know locations of all the Convey blocks, we can set the execution focus only on the path to the FConvey block, and skip all others. ✅

Now you know the trick, it’s time to see the implementation of FConvey in “ezpkg.io/conveyz”:

Read the source code for the current package

The first time any Convey|FConvey|SkipConvey, we need to initialize everything: read the source code and find all FConvey.

To read the source code, we can use runtime.Caller() to get the current file path:

_, currentFile, _, ok := runtime.Caller(2)

Then read all the test files and store in to a fileMap:

var fileMap = map[string]*testFile{}

func parseDir(dir string) (detected bool) {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return false
    }
    for _, entry := range entries {
        if entry.IsDir() {
            continue
        }
        if !strings.HasSuffix(entry.Name(), "_test.go") {
            continue
        }
        if strings.IndexAny(entry.Name(), "_.") == 0 {
            continue // files start with "_" or "." are ignored
        }
        path := filepath.Join(dir, entry.Name())
        data := must(os.ReadFile(path))
        file, _detected := parseFile(path, data)
        fileMap[path] = file
        detected = detected || _detected
    }
    return detected
}

Find any FConvey inside the source code

To find any FConvey inside the source code, we can simply read the file line by line and check if it contains FConvey( or FocusConvey(:

func parseFile(path, data string) (_ *testFile, detected bool) {
    tf := &testFile{Path: path}
    for _, line := range strings.Split(data, "\n") {
        tl := testLine{
            Indent: countTabs(line),
            Text:   line,
        }
        if strings.Contains(line, "FConvey(") || strings.Contains(line, "FocusConvey(") {
            detected = true
            tl.FConvey = true
        } else if strings.Contains(line, "Convey(") {
            tl.Convey = true
        }
        tf.Lines = append(tf.Lines, tl)
    }
    return tf, detected
}

Convert each Convey to FocusConvey or SkipConvey

The case for no FConvey in all files is trivial: no need to do anything.

But if there is any FConvey in any file, we need to convert the Convey block to FocusConvey or SkipConvey to only execute the relevant blocks on the focus path and skip others.

func Convey(items ...any) {
    defer setupGomega(items...)()
    if !pkgHasFocusConvey() {
        convey.Convey(items...)
        return
    }
    switch shouldConvert() {
    case 1:
        convey.FocusConvey(items...)
    case -1:
        convey.SkipConvey(items...)
    default:
        convey.Convey(items...)
    }
}

Implement shouldConvert()

Now we have all the FConvey locations, let’s define some rules to know which Convey should be executed and which should be skipped:

Case 1: If there is no FConvey in the current file: it means the focused block is in another file, we can simply return -1 to convert the whole Convey block to SkipConvey.

Case 2: If there is any FConvey in the current file: we need to check if any parent or child block has FocusConvey (or FConvey), and convert the Convey block to FocusConvey if it has.

Convey        -> 🟢 convert to FocusConvey (parent)
  Convey      -> leave it as is
    Convey    -> leave it as is
  FConvey     -> ✅ convert to FocusConvey (.)
    Convey    -> 🟢 convert to FocusConvey (child)
  Convey      -> leave it as is
    Convey    -> leave it as is

The final code looks like this:

func shouldConvert() (out int) {
    _, file, lineNumber, ok := runtime.Caller(2)
    if !ok {
        return 0
    }
    tf := fileMap[file]
    if tf == nil {
        return 0 // leave it as is
    }

    lineIndex := lineNumber - 1
    {
        // CASE 2: detect FocusConvey in child or parent blocks
        // -> return 1: convert the Convey block to FocusConvey
        line := tf.Lines[lineIndex]
        // check if the child code lines inside this convey block has FocusConvey (or FConvey)
        for i := lineNumber; i < len(tf.Lines); i++ {
            if tf.Lines[i].Indent <= line.Indent && tf.Lines[i].Indent != 0 /* skip empty lines */ {    
                break
            }
            if tf.Lines[i].FConvey {
                return 1 // convert to FocusConvey
            }
        }
        // check if the parent code lines of this convey block has FocusConvey (or FConvey)
        for parent := getParent(tf, lineIndex); parent > 0; parent = getParent(tf, parent) {
            if tf.Lines[parent].FConvey {
                return 1 // convert to FocusConvey
            }
        }
    }
    {
        // CASE 1: detect if there is no FConvey or FocusConvey in the whole file
        // -> return -1: convert the Convey block to SkipConvey
        detected := false
        for _, line := range tf.Lines {
            if line.FConvey {
                detected = true
                break
            }
        }
        if !detected {
            return -1 // convert to SkipConvey
        }
    }
    return 0
}

func getParent(tf *testFile, lineIndex int) int {
    line := tf.Lines[lineIndex]
    for i := lineIndex - 1; i >= 0; i-- {
        if tf.Lines[i].Indent == 0 {
            // skip empty lines
            continue
        }
        if tf.Lines[i].Indent >= line.Indent {
            // skip lines with same or greater indent
            continue
        }
        if tf.Lines[i].Indent < line.Indent {
            // found the parent
            return i
        }
    }
    return 0
}

And that’s it! Now you know how “ezpkg.io/conveyz” FConvey works, and why it fixes the problem of “goconvey” FocusConvey!

The best of both worlds

With ezpkg.io/conveyz, you can enjoy the simplicity of “goconvey” and the power of “gomega” in a single package. You can focus on the test itself, not the setup, and use the same assertions as gomega. And with FConvey, you can focus on a single block and propagate to all relevant nested and parent blocks, streamlining the debugging process.

It also comes with other small and nice things like:

Cleaner and focused stacktrace with better message.
FConvey|SConvey make go test failure as default so you don’t accidentally skip the test.
SkipConveyAsTODO to actually skip the test without failing the go test.

So, what are you waiting for? Give ezpkg.io/conveyz a try and let me know what you think! Feel free to open an issue or submit a pull request.
I’m always happy to hear from you! 🚀

If you like the post, subscribe to my newsletter to get latest updates:

Author

I'm Oliver Nguyen. A software maker working mostly in Go and JavaScript. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey. Connect with me on , , , , or subscribe to my posts.

The motivation behind ezpkg.io/conveyz

Ginkgo and Gomega are great, but verbose

Convey: Focus on the test, not the setup

Problems: Incompatible assertions and repeated FocusConvey

Meet ezpkg.io/conveyz

Solution: Make “gomega” Ω work with “goconvey”

Solution: Make FConvey smarter

ginkgo: How a single FIt can focus on a test

But “goconvey” FocusConvey must be set at every level

ezpkg.io/conveyz: Set a single FConvey to focus on a test