Validate zero enum value in Protocol Buffer
How to validate that a Protobuf message does not contain enum fields with zero value? Turn out that is not supported directly by Protobuf! We need to look into how protojson
package is implemented.
More and more companies are adopting gRPC with Protobuf for communication between internal services. It has the benefits of high performance, supporting multiple programming languages, and being backed by Google with a great ecosystem around.
For communication with front-end and external services, Protobuf can be marshaled to JSON format. The browser only understands JSON format, and we can not expect other companies to consume Protobuf directly from us. (Of course, you can, if you are big enough!)
Let’s talk about enum and some problems that we encountered recently with it.Let’s talk about enum
Sample code is written in Go.
From the Protobuf style guide, the zero value enum should have the suffix UNSPECIFIED
. It’s because enum is implemented as a uint32
, and the value 0
is considered as, well, unspecified. It’s similar to nil
for a message or an empty string. When encoding Protobuf as JSON, a nil
message, an UNSPECIFIED
enum, or an empty string is ignored.
We were following that convention, until someday, we did not.
When sending external webhook messages, we decided to not use 0
as UNSPECIFIED
. One reason is that we are using EmitUnpopulated: true
to ensure that all fields are included in the JSON representation when sending webhook messages to external parties. And we don’t want that UNSPECIFIED
value appears in the webhook messages, if somehow we forget to set an enum field to 0. Unit tests can not catch all the mistakes; we engineers know that.
This causes a lot of trouble, so we had to revert and make the value 0
as UNSPECIFIED
again. One problem is that it forces the use of EmitUnpopulated: true
everywhere! And there are places where we don’t want to emit all unpopulated fields. Like calling some third-party APIs. Some messages mix between UNSPECIFIED
enums and non-UNSPECIFIED
enums; there are no ways to send the correct format with that. Use EmitUnpopulated: true
, the third-party APIs don’t understand UNSPECIFIED
; use EmitUnpopulated: false
and some required fields with non-UNSPECIFIED
enums are omitted. Of course, they can all be refactored away, but it should be simpler to just force the use of UNSPECIFIED
at the beginning.
Well, why not just verify that all enum fields are not set to 0 in webhook messages? You may ask.Try checking for zero value
Turn out there are no simple ways to do that in Protobuf 3!
In Protobuf 2, there is required
option to prevent a field to be unset. This option was removed in Protobuf 3, because it prevents refactoring for removing fields. If we forgot to update every service to remove that no-longer-used required
field, especially in a company with multiple teams working together, the messages will be dropped unintentionally. It should be better not to require it upfront. (more)
In Protobuf 3, there was jsonpb.JSONPBMarshaler
interface. We can simply implement that interface for all enums to return error upon seeing a zero value. But again, it was removed! As a protocol, we should minimize the customization as much as possible. Otherwise, that customization will have to be implemented and maintained in all different languages across different teams!
So, how to validate zero value in enum fields?Prevent zero value in enum fields
We’ll have to reach the reflection package. The protoreflect.Message
interface has Range()
method for iterating over every populated field. We can use that method to verify that there are no enum fields with zero… Oh, wait. It only iterates over populated fields. So it won’t detect the zero value in enum!
But the function protojson.Marshal()
can still emit unpopulated fields with EmitUnpopulated
option. How does it implement that? Dive into encoding/protojson
, there is a code snippet for iterating over unpopulated fields (source). Let’s steal it:
// unpopulatedFieldRanger wraps a protoreflect.Message and modifies its Range
// method to additionally iterate over unpopulated fields.
type unpopulatedFieldRanger struct{ pref.Message }
func (m unpopulatedFieldRanger) Range(f func(pref.FieldDescriptor, pref.Value) bool) {
fds := m.Descriptor().Fields()
for i := 0; i < fds.Len(); i++ {
fd := fds.Get(i)
if m.Has(fd) || fd.ContainingOneof() != nil {
continue // ignore populated fields and fields within a oneofs
}
v := m.Get(fd)
isProto2Scalar := fd.Syntax() == pref.Proto2 && fd.Default().IsValid()
isSingularMessage := fd.Cardinality() != pref.Repeated && fd.Message() != nil
if isProto2Scalar || isSingularMessage {
v = pref.Value{} // use invalid value to emit null
}
if !f(fd, v) {
return
}
}
m.Message.Range(f)
}
What the above code does is iterating over additional fields, by looping over protoreflect.Message.Descriptor().Fields()
. Fields within oneof
fields are skipped. Unpopulated singular message
fields are set as invalid
(think of it as null
in generated JSON) before being sent to the input function.
Still a bit more code to write, like implementing a traveling method for iterating over all different Protobuf types: message, array (repeated), dynamic Struct, and of course, enum. But it’s solvable. And I can take a rest now.
Thanks for reading! If you have a better way to do that, please let me know by adding your reply below 👋
How to validate that an enum value field in Protobuf can not be empty? Turn out that is not supported directly by Protobuf! We need to look into how protojson is implemented.
— Oliver Nguyen (@_OliverNguyen)
My latest article about #protobuf #go
https://t.co/d5DycxeZID
Let's stay connected!
Author
I'm Oliver Nguyen. A software maker working mostly in Go and JavaScript. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey. Connect with me on , , , , or subscribe to my posts.