Introduction to Protocol Buffers


What Is Protocol Buffer?

Protocol buffers (or protobuf) is a method of serializing structured data, and is ideal for communication across constraint systems. It is language-neutral and platform-neutral, developed and used by Google since 2001. It was made available to the public in 2008 (free and open-sourced). Official documentation and source code are available at the following links:

The way protocol buffers works is we define a schema that encapsulates all the information we wish to communicate and we store it in a proto file. Let’s say we want to send an alarm status event which also include time and date. Then we would create a schema called AlarmStatus.proto which looks something like below:

// AlarmStatus.proto
syntax = "proto3";
message AlarmStatus {
   bool alarm_active = 1;
   string time_utc   = 2;
   string date_utc   = 3;
}

From that, we use the protocol buffer compiler to create a language specific class that implements the encoding/decoding of the protocol buffer data.

Installing the Protocol Compiler

The protocol compiler (protoc) is the common tool chain for converting our protobuf schemas to binary classes and is required regardless of what language we want to compile to.

From the download page, download the latest protoc pre-built binaries specific to our development environment: protoc-$VERSION-$PLATFORM.zip

We then unpack the zip file to a preferred folder and add an environment variable pointing the the folder containing the protoc executable e.g.:

C:\protoc-3.17.3-win64\bin

If everything is setup correctly, we should be able to execute protoc.exe from any working directory.

Protocol Buffers Using Python

Generating Python compiled protocol buffers from message definition

From the download page, download the latest protocol buffers package for Python and follow README.md for instructions.

If we run pip list we should see that protobuf is installed.

PS D:\Downloads\protobuf-3.17.3\python> pip list
Package    Version
---------- -------
pip        21.1.2
protobuf   3.17.3
setuptools 56.0.0
six        1.16.0
wheel      0.36.2

Given that we have created our protobuf schema(s), we can easily generate the associated python class files with the following command:

PS D:\Workspace\ProtobufDemo> protoc --python_out=. *.proto
Generate Python class from schema demo

Serialising protocol buffers in Python

To serialise a protobuf message we simply create a message, populate it then call SerializeToString()

# SerialisationDemo.py

import random
from datetime import datetime, timezone
from AlarmStatus_pb2 import AlarmStatus

def get_new_alarm_status():
    date_time_utc = datetime.now(timezone.utc)
    alarm_status = AlarmStatus()
    alarm_status.alarm_active = bool(random.getrandbits(1))
    alarm_status.time_utc = "{}".format(date_time_utc.time())
    alarm_status.date_utc = "{}".format(date_time_utc.date())

    return alarm_status

def serialise_alarm_status(alarm_status):
    return alarm_status.SerializeToString()

def main():
    alarm_status = get_new_alarm_status()
    print("Alarm status: {}, time: {}, date: {}".format(alarm_status.alarm_active, alarm_status.time_utc, alarm_status.date_utc))

    serialised_alarm_status = serialise_alarm_status(alarm_status)
    print("Serialised data: {}".format(serialised_alarm_status))

if __name__ == "__main__":
    main()

Deserialising protocol buffers in Python

To deserialise a protobuf message we simply create a message, then call ParseFromString() passing in the serialised data.

# SerialisationDemo.py

import random
from datetime import datetime, timezone
from AlarmStatus_pb2 import AlarmStatus

def get_new_alarm_status():
    date_time_utc = datetime.now(timezone.utc)
    alarm_status = AlarmStatus()
    alarm_status.alarm_active = bool(random.getrandbits(1))
    alarm_status.time_utc = "{}".format(date_time_utc.time())
    alarm_status.date_utc = "{}".format(date_time_utc.date())

    return alarm_status

def serialise_alarm_status(alarm_status):
    return alarm_status.SerializeToString()

def deserialise_alarm_status(serialised_alarm_status):
    alarm_status = AlarmStatus()
    alarm_status.ParseFromString(serialised_alarm_status)

    return alarm_status

def main():
    alarm_status = get_new_alarm_status()
    print("Alarm status: {}, time: {}, date: {}".format(alarm_status.alarm_active, alarm_status.time_utc, alarm_status.date_utc))

    serialised_alarm_status = serialise_alarm_status(alarm_status)
    print("Serialised data: {}".format(serialised_alarm_status))

    alarm_status2 = deserialise_alarm_status(serialised_alarm_status)
    print("Deserialised alarm status: {}, time: {}, date: {}".format(alarm_status2.alarm_active, alarm_status2.time_utc, alarm_status2.date_utc))

if __name__ == "__main__":
    main()

Protocol Buffers Using C#

Generating C# compiled protocol buffers from message definition

Given that we have created our protobuf schema(s), we can easily generate the associated C# class files with “protoc –csharp_out=. *.proto

PS D:\Workspace\ProtobufDemo> ls

    Directory: D:\Workspace\ProtobufDemo

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          21/07/2021  4:00 pm            128 AlarmStatus.proto

PS D:\Workspace\ProtobufDemo> protoc --csharp_out=. *.proto
PS D:\Workspace\ProtobufDemo> ls

    Directory: D:\Workspace\ProtobufDemo

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          21/07/2021  9:44 pm           9841 AlarmStatus.cs
-a---          21/07/2021  4:00 pm            128 AlarmStatus.proto

PS D:\Workspace\ProtobufDemo>

Serialising protocol buffers in C#

Create a new console application and add the Google.Protobuf NuGet package.

PS D:\Workspace\ProtobufDemo> dotnet new console
...
PS D:\Workspace\ProtobufDemo> ls

    Directory: D:\Workspace\ProtobufDemo

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          21/07/2021  9:44 pm                obj
-a---          21/07/2021  9:44 pm           9841 AlarmStatus.cs
-a---          21/07/2021  4:00 pm            128 AlarmStatus.proto
-a---          21/07/2021  9:44 pm            194 Program.cs
-a---          21/07/2021  9:44 pm            171 ProtobufDemo.csproj

PS D:\Workspace\ProtobufDemo> dotnet add package Google.Protobuf
...
PS D:\Workspace\ProtobufDemo>

To serialise a protobuf message we simply create a message, populate it then call ToByteString().

using System;
using Google.Protobuf;

namespace ProtobufDemo
{
   class Program
   {
      static void Main(string[] args)
      {
         var alarmStatus = GetNewAlarmStatus();
         var serialisedAlarmStatus = SerialiseAlarmStatus(alarmStatus);
         Console.WriteLine($"Alarm status: {alarmStatus.AlarmActive}, time: {alarmStatus.TimeUtc}, date: {alarmStatus.DateUtc}");
         Console.WriteLine($"Serialised data: {serialisedAlarmStatus}");
      }

      private static AlarmStatus GetNewAlarmStatus()
      {
         var rand = new Random();
         var status = rand.Next(2) == 1;
         var dateTime = DateTimeOffset.UtcNow;

         return new AlarmStatus
         {
            AlarmActive = status,
            TimeUtc = dateTime.ToString("T"),
            DateUtc = dateTime.ToString("d")
         };
      }

      private static ByteString SerialiseAlarmStatus(AlarmStatus alarmStatus)
      {
         return alarmStatus.ToByteString();
      }
   }
}

Deserialising protocol buffers in C#

To deserialise a protobuf message we simply call MergeFrom() passing in the serialised message.

...
      static void Main(string[] args)
      {
         ...

         var deserialisedAlarmStatus = DeserialiseAlarmStatus(serialisedAlarmStatus);
         Console.WriteLine($"Deserialised alarm status: {deserialisedAlarmStatus.AlarmActive}, time: {deserialisedAlarmStatus.TimeUtc}, date: {deserialisedAlarmStatus.DateUtc}");
      }

      ...

      private static AlarmStatus DeserialiseAlarmStatus(ByteString serialisedAlarmStatus)
      {
         var deserialisedAlarmStatus = new AlarmStatus();
         deserialisedAlarmStatus.MergeFrom(serialisedAlarmStatus);

         return deserialisedAlarmStatus;
      }
   }
...

Conclusion

Personally I do prefer protobuf over JSON because of it’s well defined schema approach, data size and language neutral. These are very important factors for IoT systems.

Categories: Internet of things, ToolingTags: , ,

1 comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: