Protobuf Apache Pulsar with Spring Boot

Protocol buffers(protobuf) are language and platform neutral mechanism to serialize structured data. We define the data structure in protobuf format and generate source code to write and read data to and from a variety of data streams and using a variety of languages.

Protocol buffers currently support generated code in Java, Python and more.

Apache Pulsar is a cloud-native, distributed messaging and streaming platform.

In this blog we are going to use a simple protobuf message and use protobuf-maven-plugin to generate Java source code and use protobuf as message format(schema type) for Apache Pulsar pub-sub application.

1) Protobuf model + Java Source generation:

By default protobuf-maven-plugin looks at src/main/proto folder for the .proto files. We have following proto files to represent Person and Greeting objects

Person.proto

syntax = "proto3";
package app.model;
message Person {
  string fName = 1;
  string lName = 2;
}

Greeting.proto:

syntax = "proto3";
package app.model;
message Greeting {
  string greeting = 1;
}

Here's the basic configuration for protobuf-maven-plugin. It also requires os-maven-plugin. Please refer to the github project for the full source.

 

<plugin>
    <groupId>org.xolstice.maven.plugins</groupId>
    <artifactId>protobuf-maven-plugin</artifactId>
    <version>0.6.1</version>
    <configuration>
        <protocArtifact>com.google.protobuf:protoc:3.12.0:exe:${os.detected.classifier}</protocArtifact>
        <pluginId>grpc-java</pluginId>
        <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.25.0:exe:${os.detected.classifier}</pluginArtifact>
        <clearOutputDirectory>true</clearOutputDirectory>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
            </goals>
        </execution>
    </executions>
</plugin>

 

2) Spring Boot + Apache Pulsar

PulsarClient Bean: Spring Boot doesn't have an official auto configuration for Apache Pulsar yet. So, we have to create the PulsarClient bean ourselves.

 

@Bean
PulsarClient pulsarClient() throws PulsarClientException {
    return PulsarClient.builder()
            .serviceUrl("pulsar://localhost:6650")
            .build();
}

Also the Producer. Please note the parameter of newProducer. We are using PROTOBUF  as the schema type.

Producer bean: We will need to create the producer bean for Person model at main-app and for Greeting model at greeting-service.

@Bean
Producer<THE_MODEL> personProducer(PulsarClient pulsarClient) throws PulsarClientException {
    return pulsarClient.newProducer(Schema.PROTOBUF(THE_MODEL.class))
            .topic(THE_TOPIC)
            .create();
}

 

Pulsar Listener Config: We can register the message listener using @PostConstruct as follows

final PulsarClient pulsarClient;

@PostConstruct
private void initConsumer() throws PulsarClientException {
    pulsarClient
            .newConsumer(Schema.PROTOBUF(THE_MODEL.class))
            .topic(THE_TOPIC)
            .subscriptionName(SUBSCRIPTION_NAME)
            .messageListener((consumer, msg) -> {
                
                //message handler logic

            })
            .subscribe();
}

That's all the configuration we will need.

3. How to run?

1) Clone the project https://github.com/gtiwari333/spring-protobuf-grpc-apache-pulsar and run mvn clean compile to generate java sources and compile the project

2) Run apache pulsar instance using docker(you can run it manually)

docker run -it   -p 6650:6650   -p 8080:8080   apachepulsar/pulsar:2.2.0   bin/pulsar standalone

3) Start GreetingApp and MainApp

4) Hit localhost:8082/greet/Joe/Biden to publish a message. You will see this being received by greeting-service and the greeting being sent back to the queue which will be received by main-app.

The project structure:

├── pom.xml
├── greeting-service
│   ├── pom.xml
│   └── src
│       └── main
│           ├── java
│           │   └── gt
│           │       └── greeting
│           │           └── GreetingApp.java
│           └── resources
│               └── application.properties
├── main-app
│   ├── pom.xml
│   └── src
│       └── main
│           ├── java
│           │   └── gt
│           │       └── mainapp
│           │           └── MainApp.java
│           └── resources
│               └── application.properties
├── protobuf-model
│   ├── pom.xml
│   └── src
│       └── main
│           └── proto
│               ├── Greeting.proto
│               └── Person.proto

 

References:

• https://developers.google.com/protocol-buffers/docs/overview
• https://pulsar.apache.org/docs/en/client-libraries-java/
• GitHub project https://github.com/gtiwari333/spring-protobuf-grpc-apache-pulsar