Kafka - a working docker-compose

Here's a working docker-compose that runs zookeeper, kafka and kafka-ui in same network.

version: '3.8'
services:

    zookeeper:
        container_name: zookeeper
        image: confluentinc/cp-zookeeper:latest
        hostname: zookeeper
        restart: always
        environment:
            ZOOKEEPER_SERVER_ID: 1
            ZOOKEEPER_CLIENT_PORT: 2181
            ZOOKEEPER_TICK_TIME: 2000
            ZOOKEEPER_INIT_LIMIT: 5
            ZOOKEEPER_SYNC_LIMIT: 2
        ports:
            - 2181:2181

    kafka:
        container_name: kafka
        image: confluentinc/cp-kafka:latest
        depends_on:
            - zookeeper
        restart: always
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,RMOFF_DOCKER_HACK://kafka:9093
            KAFKA_LISTENERS: PLAINTEXT://:9092,RMOFF_DOCKER_HACK://:9093
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,RMOFF_DOCKER_HACK:PLAINTEXT
            KAFKA_INTER_BROKER_LISTENER_NAME: RMOFF_DOCKER_HACK
        ports:
            - 9092:9092

    kouncil:
        container_name: kouncil
        image: consdata/kouncil:latest
        restart: always
        depends_on:
            - kafka
        environment:
            bootstrapServers: kafka:9093
            BOOTSTRAPSERVERS: kafka:9093
            BOOTSTRAP_SERVERS: kafka:9093
        ports:
            - 8888:8080

 

Spring Boot - shutdown and restart application programmatically

Today we will be discussing how to shutdown and restart a Spring application programmatically.

1) New Project

First let's create a simple spring boot app with web dependency so that we can setup an endpoint to trigger the shutdown and restart. Let's head over to  https://start.spring.io/ and choose web and other dependencies that you need

2) Few words about ApplicationContext and ConfigurableApplicationContext

ApplicationContext is the central interface that represents the Spring IoC container and is responsible for instantiating, configuring, and assembling the beans.  ConfigurableApplicationContext is another interface that extends  ApplicationContext and provides facilities to configure refresh, startup & shutdown of the application and allows registration of custom listeners to respond to those events.

3) Let's code!

3.1) Simple App

Run the application and store ConfigurableApplicationContext in a static variable appCtx so that we can use it later

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;

@SpringBootApplication
public class App {

    static ConfigurableApplicationContext appCtx;

    public static void main(String[] args) {
        appCtx = SpringApplication.run(App.class, args);
    }

3.2) Shutdown - call ConfigurableApplicationContext.close() 

This closes the application context, releasing all resources and locks that the implementation might hold. This includes destroying all cached singleton beans.

static void shutdown() {
    appCtx.close();
}

3.3) Restart

We know how to stop and start the application. To stop, we can simply do:

appCtx.close();

And to start, we can simply call the SpringApplication.run(..)

appCtx = SpringApplication.run(App.class, args);

But the catch here is, appCtx.close() would exit the application and the SpringApplication.run(..) won't be able to run

So, a solution would be to use Daemon Thread to shutdown and start the application. Daemon thread is a type of thread that can run independently in the background(independent of the main thread). 

Thread t = new Thread(() -> {
    appCtx.close();
    appCtx = SpringApplication.run(...);
});

t.setDaemon(false);
t.start();

3.4) Passing the original program arguments

We may want to use the original arguments that were passed to run method when we restart the application.

 SpringApplication.run(App.class, args);

We can do that by either storing String[] args in a static variable or use ApplicationArguments bean that Spring Boot creates to store the arguments.

var args = appCtx.getBean(ApplicationArguments.class);      //retrieve
appCtx = SpringApplication.run(App.class, args.getSourceArgs());  //pass

3.5) Web endpoints to trigger the shutdown and restart

@RestController
class Ctrl {

    @GetMapping("/shutdown")
    void shutdown() {
        App.shutdown();
    }

    @GetMapping("/restart")
    void restart() {
        App.restart();
    }
}

4) Complete Example:

package gt.restart;

import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
public class App {

    static ConfigurableApplicationContext appCtx;

    public static void main(String[] args) {
        appCtx = SpringApplication.run(App.class, args);
    }

    static void shutdown() {
        if (appCtx.isActive()) {
            appCtx.close();
        }
    }

    static void restart() {
        var args = appCtx.getBean(ApplicationArguments.class);

        Thread thread = new Thread(() -> {
            appCtx.close();
            appCtx = SpringApplication.run(App.class, args.getSourceArgs());
        });

        thread.setDaemon(false);
        thread.start();
    }

}

@RestController
class Ctrl {

    @GetMapping("/shutdown")
    void shutdown() {
        App.shutdown();
    }

    @GetMapping("/restart")
    void restart() {
        App.restart();
    }
}

5) Test

Call the endpoints

GET http://localhost:8080/restart

GET http://localhost:8080/shutdown

Read Request parameter String from URL in Spring/HttpServlet

Read entire query string(request parameter) in Spring Boot/HttpServlet

In Spring Boot and other Java web frameworks, we capture the request parameters (aka query parameters) and map them to individual variable or a Map<String,String>. 

But, if you want to read the entire query parameter in its raw form ( eg: to read name=Ganesh&country=Nepal&ageLt=30&page=1 from request: http://localhost:8080/search?name=Ganesh&country=Nepal&ageLt=30&page=1), you can utilize HttpServletRequest.getQueryString()

Example:

@GetMapping("/test1")
void endpoint1(HttpServletRequest req) {
    var qs = req.getQueryString()  //returns the entire string
    qs.split("&") //split to get the individual parameters
}

Note: Spring Boot automatically resolves the HttpServletRequest object ( its the real holder of request object in Http Servlet environment)

GIT delete merged branches

How to delete merged branches from local copy:

When you work on a project for a while, there is a good chance that you will have a list of old/already merged branches showing on your local git. 

When you run $git branch it will show all the branches since you started working on the project including the ones that are not currently active.

$git branch
    PROJX-1024-add-new-button
    PROJX-1026-add-another-button
    PROJX-1027-titleupdates
    build-fix
    develop
    release
    master 

Here's how you can find the merged branches: It will only list a branch if it exists in local but not in remote(deleted from remote after merging).

$git branch --merged
    PROJX-1024-add-new-button
    build-fix
    develop
    release

Here's how you delete the merged branches except develop and release

$git branch --merged | egrep -v "(^\*|develop|release)" | xargs git branch -d

You can add any number of branches to the skip list separated by pipe |

$git branch --merged | egrep -v "(^\*|develop|release|another_branch|yet_another_branch)" | xargs git branch -d

Java view,download,filter email using javax.mail

Java SDK doesn't come with inbuilt mail API but Java-EE (now called Jakarta EE) provides several utility library like json, servlet, web-service, jms, validation, xml, regexp including email.

In this example I'm going to show how we can connect to email server (eg: gmail) and download/view seen/unseen emails.

Required Dependency :jakarta.mail

Add this dependency with latest version in your pom.xml or manually download/add the jar file to your project's lib.

<dependency>
    <groupId>com.sun.mail</groupId>
    <artifactId>jakarta.mail</artifactId>
</dependency>

The API Usage:

Ideally, the steps would be as following:

- Create Session with host, protocal type etc

- Get message store (Store) for the email protocol. The supported protocols are IMAP, POP, SMTP etc

- Connect to the host using username/password

- Open folder and search for messages

The API Usage with code example:

- Create Session with host, protocol type etc

Properties properties = new Properties();
properties.put("mail.imap.host", host);
properties.put("mail.store.protocol", protocal);
Session emailSession = Session.getDefaultInstance(properties);

 - Get message store (Store) for the email protocol. The supported protocols are IMAP, POP, SMTP etc

Store store = emailSession.getStore(protocal);

- Connect to the host using username/password

store.connect(host, user, password);

- Open folder and search for messages

Folder emailFolder = store.getFolder("INBOX");
emailFolder.open(Folder.READ_ONLY);

Search using filters

Flags seen = new Flags(Flags.Flag.SEEN);
FlagTerm unseenFlagTerm = new FlagTerm(seen, false);
Message[] messages = emailFolder.search(unseenFlagTerm);

Complete Example:

import javax.mail.*;
import javax.mail.search.FlagTerm;
import java.util.Properties;

public class MailllllIMAP {

    public static void check(String host, String protocal, String user, String password) {
        try {
            Properties properties = new Properties();
            properties.put("mail.imap.host", host);
            properties.put("mail.store.protocol", protocal);
            Session emailSession = Session.getDefaultInstance(properties);

            Store store = emailSession.getStore(storeType);
            store.connect(host, user, password);
            Folder emailFolder = store.getFolder("INBOX");
            emailFolder.open(Folder.READ_ONLY);

            //unseen email filter
            Flags seen = new Flags(Flags.Flag.SEEN);
            FlagTerm unseenFlagTerm = new FlagTerm(seen, false);
            Message[] messages = emailFolder.search(unseenFlagTerm);

            System.out.println("Total messages: " + messages.length);

            for (Message m : messages) {
                System.out.println(("From: " + m.getFrom()[0]));
                System.out.println(("Subject: " + m.getSubject()));
                System.out.println(("Subject: " + m.getContent()));
                //other fields
            }


            //close or do more stuff
            emailFolder.close(false);
            store.close();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {

        String host = "pop.gmail.com";
        String protocal = "imaps";
        String username = "XXXX@gmail.com";
        String password = "XXXXXXX";

        check(host, protocal, username, password);

    }

}

What if you get an error like this?

    AuthenticationFailedException: [AUTH] Username and password not accepted.

You get this error when trying to gmail. You will need to turn ON the "Allow less secure apps" option on https://www.google.com/settings/security/lesssecureapps

Java HttpClient tutorial with File Upload example

Java 11 comes with a nice HttpClient API/Implementation so that we no longer need to rely on external libraries like Apache HttpClient execute http requests.

It has the following simple classes(namely HttpClient, HttpRequest, HttpResponse, BodyPublisher, BodyHandler) and they all follow builder pattern to create objects.

Creating HttpClient:

HttpClient client = HttpClient.newBuilder().build();

Creating request:

HttpRequest request = HttpRequest.newBuilder()
        .header("key1", "value1")
        .header("key2", "value2")
        .uri(URI.create("http://localhost:8080/hello"))
        .POST(HttpRequest.BodyPublishers.ofString("Request Body"))
        .build();

Sending request:

client.send(request, HttpResponse.BodyHandlers.APPROPRIATE_HANDLER);

Here, BodyPublisher and BodyHandler can be used to create request body and process returned response respectively. They both support string, byte, file, input stream etc

Upload file using BodyPublishers.ofFile

Normally file upload using a web page are implemented using multi-part(file data with additional properties). But in this example we are going to upload file's byte array in request body and file name in request param.

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Path;

public class HttpTest {
  public static void main(String[] args) throws Exception {
    uploadFile(Path.of("testFile.txt"));
  }

  public static void uploadFile(Path file) throws IOException, InterruptedException {
    HttpClient client = HttpClient.newBuilder().build();

    HttpRequest request = HttpRequest.newBuilder()
      .uri(URI.create("http://localhost:8080/uploadFile?uploader=HttpTestApp&fileName=" 
                  + file.getFileName()))
      .POST(HttpRequest.BodyPublishers.ofFile(file))
      .build();

    client.send(request, HttpResponse.BodyHandlers.ofString());
  }
}

To make the example complete, let's assume we have a following simple Spring Boot app that receives the file byte array as request body:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.*;

import java.io.*;

@SpringBootApplication
public class FileUploadApp {
    public static void main(String[] args) {
        SpringApplication.run(FileUploadApp.class, args);
    }
}

@Controller
class FileUplCtrl {

  @PostMapping(path = "/uploadFile")
  public void addPhoto(@RequestBody byte[] barr,
              @RequestParam(name = "uploader") String uploader,
              @RequestParam(name = "fileName") String fileName) throws Exception {

   System.out.println("Received file " + fileName + " , uploaded by " + uploader + " Size: " + barr.length);

    //write to disk
    try (OutputStream os = new FileOutputStream(new File("UPL" + fileName))) {
         os.write(barr);
    }

  }
}

Required Dependency on Spring Boot app:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Enjoy coding!

Thymeleaf re-usable pagination component- Spring MVC/Bootstrap

In this blog, we will discuss how we can create a reusable pagination component using Thymeleaf and Spring MVC/Data.

Goal is to create a pagination UI similar to this:

 

Controller 

Spring resolves the size and page parameter from URL : /users?size=20&page=1 to Pageable object at the Controller methods. We are returning Page<> instead of simple list. Page contains all the information that we need to create the pagination UI as above.

@GetMapping("/articles")
public String userHome(Model model, Pageable pageable) {
    Page<Article> page= articleService.findAllByStatus(Status.PUBLISHED, pageable);
    model.addAttribute("articles", page);
    return "home-page";
}

@GetMapping("/users")
public String userHome(Model model, Pageable pageable) {
    Page<User> page= userRepo.findAllByStatus(Status.ACTIVE, pageable);
    model.addAttribute("users", page);
    return "user-list";
}

 

Pagination at home-page.html   

<th:block th:each="article : ${articles}">
   <!-- display article in a table row or card/box -->
</th:block>

<!--pager at bottom -->
<th:block th:replace="_fragments/_utils :: pagination(${articles})">
</th:block>

Pagination at user-list.html

<th:block th:each="user : ${users}">
   <!-- display user in a table row or card/box -->
</th:block>
<!--pager at bottom -->
<th:block th:replace="_fragments/_utils :: pagination(${users})">
</th:block>

Reusable Fragment Definition at _fragments/_utils.html

Note that the pages are 0 based meaning page=0 equals the first page of data. You can make it to 1 based using the steps described here: https://ganeshtiwaridotcomdotnp.blogspot.com/2020/09/spring-pageable-start-index-with-one.html. You might also need to update the indices below to to handle that.

<th:block th:fragment="pagination(pagedObject)">

    <!-- works with org.springframework.data.domain.Page<pagedObject>, -->

    <div th:if="${pagedObject.getTotalPages() != 1}" class="form-group col-md-11 pagination-centered">
        <ul class="pagination">
            <!-- page number start with 0, totalPages returns actual number of pages -->
            <li th:classappend="${pagedObject.getNumber() == 0} ? disabled" class="page-item">
                <a class="page-link" th:href="@{''(size=${pagedObject.getSize()}, page=0)}">&laquo;</a>
            </li>
            <li th:classappend="${pagedObject.getNumber() == 0} ? disabled" class="page-item">
                <a class="page-link"
                   th:href="@{''(size=${pagedObject.getSize()}, page=${pagedObject.getNumber() -1})}">&larr;</a>
            </li>
            <li th:classappend="${pagedObject.getNumber() == (page  )} ? 'active pointer-disabled'"
                th:each="page : ${#numbers.sequence(0, pagedObject.getTotalPages() -1)}" class="page-item">
                <a class="page-link" th:href="@{''(size=${pagedObject.getSize()}, page=${page})}"
                   th:text="${page + 1}"></a>
            </li>
            <li th:classappend="${pagedObject.getNumber() + 1 == pagedObject.getTotalPages()} ? disabled"
                class="page-item">
                <a class="page-link"
                   th:href="@{''(size=${pagedObject.getSize()}, page=${pagedObject.getNumber() + 1})}">&rarr;</a>
            </li>
            <li th:classappend="${pagedObject.getNumber() + 1 == pagedObject.getTotalPages()} ? disabled"
                class="page-item">
                <a class="page-link"
                   th:href="@{''(size=${pagedObject.getSize()}, page=${pagedObject.getTotalPages()})}">&raquo;</a>
            </li>
        </ul>

        <p class="text-muted small">
            Showing page <span th:text="${pagedObject.getNumber() +1}"></span> of <span
            th:text="${pagedObject.getTotalPages()}"></span>.
            Total: <span th:text="${pagedObject.getTotalElements()}"></span>.

        </p>
    </div>


    <div th:if="${pagedObject.getTotalElements() ==0}">
        <div class="alert alert-warning" role="alert">
            <span>No items available.</span>
        </div>
    </div>


</th:block>

Reference: https://github.com/gtiwari333/spring-boot-web-application-seed/blob/master/main-app/src/main/resources/templates/_fragments/_utils.html

Thymeleaf pass parameter to fragment

Making reusable fragment/UI Components in Thymeleaf

Thymeleaf supports creation of 'function like' mechanism and pass parameters to make the UI components reusable.

In this example we are going to create a reusable alert component using thymeleaf:

The fragment definition is similar to how a function/method is defined:

File: _fragments/_utils.html

<div th:fragment="alertBox(message,type)">
    //do stuff with message and type
</div>

Let's call with some parameters in another file:

<div th:replace="_fragments/_utils::alertBox ('A message','success')">
</div>

The parameter order is not important if you do like this:

<div th:replace="_fragments/_utils::alertBox (type = 'success', message='A message' )">
</div>

A complete example:

<div th:fragment="alertBox (message,type)">
    <div
        th:classappend="|
            ${type == 'success' ? 'alert-success': ''}
            ${type == 'info' ? 'alert-info': ''}
            ${type == 'danger' ? 'alert-danger': ''} |"
        class="alert alert-dismissible">
        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
        <span th:text="${message}"></span>
    </div>
</div>

Calling it:

<div th:replace="_fragments/_utils :: alertBox ('Hello this is a success!','success')">
</div>

<div th:replace="_fragments/_utils :: alertBox ('Danger! danger! .......','danger')">
</div>


<div th:replace="_fragments/_utils :: alertBox ( type = 'success', message='Success message' )">
</div>

Output:

Spring pageable - start index with one

By default Spring uses 0 indexed page meaning a page number of 0 in the request equals the first page.

You can customize this to start from 1 in following by creating a bean of PageableHandlerMethodArgumentResolverCustomizer and setting the setOneIndexedParameters to true.

You can read  more about the method argument resolver and other customizations to pagable in my earlier blog post:  https://ganeshtiwaridotcomdotnp.blogspot.com/2020/09/spring-data-pagination-set-max-page.html

Configures whether to expose and assume 1-based page number indexes in the request parameters. When its true, a page number of 1 in the request will be considered the first page.

@Bean
public PageableHandlerMethodArgumentResolverCustomizer paginationCustomizer() {
  return pageableResolver -> {
     pageableResolver.setOneIndexedParameters(true); //default is false, starts with 0
  };
}

Now you can have your pagination url to start with page=1 eg: /users?size=20&page=1.

Spring data Pagination - set max page size and other customizations

Background:

HandlerMethodArgumentResolver is a strategy interface to resolve method parameters in context of given context. So, if you want to automatically resolve the parameter MyObject in the following method, you can create a bean of HandlerMethodArgumentResolver and implement logic to resolve the argument.

@GetMapping("/users")
public Page<User> getUsers(MyObject object) {

Spring Framework already provides a lot of resolvers to handle various parameters such as AuthenticationPrincipal, CSRF, Session, Cookie, MVC Model, and of course Pageable.

 

Pageable Resolver:

Spring Data comes with PageableHandlerMethodArgumentResolver to resolve pageable parameter from the request URL.

If you send a request /users?size=20&page=2, the Pageable object will be injected to the method parameter.

@GetMapping("/users")
public Page<User> getUsers(Pageable pageable) {
    return userRepository.findAllByStatus(Status.ACTIVE, pageable);
}

Customize PageableHandlerMethodArgumentResolver

To customize the Pageable resolver, we need to create a bean of PageableHandlerMethodArgumentResolverCustomizer , which will be applied at SpringDataWebConfiguration#customizePageableResolver before the pageableResolver() is created SpringDataWebConfiguration#pageableResolver.

PageableHandlerMethodArgumentResolverCustomizer is a SAM (single method interface aka FunctionalInterface). 

Setting max page size

@Bean
public PageableHandlerMethodArgumentResolverCustomizer paginationCustomizer() {
    return pageableResolver -> {
        pageableResolver.setMaxPageSize(20); //default is 2000
        pageableResolver.setPageParameterName("pageNumber"); //default is page
        pageableResolver.setSizeParameterName("elementsPerPage"); //default is size
    };
}

Now the url should be /users?elementsPerPage=20&pageNumber=2 instead of /users?size=20&page=2.

If you pass elementsPerPage more than 20, it will be defaulted back to 20.

Which will be helpful to prevent potential attacks trying to issue an OutOfMemoryError.