Showing posts with label Hibernate. Show all posts
Showing posts with label Hibernate. Show all posts

How to make integration tests faster without @DirtiesContext

If your only excuse to use @DirtiesContext is to re-initialize database between tests, then this blog post is for you.

Spring's @DirtiesContext is very useful to make your integration tests faster by indicating the underlying Spring ApplicationContext is modified and forcing it to reload the context (not the whole application) between tests.

This is particularly useful when we want to reset the state of database between tests. This allows us to group many tests together in a same class so that we can easily share some test data, assertions etc. So your test code would look this this:

@SpringBootTest
@DirtiesContext(classMode = AFTER_EACH_TEST_METHOD)
class SomeTest {

@Autowired SomeService serv;
@Autowired DataCreator dc;

@Test
void someTest1() {
//prepare data
dc.initData();
//run test
serv.modifySomeData();
//verify
}

@Test
void someTest2() {
//prepare data
dc.initData(2);
//run test
serv.modifyAnotherData();
serv.doSomethingElse();
//verify
}
}

Problem with @DirtiesContext

This has one major flaw though. By resetting the ApplicationContext, it would also drop all the databases and need to re-create them after every test. So, if the only reason to use @DirtiesContext is to reset the database, then there's a better way of doing this.

Is deleting records from all table not a good solution?

Correct. Deleting records is still a slow operation. When you have a good amount to test data, the @DirtiesContext can sometimes becomes faster than deleting the records one by one. Also, there's referential integrity between tables that enforces you to follow a series of deletes to first delete records from child tables and go up. 

Table Truncate to the rescue

Truncate (truncate table X) is faster way to delete records from database than deleting records. If we already know the list of table and join tables then we can loop through them and execute the TRUNCATE TABLE command. One advantage of this approach is that we can skip truncating some lookup tables that will always have constant records.

How to Truncate H2 Tables:

Note that before we execute the TRUNCATE TABLE we should disable the referential integrity.
em.createNativeQuery("SET REFERENTIAL_INTEGRITY FALSE").executeUpdate();
for (String t : tableNames) {
em.createNativeQuery("TRUNCATE TABLE " + t ).executeUpdate();
}
em.createNativeQuery("SET REFERENTIAL_INTEGRITY TRUE").executeUpdate();

Integrating everything together (Sample App @ GitHub):

TestDataManager to grab list of tables, truncate and populate the test data:

import gt.app.DataCreator;
import gt.app.config.MetadataExtractorIntegrator;
import lombok.RequiredArgsConstructor;
import org.hibernate.boot.Metadata;
import org.hibernate.mapping.Collection;
import org.hibernate.mapping.PersistentClass;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import javax.persistence.EntityManager;
import java.util.ArrayList;
import java.util.List;

@Service
@RequiredArgsConstructor
public class TestDataManager implements InitializingBean {
final EntityManager em;
final DataCreator dataCreator;

private final List<String> tableNames = new ArrayList<>(); //shared

@Override
public void afterPropertiesSet() {
Metadata metadata = MetadataExtractorIntegrator.INSTANCE.getMetadata();

for (Collection persistentClass : metadata.getCollectionBindings()) {
tableNames.add(persistentClass.getCollectionTable().getExportIdentifier());
}

for (PersistentClass persistentClass : metadata.getEntityBindings()) {
tableNames.add(persistentClass.getTable().getExportIdentifier());
}
}

@Transactional
public void truncateTablesAndRecreate() {
truncateTables();
dataCreator.initData();
}

void truncateTables() {
em.createNativeQuery("SET REFERENTIAL_INTEGRITY FALSE").executeUpdate();
for (String tableName : tableNames) {
em.createNativeQuery("TRUNCATE TABLE " + tableName).executeUpdate();
}
em.createNativeQuery("SET REFERENTIAL_INTEGRITY TRUE").executeUpdate();
}
}

Call the truncateTablesAndRecreate() @BeforeEach test

@SpringBootTest
class WebAppIT {

@Autowired
TestDataManager tdm;

@BeforeEach
void resetDB(){
tdm.truncateTablesAndRecreate();
}

 

This is awesome, how about other databases?

To get a List of tables: You can rely on Hibernate's Metadata to grab list of table/join tables

Truncate command: Each database has different command to set the referential integrity and truncate. This above example is for H2.

You can do the following for MySQL


//for MySQL:
em.createNativeQuery("SET @@foreign_key_checks = 0").executeUpdate();
for (String tableName : tableNames) {
em.createNativeQuery("TRUNCATE TABLE " + tableName).executeUpdate();
}
em.createNativeQuery("SET @@foreign_key_checks = 1").executeUpdate();

For other populate databases, you can take reference from Hibernate's Test code itself. They are available at: Hibernate GitHub. The *Cleaner classes have code snippet that describes how to perform truncate on each database.

How fast the tests ran after this update?

Very fast! In a big application (Spring Boot, H2) with 62 tables and 315 tests that were using @DirtiesContext, we reduced our test execution time from 18 minutes to 4minutes.

Full working example is available at this sample app.

JPA/Hibernate get find all Table and Column metadata

How to use Hibernate Metadata to find All columns and tables

Getting the Hibernate's Metadata object into the Spring application is tricky. Luckily, Hibernate Provides an Integrator API (org.hibernate.integrator.spi.Integrator) that we can use to customize/interact with Hibernate. Its the same API that Caching, Bean Validation etc library uses to integrate with Hibernate. 

Also, Spring Boot provides HibernatePropertiesCustomizer to link the 'hibernate.integrator_provider' property to Integrator implementation.

Here's how we can configure the Hibernate Integrator to read metadata.

Step 1) create a extractor implementation

This class is a singleton class and does absolutely nothing other than exposing the Metadata and Database. Since this is singleton this class and the database, metadata objects can be statically accessed using MetadataExtractorIntegrator.INSTANCE

import lombok.Data;
import org.hibernate.boot.Metadata;
import org.hibernate.boot.model.relational.Database;
import org.hibernate.engine.spi.SessionFactoryImplementor;
import org.hibernate.integrator.spi.Integrator;
import org.hibernate.service.spi.SessionFactoryServiceRegistry;

@Data
public class MetadataExtractorIntegrator implements Integrator {

public static final MetadataExtractorIntegrator INSTANCE =
new MetadataExtractorIntegrator();
private Database database;
private Metadata metadata;

@Override
public void integrate(Metadata metadata, SessionFactoryImplementor sf,
SessionFactoryServiceRegistry sr) {
this.database = metadata.getDatabase();
this.metadata = metadata;
}

@Override
public void disintegrate(SessionFactoryImplementor sf,
SessionFactoryServiceRegistry sr) {
}
}

Step 2) Register the Spring Hibernate Customizer


import org.hibernate.jpa.boot.spi.IntegratorProvider;
import org.springframework.boot.autoconfigure.orm.jpa.HibernatePropertiesCustomizer;
import org.springframework.context.annotation.Configuration;

import java.util.List;
import java.util.Map;

@Configuration
public class HibernateConfig implements HibernatePropertiesCustomizer {
@Override
public void customize(Map<String, Object> hibernateProps) {
hibernateProps.put("hibernate.integrator_provider",
(IntegratorProvider) () -> List.of(MetadataExtractorIntegrator.INSTANCE));
}
}

Step 3) Use MetadataExtractorIntegrator.metadata to extract the metadata

This is a simple Spring Component that uses Metadata.getCollectionBindings and Metadata.getEntityBindings to extract the tables, columns, PK and type

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.hibernate.boot.Metadata;
import org.hibernate.mapping.*;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;

import javax.persistence.EntityManager;
import java.util.Iterator;

@Component
@RequiredArgsConstructor
@Slf4j
public class DBMetadataReader implements InitializingBean {
final EntityManager em;

@Override
public void afterPropertiesSet() {

Metadata metadata = MetadataExtractorIntegrator.INSTANCE.getMetadata();

//Collection tables
for (Collection c : metadata.getCollectionBindings()) {
log.info("Collection table: {}", c.getCollectionTable().getQualifiedTableName());
for (Iterator<Column> it = c.getCollectionTable().getColumnIterator();
it.hasNext(); ) {
Column property = it.next();
log.info(" {} {} ", property.getName(), property.getSqlType());
}
}

//all entities
for (PersistentClass pc : metadata.getEntityBindings()) {
Table table = pc.getTable();

log.info("Entity: {} - {}", pc.getClassName(), table.getName());

KeyValue identifier = pc.getIdentifier();

//PK
for (Iterator<Selectable> it = identifier.getColumnIterator();
it.hasNext(); ) {
Column column = (Column) it.next();
log.info(" PK: {} {}", column.getName(), column.getSqlType());
}

//property/columns
for (Iterator it = pc.getPropertyIterator();
it.hasNext(); ) {
Property property = (Property) it.next();

for (Iterator columnIterator = property.getColumnIterator();
columnIterator.hasNext(); ) {
Column column = (Column) columnIterator.next();
log.info(" {} {}", column.getName(), column.getSqlType());
}
}
}
}
}

Example Project:

Checkout my sample github project and the source for working example



Read all table and columns in Jpa/Hibernate

How to get metadata about all table and columns managed by JPA/Hibernate?

There are many ways to get a list of table and columns in your project that uses JPA/Hibernate. Each has pros and cons.

Option A) Direct Query on INFORMATION_SCHEMA.

Simplest way is to do direct query on INFORMATION_SCHEMA  or similar schema that the database internally uses.

For MySQL, H2, MariaDB etc the following would work. We will need specific query for each database.

SELECT * from INFORMATION_SCHEMA.TABLES
SELECT * FROM INFORMATION_SCHEMA.COLUMNS

Option B) DB Independent query using JDBC API

We can do DB independent query by using JDBC API to return the Metadata. It would use DB specific query provided by DB driver to return the metadata.

DataSource ds = ; //create/wire DataSource object
DatabaseMetaData metaData = ds.getConnection().getMetaData();
ResultSet schemasRS = metaData.getSchemas();
ResultSet tablesRS = metaData.getTables(null, null, null, new String[]{"TABLE"});

We can iterate over the ResultSet to get the schema, table and columns. It would return everything that the Database has.

Option C) Use EntityManager MetaModel to read Entity classes

In order to retrieve only the entity/tables that the application uses, we can rely on some Java Reflection magic as following:

EntityManager em; //autowire the bean
MetamodelImplementor metaModelImpl = (MetamodelImplementor) em.getMetamodel();
List<String> tableNames = metaModelImpl
.entityPersisters()
.values().stream()
.map(ep -> ((AbstractEntityPersister) ep).getTableName())
.toList();

Option D) Use Hibernate Magic

Use hibernate's Metadata class that stores the ORM model determined by provided entity mappings.

org.hibernate.boot.Metadata metadata; //getting the Metadata is tricky though

for (PersistentClass persistentClass : metadata.getEntityBindings()) {
tableNames.add(persistentClass.getTable().getExportIdentifier());
}



JPA EntityManager get Session Object in Hibernate

How to get Session object from JPA EntityManager


With Hibernate (JPA 2.0 implementation), you would do:

Steps:


//1. Inject/Autowire Entity Manager
@Inject 
private EntityManager entityManager;    //javax.persistence.EntityManager 


//2. Get Session Object 
Session session = entityManager.unwrap(Session.class);  //org.hibernate.Session


Hibernate Create Update Delete child objects - Best way

Hibernate Create Update Delete child objects - Best way