2019-04-26

Testing with DynamoDB Locally

Introduction

Running tests locally is a widely accepted industry practice. If one is accessing a third party API, that said API is mocked so that no calls over the network is made.

Calls to a DynamoDB instance can be mocked by using AWS’s DynamoDB Local downloadable. Unfortunately, setting up DynamoDB locally is not the most straightforward task. In this tutorial, I will go over how to set up DynamoDB Local with Gradle for your Spring Boot project and a utility class for your integration tests.

Downloading DynamoDB Local and Setting Up with Gradle

For setting up DynamoDB Local with Gradle, I’ve mostly copied steps from this StackOverflow answer with some slight differences.

Add the DynamoDB repository in your `build.gradle` file:

repositories {
    maven {
        name "DynamoDB Local Release Repository - EU (Frankfurt) Region"
        url "https://s3.eu-central-1.amazonaws.com/dynamodb-local-frankfurt/release"
    }
}

I used the Frankfurt Region for the release repository but there are other AWS regions available that you can use:

id	URL
`dynamodb-local-mumbai`	https://s3.ap-south-1.amazonaws.com/dynamodb-local-mumbai/release
`dynamodb-local-singapore`	https://s3-ap-southeast-1.amazonaws.com/dynamodb-local-singapore/release
`dynamodb-local-tokyo`	https://s3-ap-northeast-1.amazonaws.com/dynamodb-local-tokyo/release
`dynamodb-local-frankfurt`	https://s3.eu-central-1.amazonaws.com/dynamodb-local-frankfurt/release
`dynamodb-local-sao-paulo`	https://s3-sa-east-1.amazonaws.com/dynamodb-local-sao-paulo/release
`dynamodb-local-oregon`	https://s3-us-west-2.amazonaws.com/dynamodb-local/release

We also need to specify the dependency that we just declared. In addition, we need to specify a task (here, `copyNativeDeps`) that copies the native SQLite libraries from the DynamoDB Local dependency to a directory (here, `build/libs`) to be included in the Java library path later. This step is necessary since DynamoDB Local uses SQLite4Java internally.

dependencies {
    testImplementation "com.amazonaws:DynamoDBLocal:${dynamoDBLocalVersion}"
}

task copyNativeDeps(type: Copy) {
    from(configurations.testImplementation) {
        include '*.dll'
        include '*.dylib'
        include '*.so'
    }
    into 'build/libs'
}

Finally, we specify that the tests depend on the `copyNativeDeps` task before it can run. We also set the `build/libs` as a system property so that it is discoverable by JUnit.

test {
    dependsOn copyNativeDeps
    systemProperty 'java.library.path', 'build/libs'
}

Utility Class

In my tests, I create a utility class to use as an extension of integration tests in order to keep the test classes free of DynamoDB setup crud. To skip to the code snippet, go here.

Before all of the test executions in a current test class, we need to create a DynamoDB proxy server, client, and tables.

@BeforeAll
fun setUpDB() {
    dynamoDBProxyServer = ServerRunner.createServerFromCommandLineArgs(
        arrayOf("-inMemory", "-port", "8000")
    )
    dynamoDBProxyServer.start()
    createAmazonDynamoDBClient()
    createTables()
}

`createAmazonDynamoDBClient` is a wrapper around the `AmazonDynamoDBClientBuilder` that builds the DynamoDB Client given that your properties are configured.

private fun createAmazonDynamoDBClient() {
    amazonDynamoDB = AmazonDynamoDBClientBuilder.standard()
        .withEndpointConfiguration(
            AwsClientBuilder.EndpointConfiguration(
                dynamoDBProperties.dynamodbRegion
            )
    ).build()
}

I have my properties configured in my `application.yaml` where it is then picked up by the `DynamoDBProperties` configuration class. Note that the following configuration class is not included in the test code but rather in the application code.

@Component
@ConfigurationProperties("amazon-dynamodb-config")
class DynamoDBProperties {
    lateinit var dynamodbEndpoint: String
    lateinit var dynamodbRegion: String
}

`createTables` creates the tables in your database. You should have an entity (in this example, `Event`) which is then mapped to a table.

private fun createTables() {
    val mapper = DynamoDBMapper(amazonDynamoDB)
    val tableRequest = mapper.generateCreateTableRequest(Event::class.java)
    tableRequest.provisionedThroughput = ProvisionedThroughput(1L, 1L)
    amazonDynamoDB.createTable(tableRequest)
}

Finally, we have a teardown function after all tests in a class are executed. In this teardown function, we delete the table and stop the DynamoDB proxy server.

@AfterAll
fun tearDownDB() {
    amazonDynamoDB.deleteTable("tutorial-table")
    dynamoDBProxyServer.stop()
}

Full Code Snippet

After putting it all together, your `DynamoDBTest` class should look something like below:

package com.stephsamson.tutorial

import com.amazonaws.client.builder.AwsClientBuilder
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper
import com.amazonaws.services.dynamodbv2.local.main.ServerRunner
import com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer
import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput
import com.stephsamson.tutorial.config.DynamoDBProperties
import com.stephsamson.tutorial.domain.Event
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.springframework.beans.factory.annotation.Autowired

/**
 * This class is to enable a local DynamoDB instance
 * to be able to test services that depend on it.
 *
 * Usually used as an extension of integration tests.
 */
open class DynamoDBTest {
    lateinit var amazonDynamoDB: AmazonDynamoDB

    lateinit var dynamoDBProxyServer: DynamoDBProxyServer

    @Autowired
    lateinit var dynamoDBProperties: DynamoDBProperties

    /**
     * Creates the DynamoDB proxy server, client, and tables.
     */
    @BeforeAll
    fun setUpDB() {
        dynamoDBProxyServer = ServerRunner.createServerFromCommandLineArgs(
            arrayOf("-inMemory", "-port", "8000")
        )
        dynamoDBProxyServer.start()
        createAmazonDynamoDBClient()
        createTables()
    }


    /**
     * Stop the DynamoDB proxy server.
     * Proactively deletes tables to prevent [ResourceInUseException].
     */
    @AfterAll
    fun tearDownDB() {
        amazonDynamoDB.deleteTable("tutorial-table")
        dynamoDBProxyServer.stop()
    }

    /**
     * Creates a DynamoDB Client provided that DynamoDB properties
     * are configured in the application.yml like so:
     *
     * amazon-dynamodb-config:
     *   dynamodb-endpoint: http://localhost:8000
     *   dynamodb-region: us-west-2
     */
    private fun createAmazonDynamoDBClient() {
        amazonDynamoDB = AmazonDynamoDBClientBuilder.standard()
            .withEndpointConfiguration(
                AwsClientBuilder.EndpointConfiguration(
                    dynamoDBProperties.dynamodbRegion
                )
            ).build()
    }

    /**
     * Creates task tables for [Event].
     */
    private fun createTables() {
        val mapper = DynamoDBMapper(amazonDynamoDB)
        val tableRequest = mapper.generateCreateTableRequest(Event::class.java)
        tableRequest.provisionedThroughput = ProvisionedThroughput(1L, 1L)
        amazonDynamoDB.createTable(tableRequest)
    }
}