As I’m getting further into my Vapor side project, I’m learning how writing Swift iOS/macOS apps is different from writing Swift server apps. One of the ways they are different is integration tests. For iOS apps, it’s usually done via automated UI testing. For Vapor, it’s API testing, or sometimes I see it called controller testing. Regardless of its name, I find myself writing more integration tests than unit tests for my server. I think there are two reasons for that:

  1. APIs tend to be well defined and easy to interact with, at least compared to UIs
  2. The integration tests give me a lot more confidence than unit tests that my server is behaving that way I intended it to

Of course, the downside is integration tests are much slower to run. So far, that’s a trade off I’ve been willing to make.

In any case, I want to talk about how I set up integration testing for my Vapor 3 app. My goal was to write test cases where calling the API under test was simple, and looked as much like real client code as possible. To accomplish that, I wrote some helper types called TestApplication and TestResponse to hide boilerplate code. I also created some fixtures to easily seed the database with data, along with some utility methods to reset the database after each test. Finally, I had to figure out how use Docker to stand up a PostgreSQL database I could run the tests against.

Example testcase

I’m going to start with a test case showing how I wrote my integration tests, then work backwards explaining the infrastructure I set up to support that.

As an example, I’m going to test an API that allows an authenticated user to invalidate all auth tokens that have been issued for their account, thereby forcing all devices to re-authenticate. Here’s the test:


class UserControllerTests: XCTestCase {
    ...
    func testResetTokens_isTrue_success() throws {
        let request = UserController.UserEnvelope(user: UserController.UpdateRequest(resetTokens: true))
        let response = try app.put("/api/users/\(user.id!)", headers: .withAuthorization(app.apiKey, jwt), body: request)
        XCTAssertEqual(response.status, .ok)

        let responseBody = try response.content(decodeTo: UserController.UserEnvelope<UserController.UserResponse>.self).user

        XCTAssertEqual(responseBody.id, user.id!)
        XCTAssertEqual(responseBody.email, user.email)

        let showResponse = try app.get("/api/users/\(user.id!)", headers: .withAuthorization(app.apiKey, jwt))
        XCTAssertEqual(showResponse.status, .unauthorized)
    }
    ...
}

Right now I’m using XCTestCase because it doesn’t rely on any external dependencies. I’ve been entertaining the idea of using Quick & Nimble so I can share more setup between tests, use custom matchers, and make the assertions a touch more readable. But for now, I just name my test functions with a specific pattern: API name, plus parameter values, plus expected results. I also toss in the app state if it’s relevant for the test.

The first line of the test is creating the request that I want to validate. The app specific request types are currently defined in the controller code, which is why there’s namespacing. The next line makes the request via TestApplication and then waits for a response. The request is performing a PUT on /api/users/:user_id with the request parameters encoded in the body. The .withAuthorization(app.apiKey, jwt) is a fixture on HTTPHeaders that creates headers with a valid authorization for this user.

Once the API performs the token reset, it returns the user object. The test validates this by decoding the response body and asserting the id and email match what’s expected. Then, to fully verify that all tokens were reset, the test makes another request (this time a GET request) with the existing token. It asserts that the request is rejected with an unauthorized status.

At this point there are a lot of undefined things that make this test function. I will eventually get to them all. But for now, I’ll start with the setup code that creates the user and JWT that are used by this test. Here’s what I got for that:


import Foundation
import XCTest
import Vapor
import FluentPostgreSQL
import JWT
@testable import App

class UserControllerTests: XCTestCase {
    var app: TestApplication!
    var connection: PostgreSQLConnection!
    var user: User!
    var jwt: String!

    override func setUp() {
        super.setUp()

        try! TestApplication.reset()
        app = try! TestApplication()
        connection = app.connection

        user = User.user(on: connection)

        let payload = AuthJWTPayload(uid: UserIDClaim(value: user.id!),
                                     iat: IssuedAtClaim(value: Date()))
        jwt = try! JWTSerialization().encode(payload, on: app.container)
    }

    override func tearDown() {
        super.tearDown()

        _ = user.delete(on: connection)

        connection.close()
    }
    ...
}

Since I’m using XCTest, I’m using the standard setUp() and tearDown() methods. In the setup I start by resetting the app via TestApplication.reset(); this resets the database. Then I create a TestApplication instance, which is the test’s interface into the app. I’ll need to add data to the database as well verify data was created, so I go ahead and create a database connection. Then I immediately use it create a user using the User.user(on:) fixture. The final bit of setup is creating a valid JWT for the user that can be used for making API calls. I wrote about using JWTs in Vapor API servers earlier.

My tearDown() is simple. I delete the user I created in the set up, and then close the database connection. This is a bit redundant in that I reset the app on each test, but I like to do it because it feels like good hygiene.

Now I have a complete test case, but there’s still a lot of infrastructure to build. The example test above used a couple of fixtures, so I’ll talk about those next.

Fixtures

My goal with fixtures is to provide a quick and easy way to create model data to test with. My fixtures tend to change a lot from app to app because they’re tied to the model types, and those change from app to app. That said, I do have a couple of patterns that I follow for fixtures. First, I prefer to make the fixtures static methods on the data type they’re creating instances of. In my experience, this scales a bit better than having one fixtures namespace or type that all fixtures are created from. Second, while I have the fixture methods take the model’s properties as parameters, I do prefer to default those parameters to sane values. I have found that reduces test maintenance when I need to add a property to a model type.

To provide a concrete example, here’s the user fixture used in example test above:


import Foundation
import Vapor
@testable import App

extension User {
    static func user(email: String = "bob.jimbob@example.com", on connection: DatabaseConnectable) -> User {
        let user = try! User(email: email)
        return try! user.create(on: connection).wait()
    }
}

There’s not much here, but I find even simple fixtures like this save time and boilerplate. The fixture creates a User instance, then saves it to the database. The most notable thing here is that the fixture waits on the database save to complete before returning. It does this by calling wait() on the promise.

The other thing that’s probably worth commenting on is my liberal use of force unwraps in the tests. Normally, I would not suffer a force unwrap to live (with a couple of exceptions). But I view tests as controlled environments that real users will never have to experience, and if something crashes it was probably a programmer error anyway.

TestApplication

Everything that I’ve written about so far has been pretty specific to my particular app. But there’s some testing infrastructure that I built that should be fairly portable. The next piece I want to talk about is the TestApplication.

The TestApplication represents the system under test. Its main function is to receive requests and send back responses. There’s some boilerplate code needed to make that ergonomic for tests. I chose to make TestApplication its own type (as opposed to an extension on Application) for a few reasons. First, I feel a bespoke type makes it clear which methods and properties are useful for tests, as opposed to production code. Second, having a separate type allows other testing properties and methods to have a convenient place to live.

With all that said, here’s the init and deinit of my TestApplication type:


import Foundation
import Vapor
import FluentPostgreSQL
@testable import App

class TestApplication {
    private let application: Application
    private let configuration: TestingConfiguration

    lazy var connection: PostgreSQLConnection = {
        return try! application.newConnection(to: .psql).wait()
    }()
    var container: Container {
        return application
    }

    init(arguments: [String] = CommandLine.arguments) throws {
        var env = Environment.testing
        env.arguments = arguments

        self.application = try App.app(env)
        self.configuration = try application.make(TestingConfiguration.self)
    }

    deinit {
        try! application.releaseCachedConnections()
        try! application.syncShutdownGracefully()
    }
}

The TestApplication type mainly wraps Vapor’s Application type. However, it also contains the TestingConfiguration, which is helpful for tests wanting to grab the API key, or fake services, or any other piece of configuration data. The TestApplication also exposes a database connection, lazily created, and a dependency injection container. These objects are needed to do anything interesting in a Vapor app.

The init creates a testing environment and sets the command line arguments on it. The command line argument functionality will be used later, when resetting the database between tests. But for now, init ensures the application will be stood up using the TestingConfiguration. After the application is created, a TestingConfiguration instance is pulled out and stored in a property.

I didn’t write a deinit method to start with. After I had written severals tests I started getting test failures because all the Postgres connections were being used, and after that no test could acquire a new one. So the deinit manually clears any cached connections (Vapor seems to keep a pool around), then waits synchronously until the Vapor app fully shuts down.

Making test requests

I’ve created my TestApplication now, and while it does some helpful things, I haven’t covered the main thing it does: send requests and receive responses. There were couple of pieces to implementing this. First, I needed to put together a method that can do basic send and receive. Then I added several methods to make it ergonomic to call from tests.

To start, I built the basic workhorse method:


class TestApplication {
  ...
  private func request<T: Content>(_ path: String, method: HTTPMethod, headers: HTTPHeaders, body: T?) throws -> TestResponse {
       let responder = try application.make(Responder.self)
       let httpRequest = HTTPRequest(method: method, url: URL(string: path)!, headers: headers)
       let request = Request(http: httpRequest, using: application)
       if let body = body {
           try request.content.encode(body)
       }
       return try TestResponse(response: responder.respond(to: request).wait())
  }
  ...
}

The first thing I’ll point out here is the method signature. It takes a path to the API to be called, the HTTP method to use, headers, and an optional body. The body can be any type as long as it conforms to Vapor’s Content protocol. The method can throw, and returns a TestResponse if it is successful. These parameters cover all the options that my tests care about.

The first thing request() does is create an object that conforms to the Vapor Responder protocol. This is the object that allows TestApplication to make requests and receive responses. Then request() creates the raw HTTPRequest using the parameters passed in, then the Request object based on that. The using parameter wants a dependency injection container, so I give it the application. If there is a non-nil body provided in the parameters, it is encoded into the request. Now that the request is fully created, it is given to the Responder which returns a promise to the response. Since this is for tests, I wait() on the response promise to be fulfilled before wrapping it in a TestResponse.

Although this method removes a lot of the boilerplate needed to make a request, I felt like it could be a bit cleaner. I figured that the HTTP method could be the method name instead of a parameter. So I added some methods that do that:


class TestApplication {
  ...
  func get<T: Content>(_ path: String, headers: HTTPHeaders = HTTPHeaders(), body: T?) throws -> TestResponse {
      return try request(path, method: .GET, headers: headers, body: body)
  }

  func put<T: Content>(_ path: String, headers: HTTPHeaders = HTTPHeaders(), body: T?) throws -> TestResponse {
      return try request(path, method: .PUT, headers: headers, body: body)
  }

  func post<T: Content>(_ path: String, headers: HTTPHeaders = HTTPHeaders(), body: T?) throws -> TestResponse {
      return try request(path, method: .POST, headers: headers, body: body)
  }

  func delete<T: Content>(_ path: String, headers: HTTPHeaders = HTTPHeaders(), body: T?) throws -> TestResponse {
      return try request(path, method: .DELETE, headers: headers, body: body)
  }
  ...
}

These methods also default the headers to an empty set of headers, in the cases where tests don’t care about them. These methods are pretty ergonomic, but there’s one more case I needed to handle.

Some of my APIs don’t send a body in the request. The workhorse request() method can handle that; the caller just passes in nil in that case. However, because body is a generic type, the compiler needs a type — any type — to compile the call. nil by itself won’t give the compiler enough information to infer the type. I needed an instance of an optional type set to nil to satisfy the type inference. I also wanted to avoid forcing the test from having to pass that instance in. If a body parameter wasn’t provided, the TestApplication methods should assume there isn’t one.

Fortunately this can be done easily, albeit with a bit more boilerplate:


class TestApplication {
  ...
  struct Empty: Content {
     static let instance: Empty? = nil
  }

  func get(_ path: String, headers: HTTPHeaders = HTTPHeaders()) throws -> TestResponse {
     return try get(path, headers: headers, body: Empty.instance)
  }

  func delete(_ path: String, headers: HTTPHeaders = HTTPHeaders()) throws -> TestResponse {
     return try delete(path, headers: headers, body: Empty.instance)
  }
  ...
}

I defined a placeholder type called Empty and conformed it to Vapor’s Content protocol. The protocol conformance was to satisfy request()‘s constraints on the body parameter. Then I declared an optional static instance of the type, and set it to nil. I could then use this static instance to represent an empty request body.

Now that I had the Empty.instance instance created, I used it to remove the body parameter. Here, I’ve only provided versions of the get and delete methods without a body, but the same technique works for put and post.

At this point I could now make requests like I showed in my example test. But how about validating the response I got back?

TestResponse

The main goal of TestResponse was to make it easy for tests to validate the response. I chose to wrap Vapor’s Response type instead of providing an extension on it for the same reason TestApplication wraps Application. The main things my tests want to do with a response is: verify the HTTP status, verify the body, and maybe verify a header or two.


import Foundation
import Vapor

struct TestResponse {
    private let response: Response

    init(response: Response) {
        self.response = response
    }

    var status: HTTPResponseStatus { return response.http.status }

    func content<T: Decodable>(decodeTo type: T.Type) throws -> T {
        return try response.content.decode(type).wait()
    }

    func header(_ name: HTTPHeaderName) -> String? {
        return response.http.headers.firstValue(name: name)
    }
}

TestResponse is simple. It exposes the HTTP status in status, provides a synchronous way to decode the body via the content() method, and looks up a given header with the header() method. For more complicated tests it might need to expose more information, but this works for all my tests.

Resetting the database

So far, I’ve been mainly concerned with sending requests, receiving responses, and validating them. But for tests to work, I needed to be able to reset the database between test runs. Otherwise tests would interfere with one another.

First, I needed to set up some Fluent commands in my configuration. These added command line arguments that can reset the database for me.


public func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services, _ configuration: ConfigurationType) throws {
  ...
  var commandConfig = CommandConfig.default()
  commandConfig.useFluentCommands()
  services.register(commandConfig)
  ...
}

There’s not much to this. Toward the end of my configure() method, I created a default Vapor CommandConfig, told it to use Fluent commands, then registered it as a service.

Now that the Fluent commands were set up, I needed to add a couple of methods to TestApplication to have it reset the database.


class TestApplication {
  ...
  func run() throws {
     try application.asyncRun().wait()
  }

  static func reset() throws {
     try TestApplication(arguments: ["vapor", "revert", "--all", "-y"]).run()
     try TestApplication(arguments: ["vapor", "migrate", "-y"]).run()
  }
  ...
}

The run() method looks a bit odd by itself. It synchronously runs the application. Which turned out to be handy, if it has some commands to run from the command line.

The reset() static method provided those command line arguments. First, it creates an instance of the app that reverts the database, and runs it. Then, it it creates an instance of the app that runs all migrations.

That testing database

I’ve blissfully ignored that all this needs a real test database to talk to. I could have set up a local Postgres instance on my laptop (which I’ve done before), but that doesn’t scale well if I have multiple apps. So I’ve been using Docker to handle standing up a test database.

I already had Homebrew installed, so I used it to install Docker:


brew cask install docker

Then I started a Postgres instance running in Docker for the test suite to talk to with this command:


docker run --name myapp-postgres-test -e POSTGRES_DB=myapp-test -e POSTGRES_USER=myapp -e POSTGRES_PASSWORD=password -p 5433:5432 -d postgres

This names the instance myapp-postgres-test so I can refer to it later. The -e flag sets environment variables, and POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD values all match the values I set in my TestingConfiguration. The -p flag tells Docker to publish Postgres’s default port (5432) on port 5433, which is where my TestingConfiguration was pointing. The -d flag tells Docker to run the container in the background, and postgres argument is the Docker image to use.

At this point, I can run all tests in Xcode (menu: Product > Test) or run vapor test from the command line.

Sometimes during development I changed the database schema in way that my migrations didn’t catch. In these cases, I was lazy, and I just reset the Docker image:


docker stop myapp-postgres-test
docker rm myapp-postgres-test

This first stops the container, then removes it, leaving me free to re-execute the docker run command above to stand up a fresh Postgres database.

As a final testing infrastructure tip, I’ll start by observing that Swift on Linux doesn’t currently have reflection capabilities. That means test cases have to be manually given to the testing framework as a parameter. I find this is tedious and error prone, especially given how forgetful I am. Therefore, I let the Swift Package Manager autogenerate the necessary information for me: 


swift test --generate-linuxmain

Unfortunately the generated code is static, so I have re-run the command anytime I add or remove a test case.

Also, I’m terrible at remembering all these Docker and Swift Package Manager commands, so I added them to my app’s README.md file.

Conclusion

With API servers, I find writing integration tests more helpful than unit tests. However, in my experience there’s some boilerplate code needed to make creating integration tests in Vapor comfortable. My approach is create a TestApplication class to model the system under test, and make to it ergonomic to send requests and synchronously receive responses. Likewise, I use a TestResponse type to make validating responses easier. I also leverage fixtures for seeding the test database, and a utility function for resetting the database between test runs. Finally, I make use of Docker to make it convenient to stand up and reset the Postgres testing database.