Safe from the Losing Fight

When I started this post, I was planning on talking about Vapor integration testing. But in the thick of that, I found myself on a substantial rabbit trail talking about environment configuration. That’s because I wanted to change configuration values based on if I was in the testing environment or not. As it turns out, there’s not an established mechanism in Vapor for doing that that I like. So I built one that I like slightly better.

To clarify, what I mean by environment configuration is static global settings like: database URL, email server configuration, api keys, etc. That is, things that will change based on environment (e.g. production, staging, testing).

I’d like to have a few things for my environment configuration. First, I’d like all the configuration to be in one file, so it’s easy to see in one place. In my experience, most web frameworks provide this. They have config files that are specific to production, testing, and development. Second, I’d like the configuration to be statically checked at compile time, so I have confidence that all the values have been provided. I want to avoid the burden of having to run it and see it crashes at start up because something was not registered as a service. Finally, I’d like my global configure() method to not have conditionals in it. i.e. no ifs checking to see if the environment is testing and then doing something different than production; or changing the configuration code flow if the environment is release.

My plan for building this out is to first describe what I want the final environment configuration to look like in Swift. Then, I’ll figure out how to create and load that configuration based on the Vapor Environment. Finally, I modify the global configure() method to make use of the new configuration data type.

Start with where I want to end

I’ll first start by defining the testing configuration I’d like to use for running integration tests:

import Foundation
import Vapor
import FluentPostgreSQL

struct TestingConfiguration: ConfigurationType {
    let apiKey = "fake-api-key"
    let databaseConfig = PostgreSQLDatabaseConfig(
            hostname: "localhost",
            port: 5433,
            username: "myapp",
            database: "myapp-test",
            password: "password")
    let mailProvider = TestMailProvider()

    func configure(_ services: inout Services) throws {
        try services.register(mailProvider)
    }
}

This is as declarative as I can get it currently. The api key is fake since this is for tests, and the database is pointing to a test database on the localhost. I also want to point out I’m using native Swift types in my configuration like PostgreSQLDatabaseConfig. Finally, a fake test provider for the mailer is created and registered as a service. This will allow me to test sending email without actually sending mail to a real 3rd party service.

Ideally, I wouldn’t need the configure(_:) method here, and could do everything completely declaratively. But given the production mail provider that I’m using, I have to swap the provider out entirely. There’s not a configuration that can be given to the production provider that allows testing.

Another approach would be to register both testing and production mail providers and then use Vapor’s Config.prefer() to choose the correct one at runtime. But that would still require some sort of configuration method on the testing and production configurations to call Config.prefer() on the correct type. And, to me, that’s less clear than simply registering the provider that I want.

Now, here’s my production configuration:

import Foundation
import Vapor
import FluentPostgreSQL

struct ProductionConfiguration: ConfigurationType {
    let apiKey: String
    let databaseConfig: PostgreSQLDatabaseConfig

    init() throws {
        self.apiKey = try Environment.require("SECRET_API_KEY")
        guard let config = try PostgreSQLDatabaseConfig(url: Environment.require("DATABASE_URL")) else {
            throw Abort(.internalServerError, reason: "DATABASE_URL not configured correctly")
        }
        self.databaseConfig = config
    }

    func configure(_ services: inout Services) throws {
        try services.register(MailgunProvider(mailgunConfig: mailgun))
    }
}

The production configuration is a bit more complicated and less declarative because it needs to pull values out of the environment. It does use a small helper method I added to Environment to make fetching a required variable a little less verbose.

import Foundation
import Vapor

extension Environment {
    static func require(_ key: String) throws -> String {
        guard let value = get(key) else {
            throw Abort(.internalServerError, reason: "Missing value for \(key)")
        }
        return value
    }
}

Environment.require() works the same as Environment.get(), except it returns a non-optional, and throws an internal error if the value is missing.

I also have a DevelopmentConfiguration that I used for running the service locally. However, it’s not as interesting as the testing or production configurations, so I’m not showing it’s implementation here.

The final bit I need for my configurations is a protocol that they’re required to conform to. This is what allows the compiler to statically check that all the values are provided.

public protocol ConfigurationType: Service {
    var apiKey: String { get }
    var databaseConfig: PostgreSQLDatabaseConfig { get }

    func configure(_ services: inout Services) throws
}

The protocol simply declares the two config values that are used, and the configure() method. I’ll also note that it derives from the Service protocol so that I can register it with the dependency injection framework.

How to load the correct configuration based on the environment

At this point, I have three environment configurations (production, testing, and development) but I need a way of loading the correct one at runtime. I know that I want to pass the loaded environment configuration to the global configure() method for it to be used. So, for that reason, I chose to load it in the app(_:) method in app.swift.

import Vapor

/// Creates an instance of Application. This is called from main.swift in the run target.
public func app(_ env: Environment) throws -> Application {
    var config = Config.default()
    var env = env
    var services = Services.default()
    let configuration = try registerConfiguration(for: env, services: &services)
    try configure(&config, &env, &services, configuration)
    let app = try Application(config: config, environment: env, services: services)
    try boot(app)
    return app
}

All I did here was call a new method called registerConfiguration() and pass the result of that into the global configure() method. Here’s the definition of that:

private func registerConfiguration(for env: Environment, services: inout Services) throws -> ConfigurationType {
    if env.isRelease {
        return try register(ProductionConfiguration(env: env), services: &services)
    } else if env.isTesting {
        return register(TestingConfiguration(), services: &services)
    } else {
        return try register(DevelopmentConfiguration(env: env), services: &services)
    }
}

private func register<T: ConfigurationType>(_ configuration: T, services: inout Services) -> ConfigurationType {
    services.register(configuration, as: ConfigurationType.self)
    return configuration
}

This is where I choose which configuration to load and return. It’s not a clean mechanism, in that it asks a series of somewhat ambiguous questions to determine which configuration it should load. First, if the environment claims that it is a release environment, I choose the production configuration. If it’s not a release environment, I ask if it is a testing environment (kind of, see below). If it is neither production or testing, then I assume it should use a development configuration.

There are some tradeoffs to this approach. One benefit is that there will always be a valid environment configuration loaded. One downside is that this only supports three configurations. If I wanted to add a staging configuration, this code would have to change, in addition to creating a StagingConfiguration type. There’s also the matter of determining what kind of environment I have. isRelease is a Bool property on Environment, and multiple environments could have set that to true. That might be ok, but it does make the determination more ambiguous.

On the other hand the isTesting property is something I made up entirely:

import Foundation
import Vapor

extension Environment {
    var isTesting: Bool {
        return name == Environment.testing.name
    }
}

isTesting checks to see if the name is the same name as the static testing instance on Environment. So unlike production, the testing configuration will only be loaded for one specific environment, named a specific name. I did it this way because Environment only provides a name property and a isRelease flag. Other than the name, I didn’t have any way to differentiate between non-release environments.

An alternative way to do the loading would be by the name only. I’ve seen this with web frameworks for dynamic languages. If they have an environment named “spackle”, the framework will try to load a configuration file named something like “spackle.config” from a known location. This approach loses the static compile time checking (assuming the language had any), but it’s straight forward and flexible. In Swift, I could do the same, except use JSON as the config file format. That would mean I couldn’t use non-JSON (or non-Decodable) types in my configuration (like PostgreSQLDatabaseConfig), and dynamically selecting which providers to register would be trickier. A compromise for Swift could be to register the ConfigurationTypes in a global table by name. This would allow rich Swift datatypes to be used and static type checking to happen at compile time. However, the downside to all these approaches is if an environment name is given that doesn’t exactly match any of the configurations, the app will crash. That could be an acceptable tradeoff though, since one could argue that’s a progammer error.

What configure() looks like now

One of the goals I started out with was to remove all the conditionals in the global configure() method. Setting up all the necessary configuration is already a complicated enough task, and adding conditionals increases the complexity. The conditionals implicitly encode the differences in environment configuration, which makes discerning the differences difficult. It also is difficult to see when the differences were intentional vs. accidental. Since the configuration is implicit and spread out across a large file, it becomes trivial to make a mistake when updating an environment configuration.

Here’s a redacted version of my resulting global configure() method:

import FluentPostgreSQL
import Vapor
import Authentication
import Leaf

/// Called before your application initializes.
public func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services, _ configuration: ConfigurationType) throws {

    /// Register providers first
    try configuration.configure(&services)

    ...

    services.register(StaticApiKeyConfig(apiConfig: configuration), as: ApiKeyConfig.self)

    ...

    // Configure database
    var databases = DatabasesConfig()
    let databaseConfig = configuration.databaseConfig
    let database = PostgreSQLDatabase(config: databaseConfig)
    databases.add(database: database, as: .psql)
    services.register(databases)

    ...
}

First off, I let the environment configuration register any Services that it needs to. For the test of the method, it’s simply a matter of the configuration code pulling configuration values off the ConfigurationType. There are no conditionals, only straight line code.

It would be nice if Vapor provided some infrastructure for this out of the box. However, until then, this is how I create and load the environment configuration in Vapor. I first create the needed configurations as a native Swift types. Then I load them in the global app() method, before passing them to the global configure(). The end result is a statically checked configuration, all in one file, and a global configure() method with no conditionals.

The last time I got to yammering on, I mentioned I was building a side project in Vapor 3. When I got to setting up authentication for this project, I decided to use JWTs for reasons that are probably inexplicable to anyone with a clue.

Fortunately my cluelessness didn’t deter me. I quickly found that Vapor has an Authentication framework and a JWT framework. Unfortunately, I also found that they don’t work together out of the box, and there’s not much documentation on either. Cue my patented flailing around and trying to grok someone else’s code. Eventually, figured out something that worked.

Here’s my plan for marrying JWT with Authentication: first, figure out what to put in my JWT. Then I’ll need some code to authenticate that JWT, and some middleware to call it. Finally, I’ll use the results of the middleware to see if the holder of the JWT is authorized to perform the request they made.

Defining the JWT Payload

First, I had to define what my JWT looked like. A JWT is essentially a signed JSON payload, so I had to decide what should go in that payload. The main thing my app needed to know was the user making the request.

Here’s an abridged version of what I ended up doing:

import Foundation
import Vapor
import JWT
import Authentication

struct UserIDClaim: JWTClaim {
    var value: UUID
}

struct AuthJWTPayload: JWTPayload {
    func verify(using signer: JWTSigner) throws {
        // nothing for now
    }

    let uid: UserIDClaim
    let iat: IssuedAtClaim
}

The JWTPayload is a protocol defined in the JWT framework. Each property in the payload is supposed to be a “claim” that can be verified, and the verify method is supposed to do the verifying. However, since the verify method only takes the signer, there’s not much verifying it can actually do. I suppose it could verify that the IssuedAtClaim isn’t older than a given date.

Speaking of claims, there’s a standard set that the JWT framework defines. The IssuedAtClaim is an example. However, it’s possible to define your own, which is what I did with UserIDClaim. As the name suggests, it represents the user ID that the holder of the JWT operates as. All JWTClaims must be Codable, and define a mutable property named value.

The weird three letter property names appear to be a convention for claims. My assumption is it’s to keep the resulting JSON, and therefore the JWT, small.

Verifying a JWT Payload

Now that I had a payload defined, I needed something to verify it. The Authentication framework defines this kind of type as Authenticatable. I decided to be a bit fancy and define a protocol for JWT authenticatables before I wrote an implementation to authenticate my specific JWT payload.

import Foundation
import Vapor
import Authentication
import JWT

protocol JWTAuthenticatable: Authenticatable {
    associatedtype JWTType: JWTPayload

    static func authenticate(jwt: JWTType, on connection: DatabaseConnectable) -> Future<Self?>
}

The Authenticatable protocol itself has no constraints. I had JWTAuthenticatable declare a generic type that conforms to JWTPayload, and a static authenticate method to verify it. The return value will be an instance of the JWTAuthenticatable if the payload validates, nil otherwise.

Now I could define my actual JWTAuthenticatable type to verify the JWT payload:

import Foundation
import Vapor
import Authentication
import JWT

struct AuthJWTAuthenticatable: JWTAuthenticatable {
    typealias JWTType = AuthJWTPayload

    let user: User

    static func authenticate(jwt: JWTType, on connection: DatabaseConnectable) -> Future<AuthJWTAuthenticatable?> {
        return User.query(on: connection)
            .filter(\.id == jwt.uid.value)
            .first()
            .map { maybeUser in
                return maybeUser.map { AuthJWTAuthenticatable(user: $0) }
            }
    }

}

There are a couple of things to note here. First, if authenticate returns an instance of Authenticatable, the middleware (described later) is going to stash it on the request. Meaning Authenticatable is a great place to stash information about the authenticated “caller” because it can be accessed by other middleware or controllers. In my case, the caller represents a user, so I stash the user on the Authenticatable so my controllers can use it later.

The second thing of note is that I’m hitting the database to verify the JWT. This negates one of benefits of using a JWT, which is being able to verify a token without hitting the database. However, for my app I always need to have the user struct in memory, so it’s easier to go ahead and load it here. An alternative approach to avoid hitting the database during authentication would be:

struct AuthJWTAuthenticatable: JWTAuthenticatable {
    typealias JWTType = AuthJWTPayload

    let userId: UUID

    static func authenticate(jwt: JWTType, on connection: DatabaseConnectable) -> Future<AuthJWTAuthenticatable?> {
        return AuthJWTAuthenticatable(userId: jwt.uid.value)
    }
}

With this approach, I’d have to fetch the user in each controller that needed the full User.

Creating Some Middleware to Authenticate Incoming Requests

Now that I have a payload and the ability to authenticate it, I need to hook into the request handling and authenticate all incoming requests. As with most web frameworks, Vapor provides middleware as a solution for hooking into the request handling. With that in mind, here’s my middleware for authenticating incoming requests:

import Foundation
import Vapor
import JWT
import Authentication

struct JWTAuthenticationMiddleware<A: JWTAuthenticatable>: Middleware {

    func respond(to req: Request, chainingTo next: Responder) throws -> Future<Response> {
        // If the request is already authenticated, go to next middleware
        if try req.isAuthenticated(A.self) {
            return try next.respond(to: req)
        }

        // This parses the JWT string from the Authorization header
        guard let tokenString = req.http.headers.bearerAuthorization?.token else {
            // If there isn't a header, that's fine. Just go to the next middleware
            return try next.respond(to: req)
        }
        // Use my helper type to decode the string into my JWTPayload struct
        let token = try JWTSerialization<A.JWTType>().decode(tokenString, on: req)

        // Ask the JWTAuthenticatable if this is a valid JWT
        return A.authenticate(jwt: token, on: req).flatMap { a in
            if let a = a {
                // The JWT was valid, so this method call stashes the JWTAuthenticatable on the request
                try req.authenticate(a)
            }
            // Regardless of if the JWT is valid or not, go to the next middleware
            return try next.respond(to: req)
        }
    }
}

extension JWTAuthenticationMiddleware: ServiceType {
    static func makeService(for worker: Container) throws -> JWTAuthenticationMiddleware {
        return JWTAuthenticationMiddleware()
    }
}

I based my authentication middleware on the authentication middlewares defined in the Authentication framework. The main difference is my middleware uses my JWTAuthenticatable protocol to do the authentication. The goal of the middleware is to authenticate any Authorization header the request might have, and, if valid, store the resulting Authenticatable on the request for future use. Notably, the middleware doesn’t try to authorize the caller (i.e. it doesn’t require a valid Authorization header). This is because I don’t yet know at this stage what the request is trying to do, and what level of authorization that would require.

I factored out the JWT deserialization into a helper type to keep the middleware a bit more focused. Here’s what the deserialization code looks like:

import Foundation
import Vapor
import JWT

enum AuthJWTError: Error {
    case invalidDataEncoding
}

struct JWTSerialization<T: JWTPayload> {

    func encode(_ payload: T, on container: Container) throws -> String {
        let jwt = JWT(payload: payload)
        let config = try container.make(JWTConfig.self)
        let encoded = try jwt.sign(using: config.signer())
        guard let jwtString = String(data: encoded, encoding: .utf8) else {
            throw AuthJWTError.invalidDataEncoding
        }
        return jwtString
    }

    func decode(_ string: String, on container: Container) throws -> T {
        let config = try container.make(JWTConfig.self)
        let jwt = try JWT<T>(from: string, verifiedUsing: config.signer())
        return jwt.payload
    }
}

There’s not much to the decoding of JWTs. The JWT type is defined by Vapor’s JWT framework and can both sign and verify the signature of a JWT. The JWTConfig is a type that I created so I could abstract out the signing key and signing algorithm. I used Vapor’s dependency injection mechanism to fetch it. Here’s what it looks like:

import Foundation
import Vapor
import JWT

protocol JWTConfig: Service {
    func signer() throws -> JWTSigner
}

class HmacJWTConfig: JWTConfig {
    private let apiConfig: ApiConfigurationType

    public init(apiConfig: ApiConfigurationType) {
        self.apiConfig = apiConfig
    }

    func signer() throws -> JWTSigner {
        return try JWTSigner.hs256(key: apiConfig.jwtKey.hexEncodedData())
    }
}

The JWTConfig is a Service protocol that creates a JWTSigner (defined in the JWT framework) on demand. I only have one concrete implementation in my app: HmacJWTConfig. HmacJWTConfig is pretty simple; it creates a JWTSigner.hs256 signer (which uses HMAC-SHA256) with a key it pulls from the app config. The config implementation isn’t that interesting; on production it pulls a hex encoded key from an environment variable.

I’ve almost got my app where it can authenticate any incoming requests. The last piece is the configuration:

try services.register(AuthenticationProvider())
services.register(JWTAuthenticationMiddleware<AuthJWTAuthenticatable>.self)
services.register(HmacJWTConfig(apiConfig: configuration.api), as: JWTConfig.self)
...
middlewares.use(JWTAuthenticationMiddleware<AuthJWTAuthenticatable>.self)
...

This is simply registering all the services and middleware I defined earlier.

Using the Authenticatable

Finally, I’m ready to use the Authenticatable to see if a request is properly authorized. For my app, the requests coming in are user-scoped, meaning the user ID being acted on is in the request URL. A user should only be able to act on their own account. Therefore, my authorization method should make sure the request’s user ID parameter matches my JWT payload’s user ID.

Here’s what I do in my controller:

private func authorizedUser(_ request: Request) throws -> User {
    let user = try request.requireAuthenticated(AuthJWTAuthenticatable.self).user
    let userIdParameter = try request.parameters.next(UUID.self)
    guard userIdParameter == user.id else {
        throw Abort(.unauthorized, reason: "Unauthorized")
    }
    return user
}

The first line pulls out the AuthJWTAuthenticatable that my middleware stashed there. If it’s missing, then requireAuthenticated will throw an error. Next, I pull the request’s user ID parameter out. Finally, I compare the two to make sure they match. If they don’t, I abort with an unauthorized error.

This isn’t the only way to handle authorization. If I just wanted to make sure certain routes had authentication of some kind (i.e. requireAuthenticated doesn’t throw an error), then I could make use of GuardAuthenticationMiddleware defined in the Authentication framework.

And that’s about it for using JWTs to do authentication in Vapor 3. While Vapor has all the tools for JWT and authentication, they’re not put together in a way that works out of the box. For that reason, I’ve done a walkthrough here on how I put them together in my app.

Recently, I’ve been dabbling in server-side Swift with a side project. Currently I’m using Vapor 3, which has its own ORM called Fluent. With my little side project I had the desire to store some data in the database (PostgreSQL in this case) as JSONB. I knew PostgreSQL could handle this, but I didn’t know if Fluent could.

Unfortunately, documentation on Vapor 3 and Fluent is sparse. I couldn’t find any docs on how to add a JSONB column for the current version of Vapor. From random GitHub issue posts, it looked like there used to be a protocol you could implement to do it, but, alas, that protocol no longer exists. There is a Discord community for Vapor, but my question (along with most others) went unanswered.

After a lot of trail and error and digging through the source code, I figured out a way that works. The approach is to define a custom type that can serialize itself to a native PostgreSQL type. In my case I used JSONB, but it could theoretically be any PostgreSQL type.

Example

Here’s an example:

import Foundation
import Fluent
import FluentPostgreSQL

struct MyDataType: Codable, Equatable {
    let foo: Int
    let bar: String
}

extension MyDataType: ReflectionDecodable {
    static func reflectDecoded() throws -> (MyDataType, MyDataType) {
        return (
            MyDataType(foo: 42, bar: "towel"),
            MyDataType(foo: 42, bar: "mostly harmless")
        )
    }
}

extension MyDataType: PostgreSQLDataConvertible {
    static func convertFromPostgreSQLData(_ data: PostgreSQLData) throws -> MyDataType {
        let decoder = JSONDecoder()
        if let binary = data.binary {
            return try decoder.decode(MyDataType.self, from: binary[1...])
        } else {
            throw PostgreSQLError(identifier: "Null data", reason: "Beats me")
        }
    }

    func convertToPostgreSQLData() throws -> PostgreSQLData {
        let encoder = JSONEncoder()
        let data = try encoder.encode(self)
        return PostgreSQLData(.jsonb, binary: [0x01] + data)
    }
}

struct MyModel: PostgreSQLUUIDModel {
    var id: UUID?
    let name: String
    let myData: MyDataType
}

In this case, the myData property will end up as a JSONB column on MyModel‘s table. Now I’ll try to explain each of the required protocols as best as I understand them from the source code.

Codable

Codable is actually required for a couple of things. First, it’s required by PostgreSQLUUIDModel (and variants) for database serialization/deserialization. Second, it’s used in the implementation of the PostgreSQLDataConvertible protocol to actually convert my custom type to JSON.

ReflectionDecodable and Equatable

ReflectionDecodable and Equatable are used to implement reflection leveraging the Decodable machinery. Reflection is apparently required on all database model properties. Without this protocol, things built fine, but I would get runtime errors saying my type (e.g. MyDataType) didn’t conform to ReflectionDecodable. The protocol is an odd duck, because although there are headerdocs describing how to implement it, there’s zero discussion on how/why it is used. Here’s my attempt at the how:

Big picture, Fluent leverages Decodable to generate reflection data (i.e. key path and type) for the properties in a data type (e.g. MyDataType). It does this by implementing a custom Decoder that records each property’s key path and the type it encodes. Although Fluent only cares about the type in this case, it needs a placeholder value of the correct type in order to satisfy the Decoder protocol. That’s where ReflectionDecodable comes in; it provides the values that the custom Decoder can use.

There are two ways the custom reflection Decoder is used. The first is to generate reflection data for all the properties in the given data type. The second is to generate reflection data for a property at a given key path. The second way is more complicated and is why reflectDecoded() returns two values.

Fluent looks for the property at the given key path by iterating the data type (e.g. MyDataType) one property at a time. It does this by running the custom Decoder for each property “offset” (an integer). It marks the active property offset by value. This is why reflectDecoded() has two values, and the headerdocs make a big deal about them not being equal. The property at the active offset decodes using the first value in reflectDecoded(). All other properties decode using the second value in reflectDecoded(). After the custom Decoder is done decoding for a property offset, Fluent takes the returned value, and compares the value at the given key path to the first value in reflectDecoded() (which is where Equatable is used). If they are equal, then the correct property has been found and it’s reflection data can be returned.

tl;dr: ReflectionDecodable is necessary to provide placeholder values for a custom Decoder. The first value is used to mark a given property as “active” by a search algorithm. The second value is used in all other cases.

PostgreSQLDataConvertible

The final protocol is PostgreSQLDataConvertible. It is used to serialize and deserialize from PostgreSQL. In my case, I’m choosing to serialize to JSONB, but really any Postgres data type could be used. JSONB is just JSON data with a version byte at the beginning. That’s why the first byte is skipped when deserializing, and [0x01] is prepended when serializing.