Archive for the 'Programming' Category

Building a declarative router for Swift Vapor

As I’ve been using Swift Vapor 3, one of the things that I felt could be better was the way routes were declared. So I built a framework to meet my Vapor routing needs. If you’d like to try it out, go to the Github page and checkout the README. For the rest of this post, I’m going to talk about what my goals were, and highlight parts of the implementation that were interesting to me.

But first, to whet your appetite, here’s a tiny example of declaring routes using my framework:


import Vapor
import RoutingTable

public func routes(_ router: Router) throws {
    let table = RoutingTable(
        .scope("api", middleware: [ApiKeyMiddleware()], children: [
            .resource("users", parameter: User.self, using: UserController.self, children: [
                .resource("sprockets", parameter: Sprocket.self, using: SprocketController.self),
                .resource("widgets", using: WidgetController.self)
            ]),
            .resource("sessions", using: SessionController.self),
            .post("do_stuff", using: StuffController.doStuff)
        ])
    )
    table.register(routes: router)
}

Goals

I had four goals when creating my routing framework:

  1. All routes in one file

    One of the challenges I ran into was determining what all the routes in the app were and how they fit together. It is possible to print out all the routes by running a command line tool, but that didn’t help me with finding where the associated code was.

    I also attempted to take advantage of RouteCollections at one point in order to make my routes() method less verbose. It did improve the verbosity, but at the expense of all the routes in one place. Ideally, I’d like to have my cake and eat it, too: all routes in one file, but expressed concisely.

  2. Hierarchical declaration

    Routes are hierarchical by nature, and I’d like to declare them that way. By that, I mean building a tree data type that is hierarchical in Swift syntax when declared, as opposed to calling a series of methods that build up a tree at runtime.

    I see a couple of benefits from making the route declaration hierarchical. First, it’s easier for me to see how the endpoints fit together or relate to one another. I can see the hierarchy in the code syntax itself, instead of parsing method calls to build up the hierarchy in my head. Second, it can reduce boilerplate code by inheriting configuration from the parent to the child.

  3. Re-usability of controllers

    By re-usability of controllers, I mean a controller can be used in more than one place in the routing hierarchy. For example, maybe a controller implements managing sprockets. It could be exposed in one place in the routing hierarchy for normal users, but also in a different place for admin users. Part of making this useful would be allowing the routing declaration to specify which controller endpoints are available at each place in the hierarchy. e.g. the admin sub-hierarchy should allow the DELETEing of sprockets, but the normal user’s sub-hierarchy shouldn’t.

    Being re-usable implies controllers don’t know where they are in the route hierarchy. To me, this makes sense because of my iOS/macOS background. In that context, view controllers don’t know where they appear in the app. Instead, app architecture relies on Coordinators (or a similar pattern) to know when and where to display the view controllers. Because of the separation of concerns, view controllers can be re-used in multiple places of the app. I think of API/web controllers in the same way.

  4. Use higher level concepts

    In my experience, few things reduce boilerplate code and increase readability like the introduction of higher level concepts. In the case of routing, I’m thinking about scopes and resources.

    Scopes add the ability to group routes logically, so that the code maintainer knows they fit together to accomplish a common goal. It also means routes in a scope can inherit the same path prefix and/or middleware. Some examples of scopes could be an API scope or an admin user scope.

    Resources allow me to describe REST-like resources in a succinct way. Although resource declaration can be succinct, the code maintainer can infer a lot about it. That’s because REST-like resources are known to support certain subpaths and HTTP methods that behave in specific ways. So although fewer things are declared about a resource, more is known about it than if I had declared each route individually.

The Results

Based on these goals, I came up with an idealized version of what I wanted route declaration to look like:


scope "api", [ApiMiddleware.self] {
    resource "users", parameter: User.self, using: UserController.self {
        resource "sprockets", using: SprocketsController.self
    }
}

What I like about the above is it has just enough punctuation and keywords to make it readable, but no more than that is unnecessary. It also makes use of curly braces to denote hierarchy. When I’m reading Swift code, curly braces make my brain think parent/child relationship in a way other punctuation doesn’t. It also seems to help make Xcode do sane things with indentation.

Here I’m going to admit that much of my inspiration for what routing could be came from Elixir’s Phoenix framework, and its routing library. I feel like its route declaration is very close to my ideal. In addition, it supports more features, including the ability to generate full URLs from a named resource.

Unfortunately, I couldn’t achieve my ideal in Swift. The example I gave above isn’t valid Swift, nor was there a way to extend Swift to support it. A lot of Phoenix’s elegance and power comes from Elixir’s hygienic macros, which Swift doesn’t have.

Instead, here’s the closest I could come in Swift:


.scope("api", middleware: [ApiKeyMiddleware()], children: [
    .resource("users", parameter: User.self, using: UserController.self, children: [
        .resource("sprockets", parameter: Sprocket.self, using: SprocketController.self)
    ]),
])

It has a lot more punctuation and keywords than is really necessary to convey what I want. It also uses square brackets for arrays to denote hierarchy, which are a bit clumsy especially when used in Xcode. But given Swift’s limitations, I feel like it comes pretty close.

Interesting Implementation Bits

When implementing my declarative router I ran into some implementation hurdles that I thought were interesting enough to write down.

Verbosity reduction

One of my stated goals was to reduce the verbosity needed to declare all my routes. Some of the reduction came for free just by using higher level concepts like scopes and resources, and making the declarations hierarchical so common configuration could be shared. But all that space savings could be undone if I messed up the declaration API. I paid particular attention to leveraging Swift’s type inference and default parameters.

I realized early on that I needed a tree that was homogenous in type, but polymorphic in behavior. There are three types that could appear in a routing declaration: a scope, a resource, and a raw endpoint (i.e. a GET, PUT, PATCH, POST, or DELETE). Each of those has its own properties, and handles registering routes differently. Of those, both scopes and resources could have children, which could themselves be scopes, resources, or raw endpoints. That left me with a couple of options.

The first option that I considered was using an enum to represent the three types (scope, resource, and raw endpoint). Since Swift enums can have associated data, each value could contain all their necessary properties. However, enums had a couple of problems. First, they don’t allow default values on construction. Which means each declaration would have to specify all values even if they weren’t used. Second, eventually each enum value would have to register all the routes it represented, and since each enum type had to do that differently, there would be a giant switch statement somewhere. That didn’t seem elegant to me, so I abandoned that approach.

The second option was to declare a common protocol (e.g. Routable) and have scope, resource, and raw endpoint types conform to that protocol. Then I had scopes and resources declare their children to be of type Routable so type homogenous trees could be built. That turned out to mostly work. The problem I ran into was the raw endpoints were more verbose than I wanted. For example:


Scope("api", middleware: [ApiMiddleware()], children: [
    RawEndpoint(.post, "do_stuff", using: StuffController.doStuff)
])

I felt having the typename RawEndpoint in the declaration was unnecessary and uninteresting. The important bit was the HTTP method, but that was obscured by the typename. My next thought was use the HTTP method name as the typename (e.g. Post, Get, etc). This worked, but at a cost. First, it meant I had five different types that all did the same thing, except for one parameter. Second, the HTTP method names are common words and had to exist at the top level scope. This made me worried about typename collisions.

I tried to fix those problems by adding static methods to my protocol as a shorthand way to create the raw endpoint type like so:


extension Routable {
    static func get<T>(_ path: String..., using controller: T) -> Routable {
        return RawEndpoint(.get, path, using: controller)
    }
}

However, when I tried to use the static methods in a declaration:


Scope("api", middleware: [ApiMiddleware()], children: [
    .post("do_stuff", using: StuffController.doStuff) // ERROR
])

Swift complained, seemingly because it couldn’t infer the type because the static methods were on a protocol. I could have specified the full type name to the method, but I felt that would have made the declaration too verbose. But I thought I was close to something that would work. I just needed the type inference to work on the static methods.

That lead me to the final option that I actually used. My hunch was I needed use a concrete type rather than a protocol in my declaration API. That would allow me to use static methods in the declaration and Swift’s type inference would work. To put it another way, I could make this work:


Scope("api", middleware: [ApiMiddleware()], children: [
    .post("do_stuff", using: StuffController.doStuff)
])

If children was declared to be an array of a concrete (i.e. non-protocol) type, and if post() were a static method on that concrete type.

The challenge now was I needed two seemingly opposed concepts. I needed each declaration item (i.e. scope, resource, raw endpoint) to be polymorphic since they each should act differently based on their type. I had achieved that via making them conform to a common protocol. However, in order to make Swift type inference happy, I needed a concrete type.

So I used type erasure, kind of. I wrapped the protocol Routable in a struct called AnyRoutable. It works like a type erasure datatype in that it implements the Routable protocol by calling the methods on the Routable instance it contains. This gave me a single concrete type while still allowing polymorphism.

To make this work, I essentially made AnyRoutable a facade to the rest of the framework. Every node in the routing tree would be declared as an AnyRoutable, which could, internal to the framework, wrap the correct declaration item type. To make building an entire tree from one type possible, I added static methods on AnyRoutable that created each declaration type: scope, resource, and each of the HTTP methods. For example, something like:


struct AnyRoutable {
    static func get<T>(_ path: String..., using controller: T) -> AnyRoutable {
        return AnyRoutable(RawEndpoint(.get, path, using: controller))
    }
}

The trick was I had the static methods deal only in AnyRoutable; children were declared as them, and the return types were AnyRoutable. Since all parameters and return types were a concrete type, Swift could easily infer the type, and the static methods could be called without them. Implementation wise, the static methods simply created the appropriate Routable subtype, then wrapped it in AnyRoutable. This had the added bonus of only needing to make AnyRoutable public in the framework. The Routable implementations for resource, scopes, and endpoints stayed hidden.

Although it took me a while to reach the final implementation, the pattern seems generally useful. It allows polymorphism in the implementation, while only exposing one concrete type to client code, which means type inference can be used. I suspect there might be other options for solving this problem. For example, I never tried a class hierarchy and I wonder if that could be made to work. However, I’m pretty happy with AnyRoutable since I got a facade pattern out of it as well.

Resource reflection

After designing the API interface, the next most difficult thing was figuring out how to implement controllers that managed REST-like resources. In a nutshell, if a router declares a resource, my framework should be able to look at the implementing controller and determine what verbs (i.e. index, show, update, create, delete, new, or edit) it supports, and automatically declare those routes. I wanted this to require as little boilerplate as possible. As a bonus, I wanted to determine the verb support at compile time.

I quickly realized that there would have to be some overhead no matter what, just because of the limitations of Swift. Swift has no way of asking “does this type implement this method” outside of having that type declare conformance to a protocol that requires the method. It doesn’t have a fully dynamic and reflective runtime like Objective-C, nor a structural type system. So I accepted that resource controllers would have to declare conformance to a protocol for each of the verbs it supported.

But even accepting that limitation, I still wanted to determine which verbs were available at compile time. Since the controller declared conformance to a protocol, and a generic method could require parameter conformance to a protocol, for a long time I held out hope this this was possible. For example, I could do this:


func indexRoutes<ControllerType: ResourceIndexable>(_ controller: ControllerType) -> [Route] {
    // generate route for index verb
} 

func indexRoutes<ControllerType>(_ controller: ControllerType) -> [Route] {
    return []
} 

class MyController: ResourceIndexable {
}

let controller = MyController()
let routes = indexRoutes(controller) // calls the correct method

The issue I ran into was trying to put all the verb routes together. The method that put all the routes together would have to know about all the resource verb protocols it conformed too. That’s because if the method didn’t require conformance to a verb protocol, Swift would act like it didn’t conform to that protocol regardless of it actually did or not. So for this to work, I would have to implement a generic method for every combination of the seven verb protocols that could occur. That seemed excessive to me. In the end, I simply had the code query at runtime which protocols the controller implemented.

This part was interesting to me because it seems like it should be solvable problem. However, in its current state, Swift doesn’t appear able to overcome it. I do wonder if a hygienic macro system would make it feasible.

Dependency injection woes

The next two implementation struggles relate to the implementation details of Vapor itself, and not something inherent to the issue of building a declarative router. But they still contained valuable learnings.

At a base level, the router connects routes with a controller that will handle requests to that route. To make the route declaration as concise as possible, I only required the type of the controller, as opposed to a live instance that might have be constructed and/or configured. I decided I would instantiate the controller later, when registering the route, using Vapor’s dependency injection system. The problem was when it came to register the route, Vapor’s dependency injection system was in the wrong state.

Vapor’s dependency injection effectively has two phases. The first phase is the setup phase, where it registers types it calls “services” which can be instantiated later, in the second phase. In the second phase, a “container” exists, and can be used to instantiate any of the service types registered in the first phase. When the router is being initialized, the system is in the setup phase, and can’t instantiate any of the controller types because there’s no container.

I considered using my own custom protocol on controllers that allowed them to be constructed without a dependency injection container. However, after trying that out, it seemed surprising in that everything else uses Vapor’s DI system. Plus my custom protocol would be more restrictive; it wouldn’t be able to allow custom init parameters to the controller (unless all controllers needed them), nor would it offer access to the DI system to allow construction of said parameters.

In the end, I was able to defer controller allocation until the first route was actually executed. By then, containers were available. Further, Vapor’s DI system took care of caching the controller for me.

This problem was interesting because I’ve run into it a few times in Vapor. I need to be on the lookout for different patterns to work around not having a DI container available when I need it.

Testing difficulties

The final issue I ran into was trying to unit test my framework. Since it’s implemented in Swift, my go to method is to define all dependencies as protocols, then for my tests to create fakes I can manipulate and observe.

Unfortunately, most of the Vapor types I had to interact with weren’t easily fakeable. The big one I needed to fake was Vapor’s Router protocol. Being a protocol, I thought it would be a slam dunk. Unfortunately, all the interesting methods on Router were in protocol extensions, meaning my testing fakes could not override those methods.

What I ended up doing was defining my own protocol for the methods on Router that I used, then using that everywhere in my code. Since the methods were declared in the protocol, they were overridable by my testing fakes. This allowed me to do the unit testing I wanted. That only left the issue of how to use Vapor’s Router in the non-testing version of the framework.

Indirection solves a lot of problems, this being another example. I declared a concrete type that conformed to my router protocol that my framework could use. I then had it wrap a Vapor Router, and implement the protocol by calling the corresponding methods on Router. Then, at the top level object, I hid the wrapping of Router behind a helper method, so it looked like my framework dealt with Routers natively.

The lesson I take from this is testing when relying on 3rd party frameworks is hard. Unit testing with UIKit and AppKit types is no different. Also, defining my own protocol and wrapping a 3rd party protocol in a concrete type to conform to it seems like a repeatable strategy.

Conclusion

When using Swift Vapor’s routing API, I discovered I wanted a few more things than it offered. Namely, I wanted all routes declared in one file, a hierarchal declaration, re-usability of controllers, and higher level concepts like scopes and resources. In building a framework to support these concepts, I ran into a few implementation concerns. The first was learning to design an API in such a way to reduce verbosity. The others were trying to determining protocol conformance at compile time, working around two-stage dependency injection limitations, and trying to test my code while not having fakeable protocols from 3rd party frameworks.

Sending email from Swift Vapor

Like most services these days, my Vapor side project needs to send welcome emails when a user signs up for the service. While I could write my own SMTP client, it would still require a lot of DNS setup to make sure those emails made it through various spam countermeasures. Given this is a side project, and sending email is a minor component of it, that didn’t seem worthwhile to me. So I decided to use a 3rd party service who would take care of all of that.

I chose Mailgun for a couple of reasons:

  1. I’m hosting my side project on Heroku, and they provide a convenient Mailgun add on that I can provision.
  2. Mailgun offers a free tier, appropriate for my side project
  3. There’s already a Swift package for Mailgun’s API.

The Swift package for Mailgun got me most of the way there. However, I still needed a bit of infrastructure to render the emails, and then to validate the emails were correctly sent in my integration tests.

In this post I’m going to cover how I went about generating the email content, sending the email to Mailgun, and finally how I ensured I could test it all. I’m not going to cover how to set up Mailgun and all the DNS stuff because that’s very app specific, and is documented in Mailgun’s docs already.

Example use

I find it helpful when building a subsystem to sketch out what the use site will look like. It helps me determine if I’m building something that will meet my needs, and gives me a concrete target to build toward. With that in mind, I thought the ideal for my project was to define an email template for each kind of email I wanted to send, have a method that could render that down to a final email to send, then send it via an abstract email delivery service. The email body allows both text and html, and the body template was defined in Leaf.

Here’s how that looked in code:


struct WelcomeData: Encodable {
    let verifyLink: String

    init(user: User) {
        self.verifyLink = "myapp://verify/\(user.verifyToken)"
    }
}
...
let template = EmailTemplate(from: "donotreply@myapp.com",
                             to: email,
                             subject: "Welcome to MyApp!",
                             bodyTemplate: "WelcomeEmail")
let welcomeData = WelcomeData(user: user)
return template.render(welcomeData, on: container).flatMap { message -> Future<Void> in
    let deliveryService = try container.make(EmailDeliveryService.self)
    return deliveryService.send(message, on: container)
}

The first step is to create the EmailTemplate for the email I want to send and render it. The from, to, and subject properties are Strings that will be passed through unchanged to the final email. The bodyTemplate is the base name for the Leaf templates that will be rendered in the render() method. WelcomeData is the Leaf context for the templates; it defines anything that the the Leaf template will need. I like to think of it as a view model. It takes a model object and transforms it into values that the view (i.e. the Leaf template) needs. In this example, the WelcomeEmail template needs an email verification link, so WelcomeData constructs that based on a token assumed to be on the User model. To render the EmailTemplate into something that can be sent, render() is called, passing in WelcomeData.

The second step is to send the resulting EmailMessage. The dependency injection framework is asked to produce a EmailDeliveryService object, which can send an EmailMessage. EmailDeliveryService is a protocol, meaning it can be swapped out later, without the client code knowing or caring. This enables testing fakes to be injected during tests, as well as making it possible to move to a new email service should I ever decide to do that.

That covers the Swift code for creating and sending the email. I still need to define the Leaf templates though. I want to send both plain text and HTML MIME parts in my email, so regardless of the user’s email app they’ll see something reasonable. Since the email body template parts are Leaf templates, I put them in the standard location: Resources/Views. I also follow a naming convention so EmailTemplate.render() can find them at runtime.

Here’s the contents of WelcomeEmail.html.leaf:


<html>
<head></head>
<body>
<p>Hello!</p>

<p>Welcome to MyApp!</p>

<p><a href="#(verifyLink)">Verify your email address</a></p>

<p>The MyApp Team.</p>
</body>
</html>

And the contents of WelcomeEmail.text.leaf:


Hello!

Welcome to MyApp!

Verify your email address by clicking on this link: #(verifyLink)

The MyApp Team.

Both templates represent the same email, but the content changes based on the format they’re being rendered into. The #(verifyLink) placeholder is filled in with value in WelcomeData.verifyLink.

Now that I’ve defined my target API, I can start building the implementation.

Rendering email with Leaf

First I need to define what an email message is, because it is the type everything else is dependent on. The EmailTemplate needs to render to it, and EmailDeliveryService needs to send it. I decided to define my own types for this because it reduces coupling on a specific service, plus it more accurately represents what my app thinks an email is. Also, the necessary types are pretty simple, so I did’t think they’d increase my maintenance burden any.

Here’s my definition:


struct EmailBody {
    let text: String
    let html: String
}

struct EmailMessage {
    let from: String
    let to: String
    let subject: String
    let body: EmailBody
}

My app’s idea of an email is simple. It has a from, to, subject, and body fields. The only thing that might look out of the ordinary is the the body has two parts: HTML and plain text. My app doesn’t care about attachments or other features, so they don’t show up here.

With email defined, I could create the EmailTemplate which takes care of rendering Leaf templates down to a EmailMessage. I started by defining the properties of the EmailTemplate:


import Vapor
import Leaf
import TemplateKit

struct EmailTemplate {
    private let from: String
    private let to: String
    private let subject: String
    private let bodyTemplate: String

    init(from: String, to: String, subject: String, bodyTemplate: String) {
        self.from = from
        self.to = to
        self.subject = subject
        self.bodyTemplate = bodyTemplate
    }
    ...
}

The template is the same as the EmailMessage with the exception of bodyTemplate, which is the base name of the Leaf templates for the email body. Most of the work of EmailTemplate is to convert the bodyTemplate into a EmailBody. The top level render method looks like:


struct EmailTemplate {
    ...
    private static let htmlExtension = ".html"
    private static let textExtension = ".text"

    func render<E>(_ context: E, on container: Container) -> Future<EmailMessage> where E: Encodable {
        let htmlRender = render(bodyTemplate + EmailTemplate.htmlExtension, context, on: container)
        let textRender = render(bodyTemplate + EmailTemplate.textExtension, context, on: container)
        return htmlRender.and(textRender)
            .map { EmailBody(text: $1, html: $0) }
            .map { EmailMessage(from: self.from, to: self.to, subject: self.subject, body: $0) }
    }
    ...
}

The render() method takes the Leaf template context and a dependency injection container, and returns the promise of an EmailMessage. The implementation relies on a helper render() method that renders one Leaf template at a time. This top level render() calls it twice: once for the plain text template, and once for the html template. It uses the and operator to let the template renders run concurrently if possible, then combines the results into an EmailBody, before a final map that mixes in the from, to, and subject fields.

The helper render() method is similarly straight forward:


enum EmailError: Error {
    case invalidStringEncoding
    case emailProviderNotConfigured
}
...
struct EmailTemplate {
    ...
    private func render<E>(_ name: String, _ context: E, on container: Container) -> Future<String> where E: Encodable {
        do {
            let leaf = try container.make(LeafRenderer.self)
            return leaf.render(name, context).map { view in
                guard let str = String(data: view.data, encoding: .utf8) else {
                    throw EmailError.invalidStringEncoding
                }
                return str
            }
        } catch let error {
            return container.future(error: error)
        }
    }
}

The helper method here takes the full Leaf template name, its context, a dependency injection container and returns a promise of String, which is the fully rendered body. To achieve this, it creates a LeafRenderer and asks it to render the template. This results in view data, which it decodes into a String and returns. If any error is thrown, it’ll convert it into a rejected promise for convenience.

The email template rendering is fairly simple, but creating the EmailTemplate type reduces the boilerplate code needed for sending an email.

Sending email

I now have a fully rendered email message, and I need to send it. As I mentioned up top, I’m used a 3rd party Swift package to actually talk to Mailgun’s API. However, I wanted an easy way to inject testing fakes, and the ability to swap out email services later if necessary. So I’m first going to show how I integrated the Swift Mailgun package, then how I abstracted it with a generic interface that can be faked.

Since it’s a Swift package, I added it to my Package.swift file:


    dependencies: [
        ...
        .package(url: "https://github.com/twof/VaporMailgunService.git", from: "1.1.0"),
        ...
    ],
    ...
    targets: [
        ...
        .target(name: "App", dependencies: [
            ...
            "Mailgun",
            ...
            ]),
        ...
    ]

Running vapor update from the command line pulled down the package and updated all my dependencies. I decided to use a Provider to set up the Mailgun package in my app:


import Vapor
import Mailgun

struct MailgunProvider: Provider {
    private let config: MailgunConfigurationType

    init(mailgunConfig: MailgunConfigurationType) {
        self.config = mailgunConfig
    }

    func register(_ services: inout Services) throws {
        services.register(Mailgun(apiKey: config.apiKey, domain: config.domain), as: EmailDeliveryService.self)
    }

    func didBoot(_ container: Container) throws -> EventLoopFuture<Void> {
        return .done(on: container)
    }
}

There’s not much to this. The only interesting bit is the register() implementation. It registers the Mailgun service from the Mailgun framework as the implementation for the EmailDeliveryService protocol. It uses apiKey and domain fields from the configuration passed in, which will come from the appropriate environment configuration. In my case, since I’m using Heroku, the production environment configuration will pull from the MAILGUN_API_KEY and MAILGUN_DOMAIN environment variables. Additionally, the production configuration will take care of registering this provider.

I decided to use a provider pattern for the sake of the testing configuration. The production provider here doesn’t really need to be a provider; it only registers one service. But since the testing configuration does make full use of the Provider protocol, I decided to make the production configuration follow suite.

Now that I had Mailgun in my app, I needed to put it behind a generic protocol so all the client code could be agnostic about the email service being used. I started by defining a simple protocol:


import Vapor

protocol EmailDeliveryService: Service {
    func send(_ message: EmailMessage, on container: Container) -> Future<Void>
}

An EmailDeliveryService is a Service (in the dependency injection sense) that implements a send() method. The send() method takes an EmailMessage and returns a Void promise, which can be used to know if the send succeeded or failed.

For the final bit of sending an email, I need to conform the Mailgun service to my generic EmailDeliveryService protocol:


import Vapor
import Mailgun

extension Mailgun: EmailDeliveryService {
    func send(_ message: EmailMessage, on container: Container) -> Future<Void> {
        do {
            let mailgunMessage = Mailgun.Message(
                from: message.from,
                to: message.to,
                subject: message.subject,
                text: message.body.text,
                html: message.body.html
            )

            return try self.send(mailgunMessage, on: container).transform(to: ())
        } catch let error {
            return container.future(error: error)
        }
    }
}

The implementation is intentionally as thin as possible. This is partly because it’s hard to test protocol extensions. If I had needed anything more complicated, I would have opted to wrap Mailgun in another type that conformed to EmailDeliveryService. In any case, this simply converts my EmailMessage type into Mailgun’s version and sends it. It also wraps any thrown errors into a rejected promise for the convenience of any calling code.

And, with that, my app is now sending email via Mailgun! But is it sending the correct emails? Well…

Testing

The goal of a couple of design decisions was to make the testing of emails possible, or at least easier. The choice of abstracting out the email service into a generic protocol means I can inject a testing fake. Making the email template rendering separate from the email sending means I can still test the template rendering, even if I swap out the email service with a fake.

For my integration testing, I didn’t actually want to send any emails to Mailgun. That means my tests aren’t full integration tests, and won’t catch a misconfigured Mailgun setup. But I didn’t want my integration tests dependent on an external, non-local service to run; that would make them too flaky. Plus I’d likely run into an Mailgun API quota pretty quick. However, even with this limitation, I was able to verify that the correct emails got sent at the correct time.

Like with building the initial email types, I prefer to start out with what a final test might look like. Here’s a simplified test from my app that validates a welcome email was sent:


func testCreate_emailShouldSend() throws {
    app.emailDeliveryService.send_stubbed?.succeed()
    ...
    // Do something that should send an email
    ...
    XCTAssertTrue(app.emailDeliveryService.send_wasCalled)
    XCTAssertEqual(app.emailDeliveryService.send_wasCalled_withMessage!.to, "bob.jimbob@example.com")
    XCTAssertEqual(app.emailDeliveryService.send_wasCalled_withMessage!.from, "donotreply@myapp.com")
    XCTAssertEqual(app.emailDeliveryService.send_wasCalled_withMessage!.subject, "Welcome to MyApp!")

    let link = "myapp://verify/\(user!.verifyToken)"
    XCTAssertTrue(app.emailDeliveryService.send_wasCalled_withMessage!.body.html.contains(link))
    XCTAssertTrue(app.emailDeliveryService.send_wasCalled_withMessage!.body.text.contains(link))
}

The first line tells the email service testing fake that the next call to send() should return success. Next the test calls into the app in a way that should send a welcome email (as represented by the comment). The final lines assert that send was called, and with the correct parameters. The test also validates that the most important piece of information — the verify link — appears in both the plain text and HTML parts of the email message. I didn’t do a full text comparison because most of the body is static content, and comparing all of it makes the test more fragile than it needs to be.

With this test written, I can work backwards and define what my email service testing fake should implement.


import Vapor
import Mailgun

final class FakeEmailDeliveryService: EmailDeliveryService {
    var send_wasCalled = false
    var send_wasCalled_withMessage: EmailMessage?
    var send_stubbed: Promise<Void>?
    func send(_ message: EmailMessage, on container: Container) -> Future<Void> {
        send_wasCalled = true
        send_wasCalled_withMessage = message
        return send_stubbed!.futureResult
    }
    ...
}

The testing fake, FakeEmailDeliveryService, records if send() was called along with the EmailMessage it was called with. It only keeps track of the last message because my tests only send one at a time. The fake also has the ability to return a stubbed value from send(). This is useful for validating what happens if there’s a failure on the email service’s end. The fake send() assumes that the stubbed promise has been allocated elsewhere before it’s invoked.

Speaking of allocating the stubbed promise, there’s a bit of cleanup that’s required because of the way Vapor’s promises work:


final class FakeEmailDeliveryService: EmailDeliveryService {
    ...
    deinit {
        // Vapor does not like a promise created but not resolved before it is destroyed. It calls them "leaked" and crashes the tests. So make sure nothing is "leaked" in our tests.
        send_stubbed?.succeed()
    }
}

As the comment states, Vapor will crash a test if there are any promises left unresolved. This happens in any of my tests that don’t exercise the email functionality of the app. I could go through all of those tests and add code to resolve the send_stubbed promise, but that’d be verbose and tedious. Instead, I opted to have deinit forcefully resolve the promise if it hasn’t already been.

The FakeEmailDeliveryService needs to be registered with the dependency injection system so that code asking for a EmailDeliveryService will get an instance of it. As with the production version of EmailDeliveryService, I used a Provider:


struct TestMailProvider: Provider {
    var emailDeliveryService = FakeEmailDeliveryService()

    func register(_ services: inout Services) throws {
        services.register(emailDeliveryService, as: EmailDeliveryService.self)
    }

    func didBoot(_ container: Container) throws -> EventLoopFuture<Void> {
        emailDeliveryService.send_stubbed = container.eventLoop.newPromise()
        return .done(on: container)
    }
}

The first thing the provider does is create the FakeEmailDeliveryService and stash in a public member variable. It does this so tests can get ahold of it and validate the send() parameters, or resolve its return value. The register() method then registers the fake as the implementation of EmailDeliveryService. The didBoot() method takes care of creating the unresolved promise for send()‘s stubbed return value. Creating promises require an event loop, and didBoot() is the earliest place in the test code that I had access to one. I chose to allocate the stubbed promise this early because it allowed tests to assume its existence during set up without worrying about race conditions.

With the registering of TestMailProvider all of testing fakes are set up and ready to be used. However, FakeEmailDeliveryService wasn’t yet accessible to the test cases, which were expecting it as a property on TestApplication called emailDeliveryService. The rest of this section is showing the plumbing of TestMailProvider.emailDeliveryService property up through to TestApplication.

I started at the TestingConfiguration level, which is where the TestMailProvider is created and registered:


struct TestingConfiguration: ConfigurationType {
    ...
    let mailProvider = TestMailProvider()

    func configure(_ services: inout Services) throws {
        try services.register(mailProvider)
    }
    ...
}

This exposes TestMailProvider on the TestConfiguration, which TestApplication can then use:


class TestApplication {
    ...
    private let configuration: TestingConfiguration
    ...
    var emailDeliveryService: FakeEmailDeliveryService {
        return configuration.mailProvider.emailDeliveryService
    }
    ...
}

And now my tests could validate emails by using the FakeEmailDeliveryService exposed on TestApplication.

Conclusion

My Vapor side project needed to send emails on user registration, which could potentially be a complicated setup. Since email isn’t a major feature of my app, it made sense to delegate sending emails out to a third party service like Mailgun. Although there’s a convenient Mailgun Swift package, I still needed to build out infrastructure for rendering emails from Leaf templates and abstracting out the emailing sending so I can test my app’s email handling.

Basic Vapor logging with Papertrail

For my little Vapor side project, I need to have some visibility into what’s going on when it’s deployed. There are lots of solutions for this sort of thing, but since this is side project, I don’t need anything fancy. I think some basic logging infrastructure will be more than adequate for now. There are two parts to my logging “strategy.” First, I need to decided what to log out, and second, where to log it out.

As far as what to log out, I’m going to start with the basic request and response information. That is: the method and path of the request, the response status code, and how long it took for my server to process the request. This should give me some idea of the health of my server. I’m not going to log out request headers, or the request or response bodies. They could have sensitive information in them that I don’t want to end up in logs.

The question of where to log this information out to is a slightly more interesting question. I want a convenient place to get at the logs, plus the ability to easily search those logs. And, theoretically, if I needed to scale up my server to multiple instances (lol), I would want the logs all in one place. In the end, I decided on Papertrail for a couple of reasons:

  1. I’m hosting my side project on Heroku, and they provide a convenient Papertrail add on that I can trivially provision.
  2. Papertrail offers a free tier, appropriate for side projects
  3. Papertrail doesn’t require my app to know about it directly. Papertrail installs a daemon on the monitored system that grabs the syslog and sends it to its servers. Which means if my app logs out to syslog, Papertrail will take care of the rest.

My plan for adding logging to my Vapor 3 server: first, create a logger type that will log out to syslog. Second, create a Vapor middleware that will intercept requests and responses, and log those out to the logger created in step one. Finally, update the configuration to include the logging middleware.

Logging out to syslog

Logging out to syslog turned out to be mildly interesting because I’m developing in Swift on a macOS system, but deploying to a Linux system.

The first thing I found out about syslog is Apple considers it deprecated on macOS, and would really rather I use os_log. os_log does sound fancy, but it doesn’t exist on Linux, and Papertrail doesn’t use it. So I have to ignore Apple’s advice on this one. Secondly, while syslog() the function is exposed in Swift, it doesn’t have convenient bindings, so I had to create one.

Here’s the start of my Vapor syslog logger:


import Foundation
import Vapor

final class SyslogLogger: Logger {
    func log(_ string: String, at level: LogLevel, file: String, function: String, line: UInt, column: UInt) {
        let message = "\(string) [\(function)@\(file):\(line):\(column)]"
        syslog(message, at: level)
    }
    ...
}

Logger is a protocol that Vapor defines, and the log() method here is its only requirement. I format the message given all the logging information, and then send the message with the log level to a helper function. That helper function takes care of actually calling syslog():


final class SyslogLogger: Logger {
    ...
    private func syslog(_ message: String, at level: LogLevel) {
        message.withCString { cstring in
            withVaList([cstring]) { varArgs in
                vsyslog(level.syslogLevel, "%s", varArgs)
            }
        }
    }
}

I’m calling the vsyslog() variant of syslog because that’s the one exposed to Swift. It requires a C variable argument list. Fortunately, Swift provides a way to convert a Swift array into one of those with withVaList(). Since I’ve already formatted my message using Swift’s string interpolation, I don’t actually want any further formatting from vsyslog(). However, as Sven Weidauer points out, if message contains any formatting placeholders (e.g. %d), it would cause problems because vsyslog() would try to format them. Therefore, I use %s as the format string, and pass my message in the var args. However, there’s another wrinkle here. On iOS/macOS a String is considered a CVarArg and can be passed directly into withVaList(). Unfortunately, on Linux — where this code will be deployed — String is not a CVarArg. So I have to convert the Swift String to a C string with withCString before passing it into withVaList(). Finally, I convert Vapor’s LogLevel type into a level that syslog understands, and call vsyslog().

Here’s my log level mapping:


private extension LogLevel {
    var syslogLevel: Int32 {
        switch self {
        case .verbose: return LOG_DEBUG
        case .debug: return LOG_DEBUG
        case .info: return LOG_INFO
        case .warning: return LOG_WARNING
        case .error: return LOG_ERR
        case .fatal: return LOG_CRIT
        case .custom(_): return LOG_NOTICE
        }
    }
}

I’m not entirely happy with the log level mapping, but from what I can gather from the docs, this is as close of a mapping as any. The .custom(String) log level in Vapor is a bit puzzling, but I guess the authors felt they needed an escape hatch in case they forgot a level. The other perhaps slightly surprising bit is that .verbose and .debug both map to syslog’s idea of LOG_DEBUG, since syslog doesn’t have a counterpart to .
verbose
.

I have one final bit to my logger:


extension SyslogLogger: ServiceType {
    static func makeService(for worker: Container) throws -> SyslogLogger {
        return SyslogLogger()
    }
}

This is some boilerplate code to make my SyslogLogger type able to be created by Vapor’s dependency injection system by conforming it to the ServiceType protocol. It’ll be used when I go to configure logging for the app.

Logging requests and responses

Now that I have a logger that will send information to syslog, I need to take advantage of it. As I mentioned at the start, I want to log: a request HTTP method and path, the response status code, and how long it took to service the request. Fortunately, Vapor’s middleware provides a convenient place to monitor requests and their responses.

I’ll start with declaring my middleware type and defining the middleware entry point:


import Foundation
import Vapor

final class LogMiddleware: Middleware {
    private let logger: Logger

    init(logger: Logger) {
        self.logger = logger
    }

    func respond(to request: Request, chainingTo next: Responder) throws -> EventLoopFuture<Response> {
        let startTime = Date()
        return try next.respond(to: request).map { response in
            self.log(response, for: request, whichStartedAt: startTime)
            return response
        }
    }
    ...
}

My middleware class takes a generic Logger type in its init, so it can work any logger that I choose in the future. The respond(to:chainingTo:) method is the middleware entry point. The idea is to wrap the request processing so I know how long it takes. Once the request is processed and the response generated, then I can log all that information. To accomplish that, the code notes the current time, then calls through to the rest of stack to actually handle the request. It then grabs the response that results, and passes it along with the request and start time to a helper method called log():


final class LogMiddleware: Middleware {
    ...
    private func log(_ response: Response, for request: Request, whichStartedAt startTime: Date) {
        let requestInfo = "\(request.http.method.string) \(request.http.url.path)" + (request.http.url.query.map { "?\($0)" } ?? "")
        let responseInfo = "\(response.http.status.code) \(response.http.status.reasonPhrase)"
        let time = format(Date().timeIntervalSince(startTime))
        logger.info("\(requestInfo) -> \(responseInfo) [\(time)]")
    }
    ...
}

This is the method where I construct the message to be logged. I build it piecemeal by the three main parts: the request, the response, and the time it took. For the request, I put together the method, path, and the query parameters, if there are any. For the response, I put together the HTTP status code, plus the human readable version of the status code. Vapor helpfully provides that on the reasonPhrase property. Finally, I compute the time elapsed and format that. Once I have all three pieces, I put them together into one string. The resulting message looks something like GET /my/path?p1=y&p2 -> 200 OK [98.20ms].

Formatting time is a bit verbose, so I put that into its own method:


final class LogMiddleware: Middleware {
    ...
    private let intervalFormatter: NumberFormatter = {
        let formatter = NumberFormatter()
        formatter.numberStyle = .decimal
        formatter.maximumFractionDigits = 2
        formatter.multiplier = 1000
        return formatter
    }()
    ...
    private func format(_ interval: TimeInterval) -> String {
        return intervalFormatter.string(for: interval).map { $0 + "ms" } ?? "???ms"
    }
}

I’m leveraging NumberFormatter to do all the hard work. I set it up to create decimal numbers with no more than 2 digits after the decimal point, and 1000 as a multiplier. That’s because I’m giving the formatter seconds, but I want it displayed in milliseconds. My format() method simply calls the NumberFormatter, and appends “ms” as the units. The trailing ?? "???ms" shouldn’t ever be executed, and is there to satisfy the compiler, since string(for:) can technically return nil.

The final bit for the logging middleware is to hook into the dependency injection system:


extension LogMiddleware: ServiceType {
    static func makeService(for worker: Container) throws -> LogMiddleware {
        return try LogMiddleware(logger: worker.make())
    }
}

This is also where the Logger used by the middleware will be instantiated and passed into the init method.

Configuring logging for the app

The last little bit is to register my logger and middleware with the dependency injection framework, and add the logging middleware in the correct order. That means modifying my configure() method:


/// Called before your application initializes.
public func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services, _ configuration: ConfigurationType) throws {
    ...
    services.register(LogMiddleware.self)
    services.register(SyslogLogger.self)
    ...
    /// Register middleware
    var middlewares = MiddlewareConfig()
    middlewares.use(LogMiddleware.self)
    ...
}

Here I register LogMiddleware and SyslogLogger with the dependency injection framework. Then, in the middleware configuration, I add my log middleware first, before all other middlewares. This will allow it to measure as much of the time it takes processing the request as possible. It will be the first middleware called when a request is received, and the last to be called when the response is returned.

Conclusion

While I need to have some visibility into how my Vapor server is doing in production, my side project doesn’t need anything overly complicated. I chose to use Papertrail because it provided an easy and convenient place to aggregate my logs. To get logs to Papertrail, I added middleware to my app that logs out basic request and response information to syslog.