5 microservice testing strategies for startups

Testing microservices is hard. I learned just how hard it could be when I first dived into a tech stack with seven separate microservices, each with its own code base, dependency management, feature branches, and database schema—which also happened to have a unique set of migrations.

Talk about hectic.

The approach I took was to run everything locally. That meant that whenever I wanted to run end-to-end tests, I needed to go through the following five steps for each of the seven microservices:

1. Ensure I was on the correct code branch (either master or feature_xyz)

2. Pull down the latest code for that branch

3. Ensure all dependencies were up to date

4. Run any new database migrations

5. Start the service

This was just a baseline requirement to enable tests to be run. Often I would forget to perform one of these steps for a service and spend 10-15 minutes debugging the issue. Once I finally had all the services happy and running, I could then begin to kick off the test suite. This experience sure made me miss the days of testing one big monolith.

So yes, I discovered that end-to-end microservice testing is hard—and it gets exponentially harder with each new service you introduce. But fear not, because there are ways to make testing microservices easier. I’ve spoken to several CTOs about how they came up with their own creative ways of tackling this complex issue.

Testing microservices versus testing monoliths

Before we go over common methods for testing microservices and strategies to deal with the challenges those methods present, let’s drill down on why testing microservices is a different beast than testing a monolith.

Right off the bat, microservices require extra steps like managing multiple repositories and branches, each with their own database schemas. But the challenges can run deeper than that.

Here are a few key challenges associated with testing microservices:

• Availability: Since different teams may be managing their own microservices, securing the availability of a microservice (or worse, trying to find a time when all microservices are available) is tough.

• Fragmented and holistic testing: Microservices are built to work alone and together with other loosely coupled services. That means developers need to test every component in isolation as well as testing everything together.

• Knowledge gaps: Particularly with integration testing (which we will address later in this article), the person who conducts the tests will need to have a strong understanding of each service in order to write test cases effectively.

Despite these challenges, the advantages of microservices are too great to forego because of testing challenges. A spectrum of methods can be used to deal with the challenges of testing microservices.

Common microservice testing methods

Unit testing

A microservice may be smaller by definition, but with unit testing, you can go even more granular. A unit test focuses on the smallest part of the testable software to ascertain whether that component works as it should. Renowned software engineer, author, and international speaker Martin Fowler breaks unit testing into two categories:

• Sociable unit testing: This unit testing method tests the behavior of modules by observing changes in their state.

• Solitary unit testing: This method focuses on the interactions and collaborations between an object and its dependencies, which are replaced by test doubles.

While these unit testing strategies are distinct, Fowler puts forth that they aren’t competing—they can be used in tandem to solve different testing problems.

David Strauss, CTO of Pantheon, told me, “The opportunity is that microservices are very straightforward to actually do unit testing on.”

Isaac Mosquera, CTO of Armory concurred, adding that Armory “instructs customers to focus on unit testing.”

Integration testing

With integration tests, you’re doing just what it sounds like you’re doing: Testing the communication paths and interactions between components to detect issues. According to Fowler, an integration test “exercises communication paths through the subsystem to check for any incorrect assumptions each module has about how to interact with its peers.”

An integration test usually tests the interactions between the microservice and external services, such as another microservice or datastore.

Pawel Ledwoń, platform lead, Pusher, said that his team “lean[s] towards integration testing. Unit tests are still useful for some abstractions, but for user-facing features, they are difficult to mock or skip important parts of the system.”

Sumo Logic’s chief architect Stefan Zier also revealed that they “invest fairly heavily into integration testing.”

However, not everybody I spoke to was a fan of the process. Mosquera’s take on the subject of integration testing, for example, is well worth noting: “Integration testing is very error-prone and costly, in terms of man-hours. The ROI just isn’t there. Each individual integration test brings small marginal coverage of use cases.

“If you consider all the code path combinations to your application, coupled with another application’s code paths, the number of tests that need to be written easily explode to a number that is unachievable. Instead, we instruct our customers to focus on unit test coverage, a handful of integration tests that will demonstrate total failure to key areas of the application,” Mosquera added.

End-to-end testing

Last but not least is end-to-end testing, which—as previously mentioned—can be a difficult task. That’s because it involves testing every moving part of the microservice to ensure that it can achieve the goals you built it for.

Fowler wrote, “End-to-end tests may also have to account for asynchrony in the system, whether in the GUI or due to asynchronous backend processes between the services.” He goes on to explain how these factors can result in “flakiness, excessive test run time, and additional cost of maintenance of the test suite.”

The best advice I can give when it comes to end-to-end testing is to limit the number of times you attempt it per service. A healthy balance between the other microservice testing strategies mentioned—like unit testing and integration testing—will help you weed out smaller issues. An end-to-end test is larger by definition, takes more time, and can be far easier to get wrong. To keep your costs low and avoid time-sink, stick to end-to-end testing when all other means of testing have been exhausted, and as a final stamp of quality assurance.

Five microservice testing strategies for startups

Testing microservices is difficult, but it’s not impossible. To tackle these challenges, I got insight from several CTOs and distilled five strategies they used to successfully approach testing microservices.

1. Documentation-first strategy

Chris McFadden, VP of engineering at SparkPost, summarized the documentation-first strategy quite nicely during our discussion:

“We follow a documentation first approach so all of our documentation is in markdown in GitHub. Our API documents are open source, so it's all public. Then what we'll do is, before anybody writes any API changes or either a new API or changes to an API, they will update the documentation first, have that change reviewed to make sure that it conforms with our API conventions and standards (which are all documented), and make sure that there's no breaking change introduced here. Make sure it conforms with our naming conventions and so forth as well.”

If you’re willing to go one step further, you could dabble in API contract testing, which—as previously mentioned—involves writing and running tests that ensure the explicit and implicit contract of a microservice works as it should.

2. Full stack-in-a-box strategy

The full stack-in-a-box strategy entails replicating a cloud environment locally and testing everything all in one vagrant instance ($ vagrant up). The problem? It’s extremely tricky, as software engineer Cindy Sridharan of imgix explained in a blog post:

“I’ve had first-hand experience with this fallacy at a previous company I worked at where we tried to spin up our entire stack in a [local] vagrant box. The idea, as you might imagine, was that a simple vagrant up should enable any engineer in the company (even front-end and mobile developers) to be able to spin up the stack in its entirety on their laptops.”

Sridharan goes on to detail how the company only had two microservices: a gevent-based API server and some asynchronous Python background workers. A relatively simple setup, by all means.

“I remember spending my entire first week at this company trying to successfully spin up the VM locally, only to run into a plethora of errors. Finally, at around 4 pm on the Friday of my first week, I’d successfully managed to get the Vagrant setup working and had all the tests passing locally. I remember feeling incredibly exhausted.” she said.

Despite her best efforts to document the obstacles she ran into—and how she got over them—Sridharan revealed that the next engineer her company hired ran into their own share of issues that couldn’t be replicated on another laptop. (I did say it was tricky!)

Stefan Zier, chief architect at Sumo Logic, explained to me that—on top of being difficult to pull off—this localized testing strategy simply doesn’t scale: “[With] a local deployment, we run most of the services there so you get a fully running system, and that's now stretching even the 16GB RAM machines quite hard. So that doesn't really scale.”

3. AWS testing strategy

This third strategy involves spinning up an Amazon Web Services (AWS) infrastructure for each engineer to deploy and run tests on. This is a more scalable approach to the full stack-in-a-box strategy discussed above.

Zier called this a “personal deployment [strategy], where everyone here has their own AWS account.” He added, “You can push the code that's on your laptop up into AWS in about ten minutes and just run it in like a real system.”

4. Shared testing instances strategy

I like to think of this fourth strategy as a hybrid between full stack-in-a-box and AWS testing. That’s because it involves developers working from their own local station while leveraging a separate shared instance of a microservice to point their local environment at during testing.

Steven Czerwinski, head of engineering, Scaylr, explained how this can work in practice:

“Some of [our] developers run a separate instance of a microservice just to be used for testing local builds. So imagine you're a developer, you're developing on your local workstation, and you don’t have an easy way of launching the image parser. However, your local builder would just point to a test image parser that’s running in the Google infrastructure.”

Czerwinski continued, “Similarly, we've been talking about doing that for the front-end developers. Hiding the database layer between a service so that people don't have to run their own kind of monolithic server, the database server, when they're testing UI features. They want to be able to rely on an external one that can be updated pretty frequently and doesn't have production or staging data or anything like that.”

5. Stubbed services strategy

Finally, we have the stubbed services testing strategy.

Zier laid out Sumo Logic’s approach to stubbed service testing: “Stubbing lets you write these marks, or ‘stubs,’ of microservices that behave as if they were the right service and they advertise themselves in our service discovery as if they were real service, but they’re just a dummy imitation.”

For example, testing a service may require the service to become aware that a user carries out a set of tasks. With stubbed services, you can pretend that user (and their tasks) have taken place without the usual complexities that come with that. Obviously, this approach is a lot more lightweight than running services in their totality.

Tools to help you test microservices

In addition to these strategies, the CTOs suggested the following tools they’ve used to make microservices testing a bit easier:

• Hoverfly: simulate API latency and failures

• Vagrant: build and maintain portable virtual software development environments

• VCR: a unit testing tool

• Pact: frameworks consumer-driven contracts testing

• API Blueprint: design and prototype APIs

• Swagger: design and prototype APIs

Microservice testing: Difficult, but doable

Testing your microservices won’t be a walk in the park, but the additional work is worth the benefits of having a microservice architecture. To review, the five strategies are:

1. The documentation-first strategy

2. The full stack-in-a-box strategy

3. The AWS testing strategy

4. The shared testing instances strategy

5. The stubbed service strategy

Naturally, you may need to tweak each strategy to match your unique circumstances, but with some good old-fashioned trial and error, your microservice testing strategy should come into its own.

[See our related story, 5 guiding principles you should know before you design a microservice]