Unit testing network interactions
Developer James Coglan explains what lessons the tech team learnt from the problems that came with unit-testing Ruby code.
Developer James Coglan explains what lessons the tech team learnt from the problems that came with unit-testing Ruby code.
Sometimes when writing Ruby code, especially when using Rails and other DSL-heavy libraries, you’ll find yourself writing some code and not being sure how to unit-test it. This might be because it hides the objects that collaborate with yours, so you can’t easily mock or stub them, or it creates a lot of singletons, or you’re just not sure which layer you should be mocking or stubbing at. We recently experienced all these problems and found it a useful lesson in how to design tests.
We were just starting out on implementing search functionality for our courses, and we’d decided to go with Elasticsearch, using the elasticsearch-persistence gem to communicate with it. After a little experimentation, we ended up with this code for indexing all our courses by title and searching that index:
require 'elasticsearch/persistence'
require 'course'
class CourseSearch
include Elasticsearch::Persistence::Repository
client Elasticsearch::Client.new(url: ENV['ELASTICSEARCH_URL'])
index :courses
type :course
klass Course
settings do
mappings do
indexes :title, analyzer: :english
end
end
def self.index_courses(courses)
create_index!(force: true)
courses.each { |course| save(course) }
end
def self.search_courses(text)
search(query: { match: { title: text } }).results
end
end
Course
is a simple value object that provides a class for Elasticsearch::Persistence
to instantiate for each result it returns. We’re storing the title
, because that’s what we want to search on, and the slug
as well; we want to render these results as links to the relevant course page, and so we need their title and slug to generate those links.
class Course
def initialize(attributes)
@attributes = attributes
end
def slug
@attributes['slug']
end
def title
@attributes['title']
end
def to_hash
@attributes
end
end
This CourseSearch
class, although minimal, does work. We can use it to index some courses and search for them using a text query. We express that as a spec:
require 'course_search'
describe CourseSearch do
before do
courses = [
{
'slug' => 'dutch',
'title' => 'Introduction to Dutch',
},
{
'slug' => 'liveable-cities',
'title' => 'Water for Liveable and Resilient Cities'
},
{
'slug' => 'orion',
'title' => 'In the Night Sky: Orion'
}
]
CourseSearch.index_courses(courses)
end
it 'returns courses matching the search terms' do
expect(CourseSearch.search_courses('city').map(&:to_hash)).to eq([
{
'slug' => 'liveable-cities',
'title' => 'Water for Liveable and Resilient Cities'
}
])
end
end
While this test works, it’s not ideal as a unit test. Since it integrates with Elasticsearch, it requires an Elasticsearch server running, it’s much slower than a unit test would be, and the way it checks the result means it’s mostly testing Elasticsearch rather than our own code.
The big risk with this test is that it omits a lot of important detail. Elasticsearch supports a wide range of different options for indexing and searching your documents, and search services in general involve all the complexity of analysing human language. While checking that the query city
returns a course with the word Cities
in its title, testing that word stemming works, this testing strategy can’t begin to get into all the nuance of search.
And nor should it: all that complexity is Elasticsearch’s job. What we ought to be testing is that our code uses Elasticsearch in the ways we care about: we have made particular choices about how to index and search our content in order to give the best user experience, and we want to enshrine those choices in tests rather than testing how Elasticsearch interprets those choices.
Finding the seams
One of the biggest challenges we faced with this code was deciding what we even want to test. We want a “unit” test, but what does that mean? Where is the boundary between the CourseSearch
class and its collaborators?
The CourseSearch
class follows the pattern that we see in a lot of Ruby’s frameworks. Most of it is really configuration for some other client: what the name of the index is, how to index each column. This class barely does any work itself, all the real action happens somewhere else. So what should we test?
We could test the interaction between CourseSearch
and the Elasticsearch::Persistence::Repository
mixin, the boundary between our code and a third-party library. In this scenario, a lot of code cannot be tested; the client
, index
, type
, klass
and settings
are all at the class level rather than the instance level. These calls happen at “design-time” when the class is loaded, rather than at “run-time” when the class is used. Maybe we can inspect the class after our test has loaded it to examine our config, but we’d just be reproducing the configuration in our tests.
What about the code that does run at runtime: the methods. We could test those using mocking, to check that each method makes the correct calls to the Elasticsearch
library.
require 'course_search'
require 'course'
describe CourseSearch do
describe '.index_courses' do
before do
allow(CourseSearch).to receive(:create_index!)
allow(CourseSearch).to receive(:save)
end
it 'creates a new index' do
expect(CourseSearch).to receive(:create_index!).with(force: true)
CourseSearch.index_courses([])
end
it 'saves each course' do
course = double(:course)
expect(CourseSearch).to receive(:save).with(course)
CourseSearch.index_courses([course])
end
end
describe '.search_courses' do
let(:response) { double(:response, results: double(:results)) }
it 'performs a match query on the title' do
query = { query: { match: { title: 'the query' } } }
expect(CourseSearch).to receive(:search).with(query).and_return(response)
CourseSearch.search_courses('the query')
end
it 'returns the results' do
allow(CourseSearch).to receive(:search).and_return(response)
expect(CourseSearch.search_courses('the query')).to eq(response.results)
end
end
end
These tests are reasonably simple, but they don’t say anything interesting; they really just reproduce the implementation. And they don’t actually check very much; yes, they check which kind of query mechanism we’re using, but they don’t check how we’ve indexed the data (the mappings
in the config), and indexing and searching need to work hand-in-hand.
These tests also don’t do much to reduce the risk of errors: we’ve only just adopted this library, and don’t have a solid understanding of how it works. When mocking, we should test at boundaries that are well-defined, stable, and that we understand, so that we can say with confidence what the methods we’ve set expectations on will do in production. If we don’t know what they’ll do, then we don’t know that we’re testing the right thing.
The above spec tests the interaction between CourseSearch
and its closest collaborator, the Elasticsearch::Persistence::Repository
mixin. Maybe there’s a boundary another layer away that we should test at. The only other collaborator visible in this code is the Elasticsearch::Client
object. We could write tests that check what commands that receives when our code runs.
But again, there are problems with that approach. First, on a practical level, the Elasticsearch::Client
object is another thing created as a singleton at design-time. It’s not something that’s passed into, or created inside, our methods, and that makes it harder to write mock expectations against it. Second, and more importantly, such a test again probably wouldn’t tell us anything useful and would test the wrong thing: it would test how Elasticsearch::Persistence::Repository
interacts with Elasticsearch::Client
, and those are interactions that happen inside third-party code, not in our own application. We’d just be duplicating the gem’s implementation and checking its internal workings, which might change at any time.
So, if we don’t want to test against our immediate collaborators, and we don’t want to test the internals of those collaborators, and we don’t want an integration test, what’s left? There’s one more boundary we’ve not considered: the calls that our application makes over the wire to the Elasticsearch server. Elasticsearch uses HTTP as its protocol, so we can use a tool like WebMock to check the requests that our app makes.
This has several advantages over our other ideas: it tests a meaningful, stable boundary, since the Elasticsearch protocol is stable and documented, compared to the internals of whatever gems we’re using. And it reduces risk: the thing we’re uncertain about is whether we’re using the Elasticsearch client libraries in the right way to get the results we want, and the things we want to check are how we index and search the data to achieve those results. Observing the requests sent to Elasticsearch will let us check everything our code is really doing.
Recording requests
Having decided on a boundary to test at, we need to find out what should be happening at that boundary. Remember that at this stage we’ve only just adopted this technology and we’re not sure what’s going on internally so we don’t know what to test for. We want to make sure our app makes the right kinds of requests to Elasticsearch, but we don’t know where in our gems’ code those requests happen, and the CourseSearch
class certainly doesn’t interact with anything that directly performs those requests.
We wanted to peek at what was being sent over the wire, and there are various tools for doing this. After trying a few of them out we found tcpflow, which lets you monitor what’s going on on any network port. Elasticsearch runs on port 9200, so we can use this command to begin watching it:
sudo tcpflow -c -J -i any port 9200
This prints all incoming and outgoing data on port 9200 to the console, colouring the output blue for data from the client and red for data from the server. With that command running, we can execute a script to create the index and index a course:
courses = [
{
'slug' => 'liveable-cities',
'title' => 'Water for Liveable and Resilient Cities'
}
]
CourseSearch.index_courses(courses)
When we run this script, the following is printed by tcpflow:
127.000.000.001.36738-127.000.000.001.09200: DELETE /courses HTTP/1.1
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: localhost:9200
127.000.000.001.09200-127.000.000.001.36738: HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 21
{"acknowledged":true}
127.000.000.001.36739-127.000.000.001.09200: HEAD /courses HTTP/1.1
User-Agent: Faraday v0.9.2
Accept: */*
Connection: close
Host: localhost:9200
127.000.000.001.09200-127.000.000.001.36739: HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=UTF-8
Content-Length: 0
127.000.000.001.36740-127.000.000.001.09200: PUT /courses HTTP/1.1
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: localhost:9200
Content-Length: 101
Content-Type: application/x-www-form-urlencoded
127.000.000.001.36740-127.000.000.001.09200:
{"settings":{},"mappings":{"course":{"properties":{"title":{"analyzer":"english","type":"string"}}}}}
127.000.000.001.09200-127.000.000.001.36740: HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 21
{"acknowledged":true}
127.000.000.001.36741-127.000.000.001.09200: POST /courses/course HTTP/1.1
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: localhost:9200
Content-Length: 76
Content-Type: application/x-www-form-urlencoded
127.000.000.001.36741-127.000.000.001.09200:
{"slug":"liveable-cities","title":"Water for Liveable and Resilient Cities"}
127.000.000.001.09200-127.000.000.001.36741: HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8
Content-Length: 142
{"_index":"courses","_type":"course","_id":"AVGGintuILmNrviy3YK9","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
Here we see a sequence of four requests:
DELETE /courses
, with a200
response containing{"acknowledged":true}
HEAD /courses
, with a404
responsePUT /courses
with data{"settings": {}, "mappings": {"course":
, with
{"properties": {"title": {"analyzer": "english", "type": "string"}}}}}
a200
response containing{"acknowledged":true}
POST /courses/course
with data{"slug": "liveable-cities", "title": "Water
, with a
for Liveable and Resilient Cities"}201
response containing
{"_index": "courses", "_type": "course", "_id": "AVGGintuILmNrviy3YK9",
"_version": 1, "_shards":{"total": 2, "successful": 1, "failed": 0},
"created": true}
This captures everything we want to assert about our search implementation: the name of the index, the mapping config, the fields we submit for each document. And they’re expressed in terms of the public, documented Elasticsearch API rather than some internal private library functions.
Writing the tests
Let’s begin writing our spec. We can start out by just stubbing out all the requests that index_courses
will make, providing just enough information that our code runs without crashing. We don’t want to include extraneous request/response data in the test that doesn’t affect the result, since this only makes it harder to tell which bits of a test are actually important. In this case, all we need to supply are the status codes.
describe CourseSearch do
let(:elasticsearch_origin) { 'http://localhost:9200' }
describe '.index_courses' do
let(:url) { elasticsearch_origin + '/courses' }
before do
stub_request(:delete, url).to_return(status: 200)
stub_request(:head, url).to_return(status: 404)
stub_request(:put, url).to_return(status: 200)
stub_request(:post, url + '/course').to_return(status: 201)
end
Then we can write some tests to check each individual call the our CourseSearch
class makes. First, we know it should delete the existing index, and we can check this by asking WebMock whether DELETE /courses
was requested.
it 'deletes the existing index' do
CourseSearch.index_courses([])
expect(WebMock).to have_requested(:delete, url)
end
That was relatively easy, and we’re already testing something far more detailed than our original integration test did.
Next, we want to check that it creates a new index, configuring the indexed fields correctly.
it 'creates a new index, analyzing the title as English' do
CourseSearch.index_courses([])
expect(WebMock).to have_requested(:put, url).with { |request|
JSON.parse(request.body).fetch('mappings') == {
'course' => {
'properties' => {
'title' => { 'analyzer' => 'english', 'type' => 'string' }
}
}
}
}
end
Rather than checking the body of the request as a string, we’re parsing it into a data structure and comparing against that. This is important for two reasons: it makes sure the body is valid JSON, and it stops the tests being sensitive to serialisation order. For example, {"analyzer": "english", "type": "string"}
and {"type": "string", "analyzer": "english"}
are equivalent structures in most languages, but they are different strings. Comparing data structures means we test only what we care about – the meaning of the string – and not the exact serialisation details.
Also notice that in the above tests, we don’t pass in any courses to CourseSearch.index_courses
. That’s because these aspects of the method don’t need any course data, and so leaving such data out makes it clearer that it doesn’t affect the test outcome.
Our final test of this functionality does need course data though: we need to check that Elasticsearch receives one POST /courses/course
request for each course we want to index.
it 'adds each given course do the index' do
courses = [
Course.new('slug' => 'dutch', 'title' => 'Introduction to Dutch'),
Course.new('slug' => 'liveable-cities', 'title' => 'Water for Liveable and Resilient Cities'),
Course.new('slug' => 'orion', 'title' => 'In the Night Sky: Orion')
]
CourseSearch.index_courses(courses)
courses.each do |course|
expect(WebMock).to have_requested(:post, url + '/course').with { |request|
JSON.parse(request.body) == course.to_hash
}
end
end
Here we index three courses, and check that each of them caused a POST
request with their hash contents as the body. We could make it more explicit which fields we’re indexing as well by writing the test as:
expect(WebMock).to have_requested(:post, url + '/course').with { |request|
JSON.parse(request.body) == {
'slug' => course.slug,
'title' => course.title
}
}
Now we have some tests that verify how we’re indexing the data, and which fields our search documents have, we can move on to how we send queries. Again, we need to observe what goes over the wire when we send a query. We run the following script with tcpflow running:
CourseSearch.search_courses('city')
And this is what we see:
127.000.000.001.37057-127.000.000.001.09200: GET /courses/course/_search HTTP/1.1
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: localhost:9200
Content-Length: 36
Content-Type: application/x-www-form-urlencoded
127.000.000.001.37057-127.000.000.001.09200:
{"query":{"match":{"title":"city"}}}
127.000.000.001.09200-127.000.000.001.37057: HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 302
{"took":37,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.15342641,"hits":[{"_index":"courses","_type":"course","_id":"AVGGintuILmNrviy3YK9","_score":0.15342641,"_source":{"slug":"liveable-cities","title":"Water for Liveable and Resilient Cities"}}]}}
We send a GET /courses/course/_search
request with body {"query:{"match":{"title":"city"}}}
, just as it says in our code (the Elasticsearch
gem exposes some parts of the wire protocol directly). The response contains this data, reformatted for readability:
{
"took": 37,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.15342641,
"hits": [
{
"_index": "courses",
"_type": "course",
"_id": "AVGGintuILmNrviy3YK9",
"_score": 0.15342641,
"_source": {
"slug": "liveable-cities",
"title": "Water for Liveable and Resilient Cities"
}
}
]
}
}
We want to test two things about our search_courses
method: that it makes the right type of query to Elasticsearch, and that it returns the results of the search. Beforehand, we need to stub out this endpoint with enough detail to make the client work. There’s a lot of fields in the above response that we can omit safely, so we’ll only include fields that are strictly necessary to avoid crashes, or that contain data we’re checking in our test. In this case, we just need to include the hits.total
and hits.hits
field with the _source
of each result, which is used to product the result objects.
describe '.search_courses' do
let(:url) { elasticsearch_origin + '/courses/course/_search' }
before do
stub_request(:get, url).to_return(
status: 200,
headers: { 'Content-Type' => 'application/json' },
body: JSON.dump(
'hits' => {
'total' => 2,
'hits' => [
{ '_source' => { 'slug' => 'much-ado-about-nothing', 'title' => 'Much Ado About Nothing: in Performance' } },
{ '_source' => { 'slug' => 'shakespeares-hamlet', 'title' => 'Shakespeare\'s Hamlet: Text, Performance and Culture' } }
]
}
)
)
end
Checking that we send the correct query to match on the title field is very similar to checking the index requests as we saw above: we just check that WebMock saw a GET
request with the right JSON payload:
it 'asks Elasticsearch to perform a match query on the title' do
CourseSearch.search_courses('performance')
expect(WebMock).to have_requested(:get, url).with { |request|
JSON.parse(request.body) == { 'query' => { 'match' => { 'title' => 'performance' } } }
}
end
To check the results, we don’t so much care about their exact type as that they have the right API; they should respond to slug
and title
by returning the appropriate value. We can map these method calls over the results to check we get what we expect:
it 'returns a Course for each search result' do
courses = CourseSearch.search_courses('performance')
expect(courses.map { |c| [c.slug, c.title] }).to eq [
['much-ado-about-nothing', 'Much Ado About Nothing: in Performance'],
['shakespeares-hamlet', 'Shakespeare\'s Hamlet: Text, Performance and Culture']
]
end
Wrapping up
What we’ve seen here is that writing a ‘unit test’ isn’t necessarily about looking at one class with respect to its immediate collaborators. Instead, it can be about one ‘conceptual unit’, which might include third party library code. That doesn’t make it ‘not a unit test’, it just means we have a fuzzy idea of what a ‘unit’ is.
The important thing is the boundaries between components, not the components themselves. Especially when your code is as simple as this, most of your testing effort will go into checking that the code has the right effects on whichever system it is connected to. It’s important to find a meaningful boundary, one that represents a genuine seam in your architecture where important information passes through, rather than one that happens to exist by accident because of how your internal code and libraries are arranged.