Unit testing emails

By Confusion on maandag 23 juni 2008 21:25 - Comments (9)
Categories: Software engineering, Testing, Views: 5.275

Sometimes the main purpose of an application is sending emails. But how do you unit test such a beast? Manually verifying the arrival and contents of the emails is fine for small amounts of email, but when the number of code paths and email varieties grows, you need an automated way to verify that the correct emails have been sent.

I have not yet done any research on existing libraries/frameworks that solve this problem, but I think it has to be either of two solutions:
  1. Have the test code poll a dedicated email box, matching emails with testcases and determining whether they are correct
  2. Launch a mock email server from your code, modify the test configuration in such a way that the mails are sent to that mail server and immediately check back whether the mails are correct.
The major drawback of the first solution is that you have no control over the moment when, and the order in which, the emails arrive. The largest problem of the second solution is writing/finding that server, that programmatically allows you to fetch the last received mail, etc.

If you already have an project layout and architecture that allows dependency injection and a seperate configuration for tests, then the second seems the obvious way to go: much more flexible and contained. Unfortunately, the project for which I would like to implement this does not have anything of the sort and adding that will not be easy, which make me prefer the first solution.

There is another thing to consider: library support. As I said, I have not yet searched, but my guess is that the second solution is the one for which there will be several solutions readily available, because it is the 'proper' solution. Moreover, I doubt whether I will find any existing solution that makes implementing the first solution easier.

I think I'll just go with the 'proper' solution :).

Volgende: The Italian 'democracy' 07-'08 The Italian 'democracy'
Volgende: JSON to XML conversion 06-'08 JSON to XML conversion


By Tweakers user H!GHGuY, maandag 23 juni 2008 22:20

tbh, that's no longer unit-testing...

you're testing the application end-to-end.
Proper unit-tests would be:
- test whether the mailing subsystem code can send large amounts of e-mails correctly
e.g. send 1million e-mails and verify that 1million e-mails arrive correctly.
The easy way to perform an end-to-end mail test is to embed a unique increasing sequence number in each mail.
- test whether all code-paths produce the correct mails
this can even occur offline

both of these tests are far easier to perform statically than what you were describing.

By Tweakers user Harm, maandag 23 juni 2008 23:57

If I remember correctly, the Mozilla project (the Thunderbird one of course) has been asking this same question just a few weeks ago, and - again if I remember things correctly - has chosen for the second option. They are developing or already have developed a 'fake' mail server for this (both for sending and for receiving). Perhaps you can use theirs?

By Tweakers user RoadRunner84, dinsdag 24 juni 2008 00:00

Why not letting the email system display a popup or write to a log file during debug or development stage, test the email system itself and pass the data through the mail system in a production stage?
Any mature language allows you to turn on debug flags that may let your functions behave different in that stage.

By Tweakers user ikke007, dinsdag 24 juni 2008 08:51

I have decoupled my mail sending process from my mail generation proces. This means, the generation of e-mails can occur today and the sending any other time. The emails are saved to a database after generation. When this gets too full i will start garbage collection but for now i am fine ;-)

Anyway, you could try this approach for the testing if
* the correct amount of e-mails are generated (test 1)
* all the code path are followed, all distinct e-mails are generated correctly (test 2)
* send alle e-mails to a dedicated mailbox and read the number of messages (manually)

added + is that the mail proces is decoupled, meaning that people do not have to wait on 'your confirmation code email has been send' like web pages (PHP)

By Tweakers user Alex, dinsdag 24 juni 2008 08:57

In this case you simply haven't got a clue where a unit test it for.
This should be an integration test. Read purpose:

Also because your unit test will emulate the real conditions the way the program is running.

By Tweakers user Confusion, dinsdag 24 juni 2008 11:07

@H!GHGuY and Alex:
True, I tend to misuse the term 'unit test', but you have to agree that there isn't a fine line between unit tests and integration tests. This test does not test the complete application, as that also involves parsing a file, adding required information from the database to the information parsed from a file entry and logic deciding which mail to send. In that respect, it isn't what I would generally call an integration test, because it does not integrate functionally seperate parts.

This test will test "given a certain method call with appropriate parameters, find the required email template, construct the email and send it". This is because that is the smallest 'unit' that can feasibly be tested without refactoring the application. Of course, it is an integration test of several classes, but in my book, a 'unit test' is not necessarily a test of a single class (we are talking Java here).

I must add: in situations like these, I would not write a seperate test checking that the correct template was fetched anyway, because that is already tested by test described above and is a type of failure than can be clearly distinguished. Writing a seperate test for a seperate 'unit' only makes sense when a higher-level test would show errors with multiple possible causes.

Motto: when in doubt, write an extra test, but be sane about the amount of tests you write.

[Comment edited on dinsdag 24 juni 2008 11:20]

By Jswdaniels, dinsdag 24 juni 2008 13:47

Hi Confusion,

Typically, component[unit] testing occurs with access to the code being tested and with the support of the development environment, such as a unit test framework or debugging tool, and, in practice, usually involves the programmer who wrote the code. Defects are typically fixed as soon as they are found, without formally recording incidents \[reference:
ISTQB foundation level sylabus].

Integration testing tests interfaces between components, interactions with different parts of a system, such as the operating system, file system, hardware, or interfaces between systems. There may be more than one level of integration testing and it may be carried out on test objects of varying size. For example:
1. Component integration testing tests the interactions between software components and is doneafter component testing;
2. System integration testing tests the interactions between different systems and may be done after system testing. In this case, the developing organization may control only one side of the interface, so changes may be destabilizing. Business processes implemented as workflows may involve a series of systems. Cross-platform issues may be significant.
\[reference: ISTQB foundation level sylabus].

In my opinion you're still unit testing. So are ther codecompilers or parsers that can help? as for the options, I would go for the second. the fake mail server can than also be used in other tests like integration, system, systemintegration testing. Although not an easy task. It's still my choise to add a separate environment for testing

By Tweakers user Confusion, dinsdag 24 juni 2008 20:34

@Harm: thanks, I'll take a look at the solution the Thunderbird developers decided to use.

@RoadRunner84: popups or other solutions requiring human interaction are not an option due to the number of different emails that need to be verified. Writing to a logfile requires the same kind of dependency injection as a mock mailserver and would probably require more work on my part to retrieve the emails. A mock mailserver already recognizes where a mail start and ends; I would have to parse a log file, introducing extra possibilities for bugs.

@ikke007: if the mail generation and sending were sufficiently decoupled, testing what was generated without actually sending the mails would be preferable. As it stands however, that may involve more work than finding a ready-made mock mail server :).

By levi_h, dinsdag 8 juli 2008 17:16

You probably already solved this, but maybe future Googlers can benefit: GreenMail (http://www.icegreen.com/greenmail/) does the job quite nicely.

Comments are closed