This was originally submitted as the pytest thread, but Andres validly
pointed out that it should probably not be part of it to reduce the size
of that patchset.
I think it's still useful though, so I'm submitting separately
(eventhough Jacob wrote it).