Composable Tests

grim_io

I'm done listening to smart people proposing fixes for my dumb code. Either I'm not smart or disciplined enough to make it work, or my colleagues are not. Mostly both.

pfdietz

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state.

mrkeen

> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same. Completely backward. This definition requires isolation, rather than granting it. In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.

akoboldfrying

> Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is. Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The only thing you lose is that a test function or method may now cover several related checks (they are related by "setup dependence"), meaning their names may need to be somewhat broader. But you can still describe the specific semantics of each assert() check in a one-line comment beforehand if you want. There's no need to cram it into a legal method name. ETA: Prefer to write tests whose "arrange" steps are as simple as possible, to minimise unnecessary overlaps. But if the simplest possible "arrange" step for a test is something that itself needs to be checked for correctness, just do that check right there, and nowhere else . Anything beyond that is ceremony that adds nothing useful.

solarengineer

Someone had written a comment about whether Kent Beck has heard of global and static variables. That comment seems to have been deleted. Kent Beck is co-creator [1] of the JUnit Testing Framework and has most definitely heard of static and global variables. [1] https://junit.org/junit4/project-info.html

Groxx

Ehhh... when it truly is two tests bodged into one, then sure. But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field? If it's the former, then it might be two tests masquerading as one, and this might make sense. If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.

nhgiang

Ah Kent Beck, I see your style of writing trivial examples hasn't changed

crabbone

1. I'm not sure what CodeRabbit has to do with the article... It's obviously an advertisement, but it's merged into the text of the article, which I find bizarre. As an aside, I used CodeRabbit at work, and I have mixed feelings about it. I'm not entirely against using it or a similar tool, but people often treat the comments from CodeRabbit as a gospel, and they can harm their code as a result. Also, I've never seen CodeRabbit being anything more than a superficial reviewer: fixing typos, other unintended errors, but it never comments on the substance of the change, which implicitly validates it for the author. 2. On test composition. Unit tests are called "unit" because they are supposed to test one thing. If a test is testing more than one thing, it's an integration test. Ideally, people writing unit tests are the developers themselves and people writing integration tests are test (automation) people. This matters for administrative reasons: in the development cycle, the unit test is the basic check that validates a particular piece of code, probably, submitted for review or for merging. Passing such a test might be a necessary condition to progress the changeset along the designed workflow path. Integration tests, on the other hand, are more of a retrospective tool that is meant for detection of problems in the entire product. A failure of an integration test should, normally, schedule new task for the developers, not reject the one being worked on. Integration tests, typically, will require a more elaborate system under test setup and a more elaborate, perhaps involving multiple teams, investigation of the failure. They can be also a lot more expensive to run in terms of equipment used. Finally, the author touched on a contentious subject a.k.a. the number of assertions in a test. A lot of people believe (me included) that the number of assertions in the unit test should be exactly one. This is often inconvenient because it requires implementation of equality for possibly ad hoc created set of results. Even so, I believe it's still worth it. When it comes to integration tests, I don't believe assertions are at all the way to go. The system under test should be monitored continuously and every reading should be compared against the desired state of the system that the test modifies simultaneously with the change effected to the system. This is because, in practice, it's rarely just two features that are tested together. If the test waits until the final step to compare the desired and the actual state of the system, the error as well as the context in which it happened might be long gone. As a side bonus: the monitoring+alerts system could well be part of the product itself, or, if not, it can be used in long-running tests intended to collect mileage (i.e. tests intended to prove that the system performance doesn't degrade over substantially long periods of time).

johnwils

Trimming the copy-paste chain is right. The pieces being green isn’t the same as the app opening.

npodbielski

What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.

cadamsdotcom

Why wouldn't you use parameterized tests or compose your setup out of fixtures that run the shared setup code? Running a test, then running it a second time, then asserting something else.. is just weird.