OLD Components Design Sketch

Contributed by schauder

From the requirements I try to come up with a rough class/component structure that might enable the requested features.

TestSourceFinder

A TestSourceFinder finds TestSources. When JUnit gets invoked it can either use it's default implementations or get additional implementations provided. This would allow for completely different creation of tests, like tests based on story files, webpages and what not.

DefaultTestSource

JUnit might/should provide default implementation (DefaultTestSource) that finds all TestSources in one or more directories, the current classpath or a filtered part of the classpath possibly filtered by some ANT-like expression (where '*' stands for anything but a '.' and '**' stands for anything).

TestSource

A TestSource upon invocation will register tests for executions. I'd expect the following implementations to be provided by JUnit:

• JUnit3TestSource which essentially wraps TestCase implementations. If the decision is not to support JUnit 3 this will not come into existence

• JUnit4TestSource wraps classes with @Test or @RunWith annotations

• JUnit5TestSource registers test by instantiating the new test classes, when the approach of test registration in initializer is used.

In order to allow execution of single tests, a test source should

1. register itself, or a specification of itself together with the test

2. register the TestSourceFinder from which it was created

This is to allow IDEs and similar tools to reexecute tests or test sets efficiently even when they don't explicitly know from where these tests are coming.

Additionally the TestSource should be able to register tags and/or key-value pairs with each test, probably the information above should just be a special case of such tags or key-value pairs.

Test Extension Points

Proposal 1 - one extension point

Rules similar to what we have in JUnit4 should be applicable based on tags/key-value pairs and allow for execution of code before, after and instantiation of a test or a test set defined by a common tag/key-value pair.

Ideally Rules would be used to implement as much as possible of the core functionality. Especially parallel execution, ordering, setup/teardown and parameterized test should be handled by rules. This ensures that Rules are flexible enough to support a wide range of use cases.

• that's a lot of functionality for one concept. Some of these are vastly different responsibilities than are provided by @Rule and @ClassRule . What's the benefit to having this be one thing vs. 2 or 3? - kcooney

  • The idea to make all this stuff work based on one concept, is to force that concept into becoming really universal so one can mix and match instances of that concept for any purpose anybody might come up with. The idea would be to make it like lists in LISP: really simple, yet powerful and omnipresent. Of course there is the other way of making it a hellish complicated thing, which can do all kinds of things but nobody understands it. That would be a bad thing. - schauder

  • if the API can be simple, then I would agree (but please don't call it a Rule), In most cases, I think we should follow the Single Responsibility Principal. More generally, I would prefer to hash out design ideas like this with prototyping in pairs - kcooney

  • About the SRP: The single responsibility of such an interface would be: modifiy a registered set of tests; I choose the term Rule just because I guess everybody here knows the term. I'm not at all glued to it. - schauder

  • Current rules act as a decorator for a single test or suite. Reordering tests are different responsibilities in JUnit currently; I'm very curious how one would create a simple interface that can do both. Parameterization is certainly a different responsibility because it involves creating an instance of the test multiple times for each method (I am having a very hard time understanding how we could allow more than one object to parameterize a test method, but we support multiple rules modifying the tests in a class). In any case, I still believe we shouldn't focus too much on design on the wiki, and instead do design in pairs in person (or for those that can't go in person, over code reviews). - kcooney

  • With tests being lambdas I don't see the need for a facility for parameterized tests. One can simply iterate over the test parameters and register multiple test. Something like this

        public class MyTest{{
            for(int i=0; i++; i<10)
                test("Test #" + i, ()->{
                    // test what ever you need to test using i
                });
        }}
    

    Of course there could be syntactic sugar for this, but that would be responsibility of the TestSource. - schauder

    • Two reasons I am not fond of that solution 1) we want to support parameterized tests even for projects that cannot use Java 8, 2) we don't want to eagerly load all parameterized data upfront - kcooney

  • I think I found the abstraction I was looking for - @schauder

In order to handle arbitrary tags, it must be possible to register Rules independent from TestSources.

Proposal 2 - MetaRule and StatementFactory

MetaRule

• Purpose: augment behavior at the method or class level
• Design Pattern: Decorator

A MetaRule would allow a single object to decorate the Statement chain at a class level and method level. It should be easy to have the meta rule create an object that lives for the life cycle of the class or suite that it modifies (without having to store that object in an evil static field).

A test class can have many meta rules.

Use cases include:

• Start a server before any test methods run, and have each method get the connection parameters from the rule
• Create a temporary database before any test methods start, and rollback the transactions after every test method
• Create a dependency injection container before any if the test methods run, and inject data into the test using method injection

StatementFactory

• Purpose: create instances of the test class
• Design Pattern: Factory

A MethodFactory would allow a single object to create one or more Statement instances that run a test.

A test class can have only one StatementFactory.

Use cases include:

• Parameterized tests
• Dependency injection using constructor injection

Modifier

• Purpose: sort, reorder or trim the test tree
• Design Pattern: Visitor

TestListener

A TestListener can get registered similar to Rules, but does not change the execution of tests. It just gets informed about events like:

• TestSourceFinder applied
• TestSource found
• Test registered
• Rule invoked
• Test invoked
• Test ignored
• Test succeeded/failed