Using Junit4: how can I filter out a class of tests with a custom annotation

Issue

I have a custom annotation to filter out tests at run-time, based on the characteristics of the device-under-test. The annotation can be applied to test classes and to test methods.

    @Target({ElementType.TYPE, ElementType.METHOD})
    @Retention(RetentionPolicy.RUNTIME)
    public @interface PhysicalKeyboardTest {
        boolean keyboardRequired() default false;
    }

To filter out the annotated tests I have a custom test runner:

    public class MyTestRunner extends BlockJUnit4ClassRunner {
    public MyTestRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        return filterKeyboardRequiredTests(super.computeTestMethods());
    }

    private List<FrameworkMethod> filterKeyboardRequiredTests(List<FrameworkMethod> allTests) {
        // create a List that we can modify
        List<FrameworkMethod> filteredTests = new ArrayList<>(allTests);

        // does the test class require a keyboard?
        if (isKeyboardRequired(getTestClass())) {
            // test class is marked "keyboardRequired", filter out all tests

            // PROBLEM: this code causes test-time 'initializationError'

            filteredTests.clear();
            return filteredTests;
        }

        // for each test: does it require a keyboard?
        for (Iterator<FrameworkMethod> iterator = filteredTests.iterator(); iterator.hasNext(); ) {
            FrameworkMethod test = iterator.next();

            // does the test require a keyboard?
            if (isKeyboardRequired(test)) {
                // test is marked "keyboardRequired", filter it out
                iterator.remove();
            }
        }
        return filteredTests;
    }

    /**
     * Determine if the given test class or test is annotated with {@code keyboardRequired}
     *
     * @param annotatable The test class or test
     * @return True if so annotated
     */
    private boolean isKeyboardRequired(Annotatable annotatable) {
        PhysicalKeyboardTest annotation = annotatable.getAnnotation(PhysicalKeyboardTest.class);
        return annotation != null && annotation.keyboardRequired();
    }

The code works as expected for individual test methods that are annotated.

However if the test class is annotated, when the tests are run I get an initializationError

java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:191)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:128)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:416)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:84)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:65)
at com.winterberrysoftware.luthierlab.testFramework.MyTestRunner.<init>(MyTestRunner.java:32)
at java.lang.reflect.Constructor.newInstance(Native Method)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at androidx.test.internal.runner.junit4.AndroidAnnotatedBuilder.runnerForClass(AndroidAnnotatedBuilder.java:63)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at androidx.test.internal.runner.AndroidRunnerBuilder.runnerForClass(AndroidRunnerBuilder.java:153)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:73)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:793)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:547)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:390)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)

This error is probably due to computeTestMethods() returning an empty list. (It fails even more dramatically if a null value is returned.)

It seems likely that filtering on the class-level annotation should be done elsewhere (probably where the list of test classes is created), but I have not been able to find where to do it.

Thanks for any help.

Solution

I figured it out.

Instead of providing a custom BlockJUnit4ClassRunner, I need to provide a custom AndroidJUnitRunner, in conjunction with a custom filter (associated with the annotation).

Add Custom Filter

I added an inner class KeyboardFilter to my custom annotation. The custom annotation now looks like:

/**
 * This annotation is used to mark tests for devices with a physical
 * keyboard (Chromebook).
 * <br><br>
 * The {@code keyboardRequired} param can be used to flag tests that
 * should not be run if a physical keyboard is not present.
 * <br><br>
 * The annotation can be applied to test classes, and to individual
 * tests.
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface PhysicalKeyboardTest {
    boolean keyboardRequired() default false;

    /**
     * When {@code keyboardRequired=true} and the test is running on a
     * device that does not have a physical keyboard, filter the test out.
     * <br><br>
     * Will be instantiated via reflection by
     * {@link RunnerArgs.Builder#fromBundle(android.app.Instrumentation, android.os.Bundle)}
     */
    @SuppressWarnings("unused")
    class KeyboardFilter extends ParentFilter {

        /**
         * Determine if the given test can run on the current device, based
         * on the presence of a physical keyboard.
         * <br><br>
         * Tests that are annotated with {@code @PhysicalKeyBoardTest(keyboardRequired=true)}
         * can only be run on devices with a physical keyboard (i.e. Chromebook).
         *
         * @param description the {@link Description} describing the test
         * @return <code>true</code> if test can run
         */
        @Override
        protected boolean evaluateTest(Description description) {
            if (TestHelpers.isChromebook()) {
                return true;
            }
            // we are not running on a Chromebook (i.e. no physical keyboard is attached)

            // check for test-class and test-method annotations
            PhysicalKeyboardTest testAnnotation = description.getAnnotation(PhysicalKeyboardTest.class);
            PhysicalKeyboardTest classAnnotation = description.getTestClass().getAnnotation(PhysicalKeyboardTest.class);

            // if the test-method and test-class are not annotated, the test can run
            return noKeyboardRequired(testAnnotation) && noKeyboardRequired(classAnnotation);
        }

        @Override
        public String describe() {
            return "skip tests annotated with 'PhysicalKeyboardTest(keyboardRequired=true)' " +
                    "if no physical keyboard is present";
        }

        /**
         * Determine if the given annotation says that a keyboard is required.
         *
         * @param annotation The annotation
         * @return True if no keyboard is required
         */
        private boolean noKeyboardRequired(PhysicalKeyboardTest annotation) {
            return annotation == null || !annotation.keyboardRequired();
        }
    }
}

Add Custom Instrumentation Test Runner

/**
 * An instrumentation test runner.
 * <br><br>
 * Provides a mechanism for filtering out test classes and test methods,
 * based on a custom test annotation.
 * <br><br>
 * This class is specified as the {@code testInstrumentationRunner}
 * in the app's {@code build.gradle} file.
 *
 * @see PhysicalKeyboardTest
 * @see androidx.test.internal.runner.RunnerArgs
 */
@SuppressWarnings("unused")
public class MyAndroidJUnitRunner extends AndroidJUnitRunner {
    // androidx.test.internal.runner.RunnerArgs looks for this bundle key
    private static final String FILTER_BUNDLE_KEY = "filter";

    @Override
    public void onCreate(final Bundle bundle) {
        // add the keyboard filter to the test runner's filter list
        bundle.putString(FILTER_BUNDLE_KEY, PhysicalKeyboardTest.KeyboardFilter.class.getName());
        super.onCreate(bundle);
    }
}

Specify Custom Instrumentation Test Runner

In the app's build.gradle I specify my instrumentation test runner:

        testInstrumentationRunner "com.mypath.testFramework.MyAndroidJUnitRunner"

Add Annotation to Tests

With this framework in place, my tests that include the annotation

@PhysicalKeyboardTest(keyboardRequired = true)

will not be run on devices that do not have a keyboard. The annotation can be applied to the test class, or to individual test methods.

Answered By - Clo Knibbe