Selenium Concurrecy Errors on Gitlab Runner

Issue

I am developing java 17 selenium 4 automated tests. Every dependency is up-to-date.

Recently I started to have a lot of concurrency errors in my automated tests, recently even in sanity test suite (which only requires selenium to login in to app from 3 different accounts). This is kind of ruining my automation, as I can't finish my runs, that succeed without an error locally. My testng.xml configs permits up to 3 tasks at the time (used to be much more). Very often the error also involes inability to run @AfterMethod, which only makes screenshot and quits driver. I rarely can finish suite without having it occur at least once. Here is sample error, which is recurring very often:

    org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
    Build info: version: '4.1.3', revision: '7b1ebf28ef'
    System info: host: 'runner-167c7273-project-30802015-concurrent-0', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.47-56.37.amzn1.x86_64', java.version: '17.0.2'
    Driver info: org.openqa.selenium.chrome.ChromeDriver
    Command: [2ab6c435b804fb5c01fb528c4baeed12, get {url=****************/auth?tab=login}]
    Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 100.0.4896.60, chrome: {chromedriverVersion: 100.0.4896.60 (6a5d10861ce8..., userDataDir: /tmp/.com.google.Chrome.PFvJ4e}, goog:chromeOptions: {debuggerAddress: localhost:40989}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), se:cdp: ws://localhost:40989/devtoo..., se:cdpVersion: 100.0.4896.60, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
    Session ID: 2ab6c435b804fb5c01fb528c4baeed12
        at app//org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:72)
        at app//org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
        at app//org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
        at app//org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)
        at app//org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
        at app//org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
        at app//org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:124)
        at app//org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:181)
        at app//org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167)
        at app//org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:567)
        at app//org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:330)
        at app//com.*******.****.base.BasePage.load(BasePage.java:52)
        at app//com.*******.****.pages.registration.LoginPO.load(LoginPO.java:38)
        at app//com.*******.****.steps.ManageAccountStep.login(ManageAccountStep.java:21)
        at app//com.*******.****.tests.register.LoginTest.loginToApp(LoginTest.java:23)
        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
            at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:66)
            ... 15 more
Sanity Test > Prod Env Login > com.*******.****.tests.register.LoginTest > stopDriver FAILED
    org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
    Build info: version: '4.1.3', revision: '7b1ebf28ef'
    System info: host: 'runner-167c7273-project-30802015-concurrent-0', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.47-56.37.amzn1.x86_64', java.version: '17.0.2'
    Driver info: org.openqa.selenium.chrome.ChromeDriver
    Command: [2ab6c435b804fb5c01fb528c4baeed12, screenshot {}]
    Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 100.0.4896.60, chrome: {chromedriverVersion: 100.0.4896.60 (6a5d10861ce8..., userDataDir: /tmp/.com.google.Chrome.PFvJ4e}, goog:chromeOptions: {debuggerAddress: localhost:40989}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), se:cdp: ws://localhost:40989/devtoo..., se:cdpVersion: 100.0.4896.60, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
    Session ID: 2ab6c435b804fb5c01fb528c4baeed12
        at app//org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:72)
        at app//org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
        at app//org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
        at app//org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)
        at app//org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
        at app//org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
        at app//org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:124)
        at app//org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:181)
        at app//org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167)
        at app//org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:567)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:622)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:626)
        at app//org.openqa.selenium.remote.RemoteWebDriver.getScreenshotAs(RemoteWebDriver.java:351)
        at app//com.*******.****.base.BaseTest.takeScreenshot(BaseTest.java:80)
        at app//com.*******.****.base.BaseTest.stopDriver(BaseTest.java:73)
        at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
        at app//org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:135)
        at app//org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:65)
        at app//org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:381)
        at app//org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:319)
        at app//org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:803)
        at app//org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:772)
        at app//org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:748)
        at app//org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:220)
        at app//org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:80)
        at app//org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:17)
        at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at [email protected]/java.lang.Thread.run(Thread.java:833)
        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
            at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:66)
            ... 35 more

And this is my gitlab-ci.yml

image: markhobson/maven-chrome:jdk-17

stages:
  - prepare-cache-sanity-test
  - prod
  - test
  - hubspot

.retry-policy: &retry-policy
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure

.cache_template: &push_build_cache
  artifacts:
    name: "$CI_JOB_NAME-$CI_JOB_ID"
    expire_in: 2 days
    paths:
      - drivers
      - build
      - .gradle

.cache_template: &pull_build_cache
  dependencies:
    - sanity-test

.artifacts: &test_report_artifacts
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - build/reports/tests/test

sanity-test:
  stage: prepare-cache-sanity-test
  script:
    - ./gradlew test -DsuiteFile="common/sanity.xml"
  <<: *retry-policy
  <<: *push_build_cache

sp-test-prod:
  stage: prod
  script:
    - ./gradlew test -DsuiteFile="common/prod.xml"
  <<: *test_report_artifacts
  <<: *retry-policy
  <<: *pull_build_cache

sp-test-regression:
  stage: test
  script:
    - ./gradlew test -DsuiteFile="common/regression.xml"
  <<: *test_report_artifacts
  <<: *retry-policy
  <<: *pull_build_cache

sp-test-hubspot:
  stage: hubspot
  script:
    - ./gradlew test -DsuiteFile="common/hubspot.xml"
  <<: *test_report_artifacts
  <<: *retry-policy
  <<: *pull_build_cache

Solution

I run 3 gitlab jobs (selenium TestNG tests) concurrently, each on 4x5gb or 4x8gb runner (whichever is free). Yet I still have reduced number of concurrent task for each runner only to two. But I still sometimes run on the issue, so I identified which exceptions are showing up when tests fails with no apparent reason. Then, I created listener, which I added to each task's .xml configuration files. The only job of this listener is simply to retry tasks that failed because of identifed exceptions.

This code is place below <suite> tag in tasks .xml files

<listeners>
    <listener classname="com.xxx.xxx.listeners.retrying.ApplyRetry"/>
</listeners>

This is ApplyRetry class:

public class ApplyRetry implements IAnnotationTransformer {

    @Override
    public void transform(
            @NotNull ITestAnnotation annotation,
            Class testClass,
            Constructor testConstructor,
            Method testMethod
    ) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

And the RetryAnalyzer which does the real job:

@Slf4j
public class RetryAnalyzer extends RetryAnalyzerCount {

    public RetryAnalyzer() {
        setCount(ConfigLoader.getInstance().getTaskRetries());
    }

    @Override
    public boolean retryMethod(@NotNull ITestResult result) {
        Throwable throwable = result.getThrowable();
        log.warn(
                "Exception caught by listener details: Cause - {}, Message - {}.",
                throwable.getCause(),
                throwable.getMessage(),
                throwable
        );
        return throwable instanceof TimeoutException
                || throwable instanceof NoSuchSessionException
                || throwable instanceof UnreachableBrowserException
                || throwable.getCause() instanceof TimeoutException;
    }
}

So the test is retried max. two times (set in config file), but only when it stumbles upon one of these exceptions: TimeoutException, NoSuchSessionException, UnreachableBrowserException or any throwable that is caused by TimeoutException.

I have flakes now less than once per 1000 test instances invocations.

Answered By - RafalLovesWorking