Firebase Robo Test Script keeps failing

Issue

We are using Firebase UI with Phone Auth and it seems Robo Script does not work properly with it. Device like LG Nexus 5, API Level 23 hammerhead works fine following some part of the script but many other physical devices we tried failed following the script.

This is a success OTP with Firebase Auth with LG Nexus 5

This one is totally incorrect at the very beginning Samsung SM-G981U1

And the list of non-working physical device with Robo script goes on, not sure what is special about Nexus 5 or probably it has something to do with the screen resolution? One thing I noticed is after putting 9876543210 in EditText of Firebase UI phone Auth the test device with Robo Script always fails to click Verify Phone Number button where most likely the error occurred except for LG Nexus 5 which successfully execute this part of the script.

Another sample

Here it supposed to hit Verify Phone Number button after successfully putting 9876543210 in EditText which is based on our Robo Script but it didn't and the whole input was replaced with 6504992804 which we never record and not in Robo Script. So our conclusion was the error usually happens due to failure of script satisfying the click of Verify Phone Number button.

Robo Script sample

[
  {
    "eventType": "DELAYED_MESSAGE_POSTED",
    "timestamp": 1627584568123,
    "actionCode": -1,
    "delayTime": 7000,
    "canScrollTo": false,
    "elementDescriptors": []
  },
  {
    "eventType": "VIEW_TEXT_CHANGED",
    "timestamp": 1627584581148,
    "replacementText": "9876543210",
    "actionCode": -1,
    "delayTime": 0,
    "canScrollTo": true,
    "elementDescriptors": [
      {
        "className": "com.google.android.material.textfield.TextInputEditText",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 0,
        "resourceId": "com.devsbitwise.cryptonian:id/phone_number",
        "contentDescription": "",
        "text": ""
      },
      {
        "className": "android.widget.FrameLayout",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 0,
        "resourceId": "",
        "contentDescription": "",
        "text": ""
      },
      {
        "className": "com.google.android.material.textfield.TextInputLayout",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 1,
        "resourceId": "com.devsbitwise.cryptonian:id/phone_layout",
        "contentDescription": "",
        "text": ""
      }
    ]
  },
  {
    "eventType": "VIEW_CLICKED",
    "timestamp": 1627584599899,
    "replacementText": "Verify Phone Number",
    "actionCode": -1,
    "delayTime": 0,
    "canScrollTo": true,
    "elementDescriptors": [
      {
        "className": "com.google.android.material.button.MaterialButton",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 2,
        "resourceId": "com.devsbitwise.cryptonian:id/send_code",
        "contentDescription": "",
        "text": "Verify Phone Number"
      },
      {
        "className": "androidx.constraintlayout.widget.ConstraintLayout",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 0,
        "resourceId": "",
        "contentDescription": "",
        "text": ""
      },
      {
        "className": "android.widget.ScrollView",
        "recyclerViewChildPosition": -1,
        "adapterViewChildPosition": -1,
        "groupViewChildPosition": 1,
        "resourceId": "",
        "contentDescription": "",
        "text": ""
      }
    ]
  },
  {
    "eventType": "DELAYED_MESSAGE_POSTED",
    "timestamp": 1627584605228,
    "actionCode": -1,
    "delayTime": 500,
    "canScrollTo": false,
    "elementDescriptors": []
  },

UPDATE

Last remaining problem is probably the consistency, here I used the same Robo script in first test it succeed but in the second with no changes in the code at all it fails.

First Test (Nexus 5 Virtual SDK 21)

Second Test (Nexus 5 Virtual SDK 21)

First Test (Nexus 9 Virtual SDK 24)

Second Test (Nexus 9 Virtual SDK 24)

First Test (Nexus 7 Virtual SDK 22)

Second Test (Nexus 7 Virtual SDK 22)

Solution

You are right that Roboscript fails to click the Verify Phone Number button, as the result says that Roboscript failed on action 3, which is clicking this button.

Overall, I think the problem might be more related to the API level rather than a specific physical (or virtual) device and the difference in API levels between the device on which you recorded your Roboscript and the device on which you perform it. The best way to overcome any discrepancies is to simplify your Roboscript as much as possible. For example, as long as your target elements have unique identifying attributes (e.g., resource id, text, etc.), you can use just those to identify them, like this:

[
  {
    "eventType": "DELAYED_MESSAGE_POSTED",
    "delayTime": 7000
  },
  {
    "eventType": "VIEW_TEXT_CHANGED",
    "replacementText": "9876543210",
    "elementDescriptors": [
      {
        "resourceId": "com.devsbitwise.cryptonian:id/phone_number"
      }
    ]
  },
  {
    "eventType": "VIEW_CLICKED",
    "elementDescriptors": [
      {
        "resourceId": "com.devsbitwise.cryptonian:id/send_code"
      }
    ]
  },
  {
    "eventType": "DELAYED_MESSAGE_POSTED",
    "delayTime": 500
  },

Answered By - Stanislav Negara