Demo: AGENT Test Generation

In this demo, we will try AGENT's test flow generator.

Pre-requisite

Before running this demo, please run through the AGENT exploration demo once to install the necessary software on your machine.

Demo: AGENT Exploration

Starting Flow Generator

If AGENT is not currently running on your machine (you have stopped the Docker compose process), start the flow generator process:

docker run -it --rm -p 9001:9001 -e SERVICE_PORT=9001 aista/flow-generator

After a few seconds, you should see output similar to below:

[2021-04-25 19:08:35 +0000] [9] [INFO] Starting gunicorn 19.9.0
[2021-04-25 19:08:35 +0000] [9] [INFO] Listening at: http://0.0.0.0:9001 (9)
[2021-04-25 19:08:35 +0000] [9] [INFO] Using worker: sync
[2021-04-25 19:08:35 +0000] [12] [INFO] Booting worker with pid: 12
[2021-04-25 19:08:35 +0000] [20] [INFO] Booting worker with pid: 20
[2021-04-25 19:08:35 +0000] [28] [INFO] Booting worker with pid: 28
[2021-04-25 19:08:35 +0000] [29] [INFO] Booting worker with pid: 29
[2021-04-25 19:08:35 +0000] [44] [INFO] Booting worker with pid: 44
[2021-04-25 19:08:35 +0000] [45] [INFO] Booting worker with pid: 45
[2021-04-25 19:08:35 +0000] [53] [INFO] Booting worker with pid: 53
[2021-04-25 19:08:35 +0000] [61] [INFO] Booting worker with pid: 61
2021-04-25 19:08:36,529 INFO success: app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-25 19:08:37.895366: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.031432: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.086896: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.118026: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.143430: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.244071: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.244866: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-04-25 19:08:38.266391: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

Generating Tests

The AGENT flow generator uses pre-trained recurrent neural networks to generate abstract test sequences. The AGENT system combines this information with element classifiers in order to convert the abstract test sequences into executable test cases.

The flow generator exposes a single API endpoint. The endpoint details are below.

Predict

POST http://localhost:9001/v1/predict

Given a seed sub-sequence for an abstract test case, generates the rest of the sequence.

Request Body

Name	Type	Description
	array	A seed sub-sequence. For example: ["observe", "required", "textbox", "firstname"]

200 The fully generated sequence representing an abstract test case. An example response is below:

{
  "sequences": [
    [
      "observe",
      "required",
      "textbox",
      "firstname",
      "try",
      "valid",
      "firstname",
      "click",
      "commit",
      "notobserve",
      "errormessage"
    ]
  ]
}

Submitting Requests

The underlying model has been trained to recognize the following field names and abstract inputs:

Field Types	Fields	Abstract Input
Dropdown	Title	VALID
Textbox	FirstName	BLANK
	Middle	WHITESPACE
	LastName	INVALID_LONG
	Suffix	INVALID_SPECIAL_CHARACTERS
	PrimaryPhone	INVALID_XSR
	SecondaryPhone
	Country
	Address1
	Address2
	City
	State/Province
	Zip/PostalCode
	Telephone

You can mix and match from the above when constructing the seed request sequence. See below for example request sequences:

["observe", "required", "textbox", "firstname"]
["observe", "textbox", "firstname"]
["observe", "required", "textbox", "firstname", "try", "valid"]
["observe", "required", "textbox", "firstname", "try", "blank"]

In particular, the below request is interesting as it was not part of the training data for the model:

["observe", "required", "textbox", "primaryphone", "try", "blank", "primaryphone"]

In each case, the test generator will complete the sequence such that it represents a full abstract test case, where a full test case has three major parts:

• Preconditions (e.g., observe required textbox firstname)

• Actions (e.g, try valid firstname click commit)

• Expectations (e.g., observe errormessage or notobserve errormessage)

The generated test sequences are abstract. The AGENT system parses a test sequence using a custom grammar, and maps each abstract concept to their concrete representations at test execution time, through the user of machine learning-based classifiers.

For example, to concretize observe required textbox firstname, one or more classifiers are needed:

• One that could classify required fields

• One that could classify (either directly or indirectly) a first name field