Exercise: Python Chopper

In this exercise, you can try out an object detection model trained internally at dev-tools.ai.

Updated Instructions (2024)

Open Notebook on Google Colab

Navigate to https://github.com/dionny/ai-tutorial-notebooks/blob/main/chopper.ipynb and click on "Open in Colab" at the top.

Execute Element Detection Model

Follow the steps on the notebook, executing each of the Python code blocks in the order in which they appear. To execute a code block, click the Play icon to the left of the block.

Note: After executing the first code block, an upload file widget will appear. This widget allows you to upload an image file from your computer. This, in turn, is the image that will be sent to the deep learning model for object detection purposes.

Widget For Uploading a File

After uploading a file, simply follow the instructions in the notebook, executing each of the code blocks in the order in which they appear.

Note that some steps may take a few seconds to execute.

Viewing the Results

When viewing the results, you can see all detected elements, each with a screenshot + bounding box, the element type, and a confidence score between 0 and 1.

See a full example below:

Discussion Topic: How useful do you think this would be for testing purposes? Note that this approach allows us to locate elements on an application screen without using a Document Object Model (DOM) or similar page source for mobile applications.

What are some drawbacks of relying on the DOM or on page source?

Congratulations!

For making it this far. Perhaps even without knowing it, you just used deep learning to identify elements using only a screenshot! How cool is that?

Legacy Instructions (Deprecated)

In this exercise, you will:

• Create your very own Jupyter server, powered by JupyterHub

• Launch the chopper.ipynb Python notebook

• Execute the specialized element detection model

• View the results produced by the element detection model

Set Up Your Jupyter Server

Navigate to https://jupyterhub.dionny.dev and make sure you arrive at the following login screen:

Jupyter Server Login Screen

Enter the following credentials:

• Username should be your e-mail address or first + last name.

• Password should be admin.

Your e-mail information is not being collected. We need to ensure your username is unique when considering all tutorial attendees.

Click on Sign In.

After a few short moments, your unique Jupyter notebook server will be created for you.

Launch Chopper Notebook

Double click the chopper.ipynb notebook.

Your Own Unique Jupyter Server

Execute Element Detection Model

Follow the steps on the notebook, executing each of the Python code blocks in the order in which they appear. To execute a code block, first, click the block, then click the Play icon.

Executing a Code Block on Jupyter

Note: After executing the first code block, an upload file widget will appear. This widget allows you to upload an image file from your computer. This, in turn, is the image that will be sent to the deep learning model for object detection purposes.

Widget For Uploading a File

After uploading a file, simply follow the instructions in the Jupyter notebook, executing each of the code blocks in the order in which they appear.

Note that some steps may take a few seconds to execute! We're running deep learning models behind the scenes!

Viewing the Results

When viewing the results, you will be able to see a Screenshot with Bounding Box, and a Chopped Element screenshot for every resulting element that is found.

See a full example below:

Discussion Topic: How useful do you think this would be for testing purposes? Note that this approach allows us to locate elements on an application screen without using a Document Object Model (DOM) or similar page source for mobile applications.

What are some drawbacks of relying on the DOM or on page source?

Congratulations!

For making it this far. Perhaps even without knowing it, you just used deep learning to identify elements using only a screenshot! How cool is that?