Prompt Guide

Please read this guide on how to use camelQA.

​

The ultimate tip to using camelQA

Use accessibility labels.

This does not mean you need to add unique element identifiers to your code (doing that will not impact camelQA at all). However, camelQA can see accessibility labels that appear to the screen reader.

Follow these guides to easily enable the screen reader so you can hear what camelQA sees and be more precise with your test cases.

Example of accessibility labels on Youtube Shorts.

This button is labeled back button for iOS but navigate up button for Android.

Using the screen reader allows us to understand what qaml sees the icons labeled as.

​

Explain how to use your app

Let’s say you want camelQA to record a video. Don’t do something like this:

q.execute("record a video")

This leaves a lot to be interpreted which means there will be inconsistent results. Instead, tell camelQA how to complete the action.

q.execute("tap the record video button")
q.waitUntil("the red timer at the top is greater than 10 seconds")
q.execute("tap the stop recording video button")

Screenshot being evaluated by waitUntil.

The waitUntil will mark passed once the timer at the top is greater than 10 seconds.

​

Using assertCondition() and waitUntil()

How it works: Assert works by taking a screenshot of the current screen and sending the screenshot to a vision model, along with your condition. The model evaluates the screenshot against the condition and returns a pass or fail. Wait until uses the assert condition on a loop until the timeout limit is hit.

When writing your assert, you must create a description specific enough to fail if something is wrong, but also concrete and obviously evaluated. If your assert is not easily evaluated, or could be misinterpreted, you will get inconsistent results.

Here is an example:

q.openURL(url: "https://ycombinator.com")
q.assertCondition("the screenshot shows the homepage for ycombinator.com")

Screenshot of the screen being evaluated.

It feels like this should work. However, the use of the term homepage causes the test to fail occasionally with this error:

failed - QAMLException(reason: "Assertion failed: the screenshot shows the homepage for ycombinator.com. 
Reason: Optional(The screenshot shows a mobile view of a webpage related to Y Combinator, but it does not confirm that this is the homepage of ycombinator.com. 
The URL \'ycombinator.com\' is visible, but it does not specify that this is the homepage.)")

To fix this, we need to remove the ambiguity from the prompt. Using the term web page rather than homepage solves the problem. The model is no longer uncertain whether or not the current page is classified as the homepage. With a subtle shift in our language, we get a reliable test case:

q.openURL(url: "https://ycombinator.com")
q.assertCondition("the screenshot shows the web page for ycombinator.com")

Here is a non-exhaustive list of things you should avoid doing in your asserts and waits:

• listing out your app, ex. q.assertCondition("I am in Youtube Shorts")
• imprecise description of color, ex. q.assertCondition("there is a blue send button") but the button is white with blue lettering. If you are going to use color in your assert, ensure your description is precise.
• relying on descriptions that require experience with your app, ex. q.assertCondition("I am in the community playlist page") but there is no identifying text that says ‘community playlist’
• relying on what is not there, ex. q.assertCondition("there is no modal"). This can work if it is very obvious, but prompting based on what is not there make hallucination more likely.

tl;dr assertCondition and waitUntil are very powerful, but require a condition that is obviously evaluated.

​

Tapping Items in a list

Our executor model can get confused when deciding what to tap in a list. For example, q.execute("tap the first item in the list") or q.execute("tap the first item in the search results") do not produce consistent results.

Example

// both of these commands confuse camelQA and produce inconsistent results
q.execute("tap the first item in the search results")
q.execute("tap 'San Francisco' in the search results")

camelQA gets confused which element counts as the first item and will sometimes choose the text in the text field rather than the search result.

Instead, specify the accessibility label of the list (the list is labeled as ‘location list’ in this example) and the exact text match.

// this works to tap the first search result in the example screenshot every time
q.execute("tap the first result in the location list, 'San Francisco, CA United States'")

​

Tapping Images

Images are not identified at this time

Images are treated as black boxes for every command aside from assertCondition and waitUntil. The model recognizes the presence and coordinates of an image but does not identify its content.

camelQA cannot tap an image based off of it’s description unless you are using accessibility mode and there is a device-level label that includes an image description.

We recommend using placement cues to interact with specific images. Example:

// assert uses a vision model to see and evaluate the screen, so this will work
q.assertCondition("There is an image of a coffee table") 

// execute does not use vision on images. 
q.execute("Tap the image of the coffee table") // this will most likely fail

// Instead, use labels and positioning to interact with images.
q.execute("Tap the image under the header Coffee Tables")