Custom Artifacts on AWS Device Farm

This post was written by Nikhil Dabhade, Cloud Solutions Engineer, AWS Device Farm

A good test report is essential to proper analysis of mobile app testing. It is not only the summary of test results, but also artifacts like screenshots, videos, crash reports, logs, and app-generated files. A well formed test report provides useful insights, leads to faster resolution of issues, and helps teams focus on the right priorities. With so many great testing frameworks available, with their own strengths and weaknesses, the configuration capabilities for reports are important.

AWS Device Farm provides comprehensive test reporting that includes all of the standard capabilities: screenshots, videos, and crash reports. We also know that developers need access to custom reports and artifacts that are generated by the test framework. Today, we are providing access to any custom artifact (log file, test framework generated file, or application generated file) when accessing the test results within AWS Device Farm.

How to access custom artifacts in AWS Device Farm

Upload your app

Upload your app by using the AWS Device Farm console or the AWS CLI.

Update your test code to configure the path to custom artifacts

In your tests, configure the comma-separated path on the host machine where you want the custom test artifacts to be stored.

Suppose that your base use case is to take a screenshot in a test and save this screenshot. Next, we explore how to configure tests so that you can retrieve this artifact on AWS Device Farm.

Best practice for Appium TestNG and other test frameworks:

You can set the path to save a screenshot of the device screen to the default path $WORKING_DIRECTORY in your test code.  The sample code snippet looks like this:

public boolean takeScreenshot(final String name) {
    String screenshotDirectory = System.getenv("WORKING_DIRECTORY");
    File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
    return screenshot.renameTo(new File(screenshotDirectory, String.format("%s.png", name)));
}

The result is that AWS Device Farm stores the screenshot at $WORKING_DIRECTORY on the host machine where the tests are executed. You can use almost any path on the host machine with a few exceptions (and they are explained in the console tool tip).

We recommend using the environment variable $WORKING_DIRECTORY in your test code. Your code can be in any language and framework that AWS Device Farm supports.

Build and package your tests

Package the test, using the instructions in Working with Appium Java TestNG for Android and AWS Device Farm.

Upload your tests to AWS Device Farm

Upload the .zip package you just generated to AWS Device Farm.

Choose the devices against which you want to run your tests

Select the devices against which you want your tests to run.

Configure your test run

On the Specify device state page, configure the path on the host machine where you want the screenshot to be saved. If you had an artifact that gets saved on the device, you can specify that path as well. The following image shows how to specify both of these paths.

Schedule your test run

On the Create a new run page, review and confirm your settings. When ready, choose Confirm and start run.

Retrieve the artifacts

After your tests are complete, you can retrieve your custom artifacts through the AWS Device Farm console from the Files section for the Teardown test of your test suite as shown in the following image.

Access downloaded artifacts locally

To access the artifacts from the test, choose Customer Artifacts and download the .zip file. The contents of the .zip file are shown next.

Conclusion

We described how to configure the path where you store the custom test artifacts in your test code and how to provide the same path to AWS Device Farm. This enables AWS Device Farm to retrieve these artifacts from the specified location on the host machine and the device under test.

With this feature, you have easy access to any type of file produced by your test code and app. This gives you the ability to augment your test reports with relevant artifacts. You can also use these to store files like videos, screenshots, excel files, xml files, custom logs and binary files.