Testing Android

#Testing Android We want to test the key challenges of Kiwix for Android. Here's an overview of the high and medium priority areas. They can and will be expanded as we progress with our testing.

##High priority Here are the current high priority areas we want to test effectively:

• storage: where the zim files are stored. Some devices have several locations. Some users report problems reading from the storage on their devices. How can we help users recognise limitations in the storage available e.g. if they try to download a large zim file (say 30GB) to a device that only has 23GB available (perhaps on a '64'GB micro sd-card). Do we warn them, prevent them, offer a file management facility, or ...?
• format of the partition: large zim files can't be stored as a single file on some file formats. Hopefully we help users download and transfer the content packaged so it'll work on their devices. They may also try transferring and/or copying zim files to devices. How can we make the UX as easy and helpful as possible?
• effects of other software, particularly screen overlays. The effects are particularly annoying if users have invested time and money downloading the large custom apps e.g. Medical Wikipedia only to face installation problems limitations and problems with the Google Play download for the custom apps.
• search, including full text search. Does it work correctly, is it performant, how do users discover / realise full text search is available??? What are the practical limitations? what are the accidental limitations? How are results displayed so they meet user's expectations and are helpful show their relevance? Overall, how can we improve the UX of search?
• supporting the variety of device models e.g. those with a physical menu button to provide a recent example? also the screen resolution is likely to have a material effect, etc. testing the functionality on sufficient variety of devices e.g. the fairly recent exception when trying to inflate a button on Android 4.x There are also comments about night-mode, etc. in the App Store - can we help improve the UX for this and other features such as read-aloud, changing font size, etc. multi-zim file support especially if/when we split content such as the medical topics into sub-sets where users may want to download topics/themes (or whatever we call them) and then search across them, etc.
• updates and upgrades. user-centric data such as bookmarks might be worth preserving when users change devices. Could they save and restore them across versions, etc.?
• functionality and behaviour across languages/locales.

##Medium priority

• meeting user's wishes and expectations. This will probably include comparisons with Kiwix on other platforms, the official Wikipedia app(s), and other similar apps (again some reviewers have asked for us to incorporate functionality from other wikipedia-like apps)

#Testing strategy We would like to have a strong suite of automated tests for the Android app. Automated tests will reside in the same repo as the Android app.

The automated tests will not cover various aspects of how the app behaves, and they're unlikely to extend beyond the application's GUI. Therefore we also need to test the app interactively.

Data collected via Google Play's Developer Console, including reviews, crash reports, etc. can help us triagulate potential problems and devise tests for them.

#Ways to test Thanks to Bitbar we now have access to their device farm. Individually we have several devices, Julian Harty also has a fairly eclectic collection of older mainstream and oddball devices. We can test the app interactively on any and all of these devices.

We can also create automated tests to test some aspects of the app. The initial set of candidate frameworks are:

• Espresso (from Google) which seems powerful, capable and likely to be well used in the industry as it's from Google and part of the Android SDK.
• UI Automation (from Google) which can test beyond a single app, for instance it can access the Settings app to change settings on the device.
• Robotium (created by Renas Reda). It's a mature and well respected test automation framework. I'd be happy to ask if he'd be willing to help create some Robotium tests for Kiwix and/or give us his advice on testing Kiwix using automated tests.

At the moment, we don't expect to focus on automated unit tests for the app as the main challenges are at the Activity / app scope.

As Kiwix is heavily content-centric many of our tests will need to account for the content in the zim files installed and available where the tests are performed. We could start by using some known existing content, however as the tests get more sophisticated, particularly for search testing in various languages, we may benefit from generating / obtaining custom zim files for testing.