Improving the QA Testing Process of Desktop, Web, Mobile, and XR Applications through Scriptless Testing using the TESTAR Tool
- Autores: Fernando Pastor Ricos
- Directores de la Tesis: Tanja E.J. Vos (dir. tes.), Beatriz Mariela Marín Campusano (dir. tes.)
- Lectura: En la Universitat Politècnica de València ( España ) en 2025
- Idioma: catalán
- Tribunal Calificador de la Tesis: Pablo Javier Tuya González (presid.), Manuela Albert Albiol (secret.), Emil Alégroth (voc.)
- Programa de doctorado: Programa de Doctorado en Informática por la Universitat Politècnica de València
- Materias:
- Enlaces
- Tesis en acceso abierto en: RiuNet
- Resumen
Software is a fundamental pillar in sectors such as banking, video games, and healthcare. Software testing is the most commonly used method to ensure the quality and reliability of these systems. By rigorously exercising software systems and collecting evidence, testing enables informed decisions about a software product's quality.
Human testers play a vital role in software testing, bringing intuition, creativity, and a deep understanding of real-world contexts. However, testers' efforts should not be misdirected to execute manual and repetitive software interactions, as this leads to excessive workload, frustration, and fatigue, resulting in decreased attention and limited testing time.
Scripted testing automates testing through sequential scripts and test oracles, reducing manual effort and helping maintain software quality by detecting unintended regression failures. However, the need to maintain these scripts and the unmanageable number of scripts required for large and complex systems make scripted testing an insufficient approach.
Scriptless testing is a complementary automated approach that does not rely on scripts but dynamically explores software with non-sequential actions. This approach introduces randomness, which helps to uncover states and failures not identified by manual or scripted testing.
TESTAR is an open-source scriptless testing tool that has demonstrated its coverage and fault detection effectiveness when testing industrial desktop and web applications through the Graphical User Interface (GUI).
The modular architecture of TESTAR enables the tool for further improvement to: (1) connect with other complex software systems, (2) incorporate additional exploratory algorithms for test effectiveness and efficiency, (3) integrate a diverse set of test oracles to cover a wide variety of features or discover its faults, and (4) be validated with industrial and well-known open-source projects.
In this thesis, we conducted several research collaborations with industrial partners and open-source projects to improve TESTAR. We extended its interaction capabilities by integrating frameworks to test desktop, web, mobile, and eXtended Reality (XR) applications, added exploratory algorithms to improve the test effectiveness and efficiency of action selection mechanisms, and integrated a new set of test oracles for automated software assurance and fault detection. These novel extensions were evaluated with industrial applications and well-established open-source projects.
Our empirical results demonstrate a significant improvement in TESTAR capabilities. First, TESTAR can successfully connect to and test various desktop, web, mobile, and XR game applications. Second, a state model inference processes, a distributed inference approach, grammar-based action selection rules, and an interactive algorithm for navigating game environments help significantly improve the effectiveness and efficiency of scriptless exploration. Third, the integration of online test oracles and offline delta change detection oracles enabled TESTAR to detect diverse types of software failures and identify and highlight GUI changes between delta versions of the same software application. Fourth, TESTAR has proved to be complementary to manual and scripted testing approaches in industrial organizations. Finally, we additionally collaborated to implement a Behavior-Driven Development (BDD) approach for automating regression tests of an XR industrial game system.
This thesis concludes that using a scriptless testing tool like TESTAR can significantly improve the Quality Assurance (QA) testing process of desktop, web, mobile, and XR applications by increasing coverage and detecting unexpected failures.
