This case study highlights AIM Consulting’s recent work on a custom point-of-sale system for one of North America’s largest food and beverage retailers. The client faced challenges with the speed of feature development and was getting delayed in releasing code due to extended test cycles.
AIM Consulting successfully addressed these challenges by identifying high value tests and automating them with a UI Automation framework. Collaborating closely with the client’s teams, AIM refined their test automation strategy, ultimately enhancing the developer experience by refining feedback loops, reducing cognitive load, and improving flow state.
Case Study Contents:
- Business Challenge: New POS System, Focus on Software QA
- Approach: Work with QA Team, Improve Overall Test Automation Strategy
- Results: Improved Developer Experience and Test Automation
What AIM Consulting did:
- Improved feedback loops by letting developers know early and clearly when bugs appear
- Reduced cognitive load by creating dependable and transparent tests that avoid false alarms
- Improved flow state by focusing test result messaging to alert only those affected, allowing the rest of the team to stay focused
- Enhanced the robustness of the test automation framework to reduce false failures
- Automated a significant portion of the manual regression tests, reducing the time and effort required for manual testing.
Business Challenge: New POS System, Focus on Software QA
A major food and beverage retailer embarked on a project to develop a new, custom point-of-sale (POS) system, intended to replace its current commercially licensed platform. This new system features an HTML/JS/React User Interface and a GraphQL API that connects requests to multiple back-end services.
Given the importance of maintaining seamless employee and customer experiences, rigorous Software Quality Assurance (SQA) was deemed critical for the project’s success.
Approach: Work with QA Team, Improve Overall Test Automation Strategy
AIM Consulting was engaged to enhance SQA and test automation with a focus on improving the developer experience (DevEx), which considers three main factors: feedback loops, cognitive load, and flow state. We assembled a team of test automation experts, including three Software Development Engineers in Test (SDETs) and a Technical Program Manager (TPM), to address these factors by:
- Improving Feedback Loops: Providing developers with early and clear notifications of bugs in the application.
- Reducing Cognitive Load: Creating reliable and transparent tests to avoid false alarms.
- Enhancing Flow State: Tailoring test result messaging to alert only relevant team members, allowing others to maintain focus.
AIM’s team embedded with the client’s test engineers to understand existing workflows and align improvements with the team’s current practices.
Our experts began by identifying numerous manual regression test cases, many of which were high-priority but impractical to perform manually for each software release. The team prioritized automating these high-priority tests using a UI Automation framework. These automated tests were integrated into the build pipeline to run as a final stage, thus providing developers with rapid feedback on new builds.
Addressing Project Dynamics and Challenges
As AIM progressed, we encountered several challenges:
- Complex Manual Tests: Manual tests were high-level and required the expertise and in-depth system knowledge of a QA engineer to run.
- Constantly Changing API Layer: The API layer the UI under test relied on was the product of hundreds of engineers’ work and was constantly changing. Data structure and content were highly variable and impacted the UI’s behavior, which introduced instability in test automation.
- Large and Redundant Test Case Body: The large number of test cases lacked clear prioritization and contained significant duplication, complicating the automation process.
Strategic Improvements
AIM listened carefully to the client’s software quality stakeholders and implemented a multi-faceted strategy to address these challenges:
- Continued automation of manual cases
- Collaborated with the manual QA team to develop detailed test steps for automation
- Refactored the test automation framework to be more resilient to data changes in the API
- Prioritized high-impact, high-value cases in collaboration with the manual QA team
- Continued to emphasize the importance of test result clarity by improving both the instant message notifications and the test results and output generated with each run
The simple approach for this project would have been to automate the test cases executed by the manual team one by one, thereby gradually increasing the number of automated cases in the pipeline. This method would have incrementally improved the developers’ experience by reducing the time needed for testing with each release.
However, AIM’s test engineering experience allowed us to take on systemic improvements that made more fundamental changes to the overall test automation strategy.
Results: Improved Developer Experience and Test Automation
Work continues on all of this project, and significant opportunities remain for improvement. But in the course of this engagement to date, the AIM team has improved the overall experience by:
- Increased Test Coverage: AIM successfully automated a significant portion of the manual regression tests, increasing overall test coverage and reducing the time and effort required for manual testing. This automation allowed for more frequent and thorough testing of new builds.
- Clarified Test Knowledge: Through detailed analysis and collaboration, AIM clarified and refined the existing body of tests. This process reduced redundancy, improved prioritization, and ensured that automated tests were focused on high-impact areas.
- Reduced False Failures: By enhancing the robustness of the test automation framework, AIM reduced the number of false failures caused by frequent changes in the API. This improvement increased the reliability of test results and minimized unnecessary disruptions for developers.
- Improved Test Output Understandability: AIM focused on improving the clarity and relevance of test results and notifications. This included refining instant message notifications and the output generated by each test run. These improvements provided more actionable feedback to developers, helping them address issues more efficiently.
AIM worked in close partnership with all client stakeholders and was careful to acknowledge the QA team members’ and project leaders’ expectations and needs.
At the same time, we brought extensive strategic experience to move forward the fundamentals of the QA practice and toward an improved overall experience for the project. Our experts provided strategic, high-level value by delivering on the tactical needs of the client with the overall strategy in mind.
The incremental and strategic improvements made by AIM led to a more dependable and efficient testing process. The enhanced developer experience contributed to better productivity and overall project success. The software development team has benefitted from these improvements with both higher job satisfaction and increased speed to market for critical features.
By embedding with the client’s team, understanding their needs, and addressing fundamental challenges, our experts delivered high-value, systemic improvements that supported the project’s long-term success. This partnership not only strengthened AIM’s relationship with the client but also demonstrated the critical role of strategic test automation in complex software development projects.
Insights By
Ready to Improve Your Developer Experience?
AIM’s experts can help you streamline your software quality assurance, enhance test automation, and boost your team’s productivity. Learn how we can transform your development process and drive your project’s success.
