The following post is an inside peek into Magento’s QA process as written by Joanna S., Magento’s QA Manager.
Testing an e-commerce application (such as Magento) is not a trivial task and requires careful consideration - everything the user could do must be anticipated, as well as the effect their actions could have upon the system. Every link must be checked, every form must be filled out and submitted, every page must be visited, every transaction must be carried out, etc.
Therefore it is vital that a thorough test plan, including testing scenarios and testcases be conceived before actual testing can be performed. Test cases are developed and managed with TestLink, an open source test management and execution system.
The above mentioned test plan needs to be based on and incorporated into the development schedule in order to assure that all new functionalities are tested as soon as they have been implemented, leaving enough time to report and fix any possible issues.
Apart from system functionality, the following website quality aspects also need to be considered during the testing process:
- Essential analysis - navigation, information structure, broken links, content errors, etc.
- Usability analysis
- Accessibility problems
- Code quality - markup errors, broken scripts, code structure, etc.
- Cross-browser compatibility issues
- Layout issues
We all want Magento to be as bug-free as possible, therefore the testing procedure we employ needs to be both extensive and comprehensive ensuring the highest quality solutions possible are delivered. Our quality procedures are built into the development life cycle and therefore ensure reliability at each stage of development.
Magento internal testing is performed by a team of experienced QA testers, who report any possible bugs, missing features or possible improvements found using Mantis bug tracking system. A detailed and thorough report on any issue is provided, including problem description, replication steps, screenshots, bug severity and priority, as well as suggestions for possible solutions. Each issue is assigned to a proper developer designated for the project, according to their scope of responsibility.
Developers resolve bugs assigned to them based on the schedule and priority specified. Bugs related to upcoming release are treated as top-priority. When a bug/issue is resolved, its status changes and it’s immediately being checked by a QA person who either closes it or re-opens when the problem still occurs.
Feedback from Magento community is a great benefit for the whole QA process. Community members also have the possibility to report any bugs found, which are then reviewed, verified and assigned to a proper developer to fix. Posts on the forum are a great help too, as community memebers often come up with ideas for further improvements.
Testing New Releases
Testing new releases is based on the list of functionalities scoped for an upcoming release. First, a feature is compared against its specification and any inconsistencies identified are carefully examined. Not only are separate features carefully tested, but also their interrelation and the way they might affect other system functionalities. The process also includes regression tests done on all related areas in order to assure that both the existing and new functionalities work as expected. System testing is the final step performed on a complete, integrated release to evaluate the application’s compliance with its specified requirements.