Zou Dewen , Director of qunar mobile application development , Responsible for the management of pre-sale and client .2012 Where to join in , Successively worked in strategy and air ticket business department , Good at client and cross end technology stack RN,Flutter, Deep research on device fingerprints , Leading the domestic mainstream of air tickets RN transfer ,TARS Development and promotion of automatic test system .
Cui Yu ,2018 Joined qunar.com in 2007 , Mainly responsible for the main route of air tickets 、 Plane ticket iOS client 、TARS UI Automation system 、iOS The end fingerprint reinforcement and other aspects of the work .
1. Project background
Business pain points
Internet APP Test in “ Fast iteration ” In the development mode of , Every iteration is accompanied by a variety of tests , Among them, functional integration testing is an important and main aspect , And because of the functional feature verification and existing function regression that need to be carried out in the process of version iteration , It requires testers to be able to test all kinds of features of the functionality , But also to ensure that the original function is not affected . In frequent iterative engineering ,APP The function and structure of is becoming more and more complex , The impact of the new code is growing , There's more and more scope for regression, and APP There are all kinds of models and system versions at the end , This leads to the potential risk of missing test and more and more repeated work . Take qunarnet's air ticket business as an example , smoking Case The test point is 300+, The monthly version is back Case The test points are 1000+, Every time the monthly version returns, it costs testers a lot of energy , The quality of testing depends on the quality of testers , And it's possible to miss , Every time the monthly version returns, at least 10 individual QA One day focused testing , Very inefficient , And it's often online bug And faults .
In order to solve the problem of regression testing workload, which is the biggest workload , Improve work efficiency , Reduce the potential risk of omission , Reduce repetitive work , The project hopes to complete the script recording in a WYSIWYG way , Even if the tester can't program and doesn't understand scripts , You can also click and drag through normal users , Automatically complete script recording , So as to greatly reduce the cost of automation maintenance of enterprises and projects . In view of the multi terminal characteristics of mobile manufacturers , Developing a set of UI Automated testing solutions . Support TestCase Automatic repetition of , In a variety of models compatibility test , The back-end response data is controllable , And request data for the execution result 、 Response data and interface screenshots for verification , The correctness of the assertion results in .
2. Scheme selection
2.1 Test categories
Automated testing is a testing technology developed with software development technology . Software testing includes white box testing and black box testing , And automated testing was the first to use unit testing in white box testing . This test method is more efficient , Easy to implement . The rise of black box automated testing is a little later , But it is also widely used at present . The principle of black box automatic testing is to simulate the user's operation behavior with programs and scripts , Verify key checkpoints of software in an intelligent way . In desktop software and Web The age of technological prosperity , A large number of excellent automated testing tools have emerged, such as QTP and webdriver, These tools have gradually matured through continuous improvement , It can meet the requirements of automatic testing in traditional applications .
APP The automated testing of includes the following ：
1、 According to the purpose of the test ：
A functional test ： Test the functions of products and modules .
Performance testing ： Test the performance of the system .
Pressure test ： Test the load capacity of the software or system , Dig for hidden dangers .
Compatibility test ： Test the compatibility between products and hardware and software , For example, the compatibility of software on different Android models .
Safety test ： Find software security problems in different ways , Like information leakage 、 Illegal use 、 Vandalism and so on .
Other special tests ： For example, weak network test 、 Power consumption test 、 Fluency test and so on .
2、 If we divide it according to the software development stage , Every stage can do ：
unit testing ： White box test the independent modules in the program , The purpose is to test the correctness of the basic components of software .
Integration testing ： Through the combination test of unit modules , The purpose is to verify whether the interface between unit modules is correct .
The system test ： Test the whole system completely , Verify the correctness and compliance of the whole system .
regression testing ： When software changes , Verify the functional modules that may be affected by this change .
The acceptance test ： The last phase of testing , Ensure software quality before software release or launch .
3、 Other common test concepts ：
Smoke testing ： Smoke test is a simple test of the most basic functions of the software , A low cost way to determine whether software is testable .
Smoke testing comes from hardware testing , When the circuit board is ready , First, there will be a power up , If there is no smoke, the next test will start , Otherwise, the product does not meet the most basic quality requirements , It needs to be remade .
Exploratory testing ： Exploratory is more dependent on the personal experience or expertise of the tester , It depends on the subjective initiative of the tester .
The importance of exploratory testing can be found in the field of game testing , Thousands of players will play games in unexpected ways in various unexpected environments , So the tester of the game should not only master the testing methodology of the system 、 Beyond advanced testing tools , But also have rich experience in the game and explore the test thinking .
Functional integration testing 、 performance 、 Compatibility 、 Interface interaction 、 Access right 、 User experience and so on , There are a variety of models 、 There are so many versions of the device . Among them, function integration test , In the process of each version iteration, we need to verify the functional features and regress the existing functions , It requires testers to be able to test various features of new functions , But also to ensure that the original function is not affected .
as time goes on ,APP It's getting more and more complete , The structure is becoming more and more complex , Every code change has a wider and wider impact , And some of the test conditions are very complex , The corresponding regression range will be more and more , Plus frequent iterations , This turns quality assurance into too much repetitive and meaningful work , In the process of implementation, it will also cause potential missing test risk to a certain extent . Limited by the economy 、 Manpower and other factors , It's also very difficult to fully test the compatibility within a certain period of time .
in summary ： One is that changes in functional testing may not be easy to execute , Second, regression is repeated frequently , Third, compatibility will multiply the above two situations , So in order to solve these problems , Improve work efficiency , in the light of APP characteristic , Developing a set of APP Automation solutions . Support automatic test case execution after packaging , In a variety of models compatibility test , Reduce the repetitive work of testers , Improve test quality , An automatic test system that can judge whether the execution result has passed or not .
2.2 UI Automation trends
The application of smart phones brings a new revolution in software testing technology , Compared to traditional testing , Mobile phone testing is separated from the hardware PC The traditional architecture of , There is a big difference in the way of interaction with users . Mobile software has a unique way of operation in use , Like the user's slide 、 Touch and click operations , How to implement automatic script execution on mobile client becomes the difficulty of this kind of test . And because of IOS,Android The market structure of the two platforms , Most mobile applications need to support multiple platforms , They have different infrastructures , This makes the mobile phone automation testing framework have higher requirements for compatibility . The intelligence and popularization of mobile devices , It also makes more and more people enjoy the convenient life brought by scientific and technological innovation and development . Especially this year , For the first time, the world's network traffic has historically exceeded PC End , This shows that the trend in the future is that the mobile market will dominate , The most widely used mobile devices are APP, Because of its convenient operation , Friendly interaction , And to solve the problems of users , First of all, it is widely used by people . To do a good job in mobile client automation testing , First of all, we need to solve the problem of the robustness of the test script and the compatibility of the platform .
stay APP In the process of using our software , It is necessary to reduce the problems encountered by users as much as possible , Avoidable cause APP The emergence of bug Bring inconvenience to users , The original convenience brings , Become less “ convenient ”. So in the development process , We need to put quality as the top priority , Conduct comprehensive and in-depth quality testing . And the Internet industry is changing , To meet the ever-changing changes, constantly bring forth the new in business and Technology . therefore APP The development mode of the software engineering is adopted “ Fast iteration ” Pattern , In response to the impact of rapid change , It is necessary to ensure comprehensive test coverage , It's important to use less time and cost .
In recent years, mobile client automation tools have developed rapidly , There are a lot of automation frameworks , For example, based on Android Platform UiAutomator and Robotium ,IOS Self contained Instrument etc. , However, none of these tools can effectively solve the problem of platform compatibility , Can't drive with a unified framework .
Testing on different platforms , And there's a bias in identifying client content and elements , The test script is not stable and robust enough . At present, the tools with good compatibility and mature and stable framework are Appium.Appium It's an open source 、 Cross platform and multi language testing framework , Compared to other frameworks , It does not need to recompile the source code when writing test scripts and running tests , There are not too many requirements for programming language in script writing and implementation , The test is lighter, more flexible . Besides ,Appium Adopted C/S framework , Provides a unified external service interface , Make the interaction and control of client or simulator transparent .
There are already some on the market APP Test framework for , Some of them don't have strong cross terminal capability , Support www Web Testing , Some only support Android、IOS Mobile , Support at the same time Android、IOS、www、 There are very few small programs , Some solve the problem of function test , But based on the rapid change, the cost of maintenance is far greater than the cost of use , There are no good solutions to the pain points .
TARS The system through the self-developed program , Using AI image recognition technology ,OCR Character recognition technology , And ge UI Element search engine , It can be done across the end , Cross platform . And based on C/S frame , By sending http Control the mobile device by request , Multi language support can be achieved . combination Jenkins And continuous integration tools , Set up a set of self-development UI There's nothing wrong with the automated test system .
TARS The name comes from 《 Interstellar through 》 This famous science fiction film ,TARS It's a powerful intelligent robot in the movie , Assist the protagonist to complete a lot of difficult tasks .
3. Frame building
3.1 Function sorting
3.2 Frame structure
use TARS+Jenkins+STF As a whole control scheduling system , The whole system is divided into application layer , Control layer , Equipment layer . Through the equipment cluster control system , Select device cluster , Batch run automated testing Case.Case Input system can be based on the picture ,UI Element location identification to determine Case Operation steps of . Almost all systems use less code 、 More flexible control of data python As a programming language .
The first is preparation ： The control system selects the equipment to be tested through the equipment management platform , Install the client , Configure the test environment parameters , Such as AB,MocK. The control system modifies the response data , Save the response data to a testcase in （ In case next time MocK）, Return the server response directly , Or get the corresponding testcase Of Mock data , Controllable APP All kinds of data fields obtained by the client .
Then start execution testcase： Set up the client in the specified device , Execute assignment testcase.
And finally diff Assertion phase , Compare the returned results , Request parameters , The difference between styles and pictures .
- Control system , Control and schedule the actions to be performed in each step correctly . It can forward the request correctly and intercept the response .
- Mock The system can be configured correctly or modify the back-end response data 、 Save and return to the front end , Or a Mock The data goes back to the front end , And save the data of request and response correctly , complete http Forwarding agent .
- Equipment platform , According to the existing device mounted on the platform , Judge availability , Provide the equipment for testing .
- Case entry , Can let the user operate the screenshot manually ,UI Element recognition ,OCR Character recognition and other input methods Case, simplify Case Input method .
Pictures and data diff System , Make the test data and screenshots accurate diff, Eliminate possible factors （ Such as data usage Mock Interface ）.
3.3 Element lookup scheme
automation Case Writing requires the test framework layer to provide functional support for the operation of interface elements , One of the most important is the search and operation of elements .
To write Case Or record Case, We need a basic ability is how to accurately and stably get the page elements . It determines that we record Case The cost of and Case The cost of maintenance .
If the search is not accurate enough , Will directly lead to our Case Unstable operation , Affect the accuracy of execution results .
If the page structure changes , The way we look up elements is not stable enough , Then every function iteration needs to be corrected Case For maintenance , Maintain like that Case The cost of automated testing is much greater than the cost of automated testing , Automated testing doesn't make sense .
A good way to find elements , We need to be in UI In the process of automation testing practice, we continue to explore and improve .
At present, there are three popular element search schemes in the market ：
- adopt XPath lookup
What we got UI The elements are basically XML Organized in form ,XPath Is a door in XML The language in which information is found in a document ,XPath Can be used in XML Find elements and attributes in the document .XPath You can pinpoint an element , But it depends on XML The context of the document structure , May be affected by layout changes , And in iOS It doesn't perform well on the computer .
- Find... By identification
You can do this by identifying elements (accessibility、testID etc. ) To locate an element directly , Or you can match text content to find elements . such , Even if our page structure changes , But the identity of the element to be operated on has not changed , So we don't have to change the code . But we need to identify specific elements , There is also a certain amount of workload and maintenance costs .
- Find... By image recognition
Compared with the element search method of white box , Image recognition does not need to care about the page element structure , Very free element lookup seems to be the future of automated testing . Just take a screenshot of the elements you want to operate on , You can find the element according to the image and operate it .
But at present this way is still very immature , The accuracy of image search is poor , And on different models , It's very likely that the picture won't be universal , Need to take a new screenshot for the model .
|XPath||The positioning mode is flexible and convenient||Context dependent , Interface layout changes will affect positioning|
|identification||Accurate and stable positioning , Not affected by layout changes||It's not easy to distinguish repetitive elements such as list items|
|Image logo||More in line with people's operating habits||Inaccurate positioning , Need to adapt to a variety of resolutions|
In actual use scenarios , We're catering to all kinds of situations , Combine the above three element positioning methods . adopt POCO Libraries encapsulate them .
POCO It's a cross platform UI Automation open source framework . It encapsulates a variety of automation tools , for example ：WDA、UIAutomation etc. .
POCO Will others UI The function of automatic tools to find elements is encapsulated in a unified way , by Case Writing provides convenient, flexible and platform consistent element operation api.
POCO It is very convenient to realize the following functions ：
- Element attributes or id location
- Element relative position search
- Regular matching of elements
- Element interaction control
POCO The way to get elements is to use the factory pattern , So we can also easily extend the custom function , Like image recognition or OCR Character recognition .
because POCO Is based on python Realized , So we can easily deploy and use , Debugging scripts is also very easy .
3.4 Case Write design patterns ——Page Object Model
Only the functional support provided by the framework layer is far from being able to Case The cost of maintenance is reduced to an acceptable range .
Because we're actually writing Case When , You need to write a lot of business logic code . We usually have two problems ：
- Write a Case It involves a lot of interface operations , Just write one Case It's already very time-consuming .
- Business logic or elements change , All the relevant Case All have to be changed , If Case More and more , It would be a disaster to maintain it .
So we need a reasonable design , Be able to reuse code as much as possible , And abstract the concrete business logic , There is no need to write a lot of tedious element search operation code .
Page Object Model (POM) Literal translation “ Page object model ”. This design pattern is designed to create a page object for each page to be tested (class), Encapsulate those tedious positioning operations into this page object , Only necessary operation interfaces are provided .
POM Separate page positioning from business operations , Separate test object and test script , If UI Change page , The test script does not need to be changed , Just change some code in the page object , Improved maintainability .
for instance , We need to test the search results on the flight list page , You can directly call the function of the home page object to jump to the ticket home page , Then call the city and date search function of the first page of the ticket to jump to the flight list page , Then on the flight list page, assert the data you want to test .
How to achieve POM Design patterns
One Page Object object , There are two characteristics ：
- Self elements
- Realization function
Self elements are real page elements , The implementation function is the business function provided by this page object , Such as user login .
such , Even if page elements and interaction logic change , It's also possible to change only a small amount of code to make the relevant Case Can operate normally .
POM Design patterns have the following design specifications ：
- public Methods represent Page Features provided
- Try not to expose Page Inside details of
- Don't assertion
- Method can return other Page Objects
- Page Objects Don't represent the whole page , It can be any part
- Same operation , Different results should be separated （ Log in correctly , Wrong login ）
be based on POM Design patterns , We have achieved very low Case Maintenance cost . Even if there's a big change in the page ,Case There's no need to change .
3.5 Json Mock programme
Actual operation Case In the process of , We found that in addition to the equipment environment , There's another problem that really affects Case The success rate of execution , Namely Case The required data online environment cannot be provided stably .
This problem theoretically provides a set of dead data for Mock that will do , But a lot of Case Need to verify the birth order , Here we use Mock Data that will fail , And to a certain extent , We want to be able to use actual data for validation , In this way, we can find the problem of online interface at the same time .
So we need to analyze the specific problems of various data instability .
Question 1 ： How to get online data needed for testing in real time
Because of the complexity of our business , Data is not reusable to some extent . For example, a set of data of the birth order , Can only be used once , The next time you use it, it will be invalid . in addition , If we want to test the round-trip cooperation Case, We need to know which route has this type of offer . Based on this objective situation , Our test team has developed a set of main process parameter generation interface based on online data .
The interface will filter the online data stored in history , According to the data characteristics we specify , Return to our corresponding online quotation . such , We can get specific data accurately for testing .
Question two ： How to deal with online data in a specific way
For example, we often do some activities online , Some pop-up windows may pop up unexpectedly on the page , Blocking our operations . If our Case Deal with these situations accordingly , That's going to increase a lot Case Complexity , And to judge the situation will also reduce Case Execution speed .
In addition, we may need different types of people to take advantage of the opportunity , We need to make sure that there are such types of passengers in the account .
In this case , We need to modify the online data in real time , In order to make sure Case The data environment needed for execution .
We go through json mock Tools to implement this function . We'll write the handling of the interface in Case in , Start running Case When ,Case Send the processing configuration to the client , When the client gets the data , Be first Mock The tool intercepts according to Case The sent processing configuration processes the data , For example, delete the active pop-up node , Or replace the passenger data . After processing, continue to return the data to the business code . such , To a certain extent, the online data Mock Become the data we need , Guarantee Case Accuracy of implementation .
3.6 Continuous integration solution
In real automation scenarios , We generally need to carry out automated testing in the daily development and testing process . The whole process generally includes the following stages ：
- Build the latest development branch beta package
- Installed on automation equipment
- Set up the operating environment
- Execution automation Case
- End of execution send result report
and , According to different use scenarios , We will perform different tests , such as ：
- Smoke testing —— Back to the point Case, Ensure the basic quality of the measurement
- Business regression testing —— Return to demand related business Case, auxiliary QA History Case Regression
- Performance testing —— Compare the performance degradation caused by this modification
- Compatibility test —— Testing stability on different devices
It must be unrealistic for people to operate , So we need a continuous integration tool to help us do this automatically .
We use jenkins As our platform for continuous integration , The whole process is shown in the figure below :
When developing and testing , Will automatically trigger the construction of automated tasks ：
QA Students can also choose the relevant business during the test Case Regression .
4. Practice process
To 9 Overall monthly earnings ：
Access to the R & D process ：109 A project （9 month 57 individual ） And execution 205 Time （9 month 90 Time ）, Automation return saves manpower about 370 pd（9 Yue Yue 180 pd）; Access to the release process ：Q3 Temporary grayscale version 、 Temporary version and monthly version are issued together 24 Time （ iOS 6 Time and Android 18 Time ）, perform UI automation 100 Times or so （ Cover the main package and 8 An Android channel pack ）, Publishing returns to human resource saving 150+ pd;DailyRun： Daily execution 2 On line inspection for the first time ; Defect interception ： Find out 2 A breakdown and 3 individual P1 bug.