Download to read offline
![We measure how difficult it is to get an
accepted answer for a Selenium question
Part 1: Mining Selenium questions on Stack Overflow
Prior work also measures difficulty using the above two metrics [Rosen and Shihab (2016)]
Answering speed
Ratio of accepted
answers
19](https://image.slidesharecdn.com/mscdefenseslidesshahnazfinal-190320180637/75/Investigating-the-Challenges-in-Selenium-Usage-and-Improving-the-Testing-Efficiency-of-Selenium-based-Load-Tests-19-2048.jpg)
![We measure how difficult it is to get an
accepted answer for a Selenium question
Part 1: Mining Selenium questions on Stack Overflow
Prior work also measures difficulty using the above two metrics [Rosen and Shihab (2016)]
Answering speed
Ratio of accepted
answers
19](https://crownmelresort.com/image.slidesharecdn.com/mscdefenseslidesshahnazfinal-190320180637/75/Investigating-the-Challenges-in-Selenium-Usage-and-Improving-the-Testing-Efficiency-of-Selenium-based-Load-Tests-19-2048.jpg)
More Related Content
Similar to Investigating the Challenges in Selenium Usage and Improving the Testing Efficiency of Selenium-based Load Tests
Investigating the Challenges in Selenium Usage and Improving the Testing Efficiency of Selenium-based Load Tests
- 1. Msc Thesis Defense 1 Investigatingthe Challenges in Selenium Usage and Improving the Testing Efficiency of Selenium-based Load Tests Msc Candidate: Shahnaz Mohammedi Shariff Supervisor: Ahmed E. Hassan
- 2. 2 Selenium is apopular browser-based automation tool • Launches browsers to capture the content of web applications • Simulates users’ interactions with web applications According to a Survey on Test Automation Challenges: • Almost 9 out of 10 testers are using or have used Selenium in their projects
- 3. 3 Selenium is afast-growing topic on Stack Overflow 0.6% of Stack Overflow traffic in 2018 More than 39k users 73k+ Questions 100k+ Answers
- 4. 4 In the firstpart of my thesis, I study the challenges in Selenium usage The popularity of Selenium motivates us to understand how Selenium is used in practice and to examine the issues faced by Selenium users We observe that Selenium-related questions on Stack Overflow are increasing rapidly
- 5. 5 Performance overhead inSelenium tests is a well-known issue “Are all automated testing softwares slow? Having speed issues with Selenium” “Selenium performance issue (1,000+ instances)” “Selenium starts running slower and slower” Users’ concerns on StackOverflow Literature that discusses performance issues in Selenium “Selenium WebDriver is time and resource consuming - When an automation test case is run, the web application needs to be started in the browser. This takes time and system memory because the whole web page should be loaded.” Vila et al.
- 6. 6 Load testing usingbrowser-based tools has a significant performance overhead Load testing is usually performed to determine a system’s behavior under anticipated peak load conditions Generating such workloads would be difficult using Selenium-based load tests as a normal testing machine can only generate the workload of a few users
- 7. 7 In the secondpart of my thesis, we develop an approach to improve the testing efficiency of Selenium-based load tests Due to the several advantages of Selenium-based load testing, we develop an approach to reduce the performance overhead in Selenium-based load tests Submitted this part of the thesis to the International workshop on Automated Software Testing (AST 2019)
- 8. 8 Thesis Statement Selenium-related questionson Stack Overflow offer an overview of the challenges facing practitioners who use Selenium. Since protocol-level load testing tools cannot be used to verify the complex behavior of web applications, reducing the performance overhead of Selenium tests would enable us to effectively leverage browser-based tests for load testing.
- 9. 9 Related Work Automated testgeneration (Milani Fard et al., 2014; Stocco et al., 2015; Mirshokraie et al., 2013) Tools & Frameworks (Gojare et al., 2015; Le Breton et al., 2013) Experience reports (Dowling and McGrath, 2015; Debroy et al., 2018; Arcuri, 2018. Leotta et al. 2013) JMeter for load testing Abbas et al. (2017), Kiran et al. (2015) Load testing Identifying trends over a period of time (Barua et al., 2012; Ponzanelli et al., 2014) Studies on specific Software Engineering aspects (e.g.: testing) (Venkatesh et al., 2016; Abad et al., 2016, Yang et al 2016) Stack Overflow Data Selenium
- 10. 10 In this thesis,we investigate the challenges in Selenium usage and improve the testing efficiency of Selenium-based load tests Part 1 Mining Selenium questions on Stack Overflow Part 2 Improving the testing efficiency of Selenium-based load tests
- 11. 11 Understanding Selenium-related issueson StackOverflow Part 1: Mining Selenium questions on Stack Overflow
- 12. 12 Data extraction process Stack Exchange Data Stack Exchange Data Explorer Query Selenium- relateddata from Stack- Overflow Questions Answers Users ‘%selenium%’ tag or ‘%selenium%’ in title of post Part 1: Mining Selenium questions on Stack Overflow
- 13. 13 We study thechallenges in Selenium usage from four aspects Part 1: Mining Selenium questions on Stack Overflow Co-occurring tags Programming language related tags Browser related tags Answering speed and ratio of accepted answers
- 14. 14 Programming language relatedquestions are the most popular and fastest-growing Selenium questions Part 1: Mining Selenium questions on Stack Overflow Number of new questions on Programming Languages is increasing very fast from 2010 to 2018 Number of new questions on Testing frameworks has become stable
- 15. 15 Selenium supports severalprogramming languages Part 1: Mining Selenium questions on Stack Overflow Third party support
- 16. 16 Selenium users ofJava and Python form mature communities Part 1: Mining Selenium questions on Stack Overflow 10,000+ askers and answerers 27% of the non-casual answerers answer 73% of the questions 7,000+ askers and 9,000+ answerers 30% of the non-casual answerers answer 74% of the questions Other programming languages have less than 22% of non-casual answerers and less than 4.1k answerers
- 17. 17 Selenium supports thetesting of web applications running in several browsers Part 1: Mining Selenium questions on Stack Overflow Headless browser
- 18. 18 Chrome tag isthe most dominant browser-related tag in Selenium Part 1: Mining Selenium questions on Stack Overflow PhantomJS is widely used for web-scraping purposes Firefox is used along with Selenium IDE
- 19. We measure howdifficult it is to get an accepted answer for a Selenium question Part 1: Mining Selenium questions on Stack Overflow Prior work also measures difficulty using the above two metrics [Rosen and Shihab (2016)] Answering speed Ratio of accepted answers 19
- 20. 20 Less than halfof the Selenium-related questions received an accepted answer Part 1: Mining Selenium questions on Stack Overflow 53% of questions have accepted answers on StackOverflow Most of the Selenium-related tags have about 40% accepted answers Selenium questions have less accepted answers in general Less Accepted Answers Highest % of Accepted Answers
- 21. 21 Questions related tobrowsers and Selenium components take the longest time to receive an accepted answer Part 1: Mining Selenium questions on Stack Overflow Selenium Components SLOW FAST
- 22. 22 Thesis Contribution #1 Part1: Mining Selenium questions on Stack Overflow New users of Selenium can understand how Selenium is used in practice The developers of Selenium can prioritize their efforts to address the difficult aspects in Selenium A detailed analysis of the associated challenges with the use of Selenium in practice through the mining of Selenium-related questions on Stack Overflow First work to study ‘Selenium’ related questions from Stack Overflow
- 23. 23 In this thesis,we investigate the challenges in Selenium usage and improve the testing efficiency of Selenium-based load tests Part 1 Mining Selenium questions on Stack Overflow Part 2 Improving the testing efficiency of Selenium-based load tests
- 24. 24 Browser based loadtesting is necessary to effectively test complex applications Protocol-level tools (JMeter) cannot be used to capture complex application behaviour • Tests applications that dynamically load web content (e.g. AJAX) • Executes JavaScript in web applications • Captures real usage scenarios • Requires less effort to develop such tests Part 2: Improving the testing efficiency of Selenium-based load tests
- 25. Using Selenium forload testing 25 Selenium-based load tests are resource-heavy A normal testing machine can only simulate the workload of a few users Part 2: Improving the testing efficiency of Selenium-based load tests
- 26. 26 Performance overhead inSelenium-based load tests • We observed that Selenium users are concerned about the performance issues in Selenium-based load tests Part 2: Improving the testing efficiency of Selenium-based load tests
- 27. We use awebmail client (Roundcube) as our subject application 27 ● Reply email ● Reply all ● Send email ● Forward email ● Delete email ● View emails ● Browse contacts ● Permanently delete Test Cases (Based on the MMB3 benchmark) Part 2: Improving the testing efficiency of Selenium-based load tests MMB3: A popular benchmark used to measure the performance of mail servers (Microsoft Exchange)
- 28. We follow theMMB3 benchmark to schedule the tests 28 According to the MMB3 framework, 295 tasks are scheduled to run in an 8-hour day We use a shorter yet same intensity workload by executing 19 tasks in 30 minutes Part 2: Improving the testing efficiency of Selenium-based load tests
- 29. Traditionally, a browseris launched for each user instance 29 Each user launches a browser at the start of the experiment to execute all the tasks Part 2: Improving the testing efficiency of Selenium-based load tests
- 30. Headless browsers ismore efficient than regular browsers 30 Part 2: Improving the testing efficiency of Selenium-based load tests
- 31. We study whatimpacts the resource usage in Selenium-based load testing We compare the resource usage of idle and busy browsers for a period of 10 mins 31 Part 2: Improving the testing efficiency of Selenium-based load tests
- 32. We study whatimpacts the resource usage in Selenium-based load tests (CPU/Mem spikes) The CPU & memory usage is significant even in idle browser instances 32 Part 2: Improving the testing efficiency of Selenium-based load tests
- 33. We propose anapproach to improve the testing efficiency of Selenium-based load tests 33 Traditionally, a browser is used to generate the load for one user We share browser instances among users Part 2: Improving the testing efficiency of Selenium-based load tests Sharing resources Increases the number of user instances (i.e., produced workload)
- 34. 34 We use asmaller number of browsers to simulate the workload of many users Part 2: Improving the testing efficiency of Selenium-based load tests
- 35. 35 We merge thetasks of all users to form a common list of tasks Part 2: Improving the testing efficiency of Selenium-based load tests
- 36. We use metricssuch as error ratio, delay ratio and resource usage to measure test efficiency 36 We identify the maximum number of error-free users Delay Ratio: The ratio of tasks that miss the scheduled starting time (<5%) Error Ratio: We calculate the ratio of failed tasks in each test run (0%) Resource Usage: We monitor CPU and Memory of all the processes (chrome, chromedriver and selenium scripts) used in our test setup Part 2: Improving the testing efficiency of Selenium-based load tests
- 37. Our approach increasesthe number of error-free users by 20% (headless browsers) 37 Part 2: Improving the testing efficiency of Selenium-based load tests 50 user instances 60 user instances Traditional approach Our approach 50 users 60 users 60 users (20 browsers) Median System %CPU 353 388 383 Median System %Mem 92.3 93.4 55.5 Our approach reduces the overall memory consumption
- 38. Our approach increasesthe number of error-free users by at least 22% (Xvfb) 38 Part 2: Improving the testing efficiency of Selenium-based load tests 18 user instances 22 user instances Traditional Approach Our Approach 18 users 22 users 22 users (10 browsers) Median System %CPU 429 490 486 Median System %Mem 96.4 97 84.2 Our approach reduces the overall memory consumption
- 39. 39 Thesis Contribution #2 20%improvement in terms of the number of error-free user instances Part 2: Improving the testing efficiency of Selenium-based load tests Understanding the resource usage of different browser settings 20% Regular Headless Xvfb An approach to share browsers among user instances in a Selenium-based load test that reduces the performance overhead and thereby improves the testing efficiency
- 40.
- 41.
- 42. Future Work 42 • Studyingall the Selenium-related tags in order to get more insights about how Selenium is used. For example, our study fails to cover new technologies and tools as they might not have enough questions on Stack Overflow • Using LDA to process text from the title and body of questions, answers and comments on Selenium-related posts to gain more details about the issues in Selenium • Developing an automated approach to tune parameters (number of browsers and users) dynamically during the execution of a load test • Developing an automated approach to migrate test scripts with regular browsers to test scripts with headless browsers (fixing compatibility issues), to enhance the testing efficiency of their tests
- 43. 43 Understanding the factorsthat impact answering speed using linear regression Part 1: Mining Selenium questions on Stack Overflow Community size, user reputation & platform age Answering speed • Community size: Number of users in the community • User reputation: The reputation of the user on the platform • Platform Age: Time since Stack Overflow account was created • Answering speed: Time to get an accepted answer Answering speed = β1 Community Size + β2 User Reputation + β3 Platform Age + ∈
- 44. 44 Community size andthe user reputation have a significant impact on the answering speed Part 1: Mining Selenium questions on Stack Overflow Community with a larger size and members with a higher reputation (Java/Python) is more likely to get faster answers New Selenium users on Stack Overflow are more likely to answer questions faster
- 45.
- 46. 46 Browsers are re-launchedfor every task Independent non-persistent browsers Part 2: Improving the testing efficiency of Selenium-based load tests
- 47. 47 Shared non-persistent browsers Part2: Improving the testing efficiency of Selenium-based load tests
- 48. Browser definitions 48 Part 2:Improving the testing efficiency of Selenium-based load tests 1. Depending on whether the browser persists throughout the test session …. Persistent Vs Non- persistent 2. Depending on whether a browser is shared among user instances … Independent Vs Shared 3. Depending on the type of GUI displays … Headless Vs Regular Vs Xvfb
- 49. Resource usage -Persistent headless browsers 49 Metric Traditional Approach (50 users) Our Approach (20 browsers -> 60 users) Median CPU 259 267 Median Memory 20.4 28.26 95th Percentile CPU 407.3 410.79 95th Percentile Memory 48.70 41.56
- 50. Resource usage -Non-persistent headless browser 50 Metric Traditional Approach (50 users) Our Approach (15 browsers -> 60 users) Median CPU 228 275 Median Memory 17.81 20.1 95th Percentile CPU 400 445.5 95th Percentile Memory 31.77 33.27
- 51. Resource usage -Persistent regular browsers 51 Metric Traditional Approach (18 users) Our Approach (10 browsers -> 22 users) Median CPU 279 343.35 Median Memory 20.77 27.69 95th Percentile CPU 477.4 500 95th Percentile Memory 40.18 43.22
- 52. Resource usage –Non-Persistent regular browsers 52 Metric Traditional Approach (20 users) Our Approach (6 browsers -> 25 users) Median CPU 129 161 Median Memory 18.11 15.49 95th Percentile CPU 294.75 328 95th Percentile Memory 28.14 22.79