Downloaded 60 times
Uploaded byKeynote Mobile Testing
Mobile Testing in the Cloud
The document discusses the importance of mobile testing in a mobile-first world, addressing challenges such as device fragmentation, evolving quality expectations, and decentralized testing approaches. It presents strategies for optimizing device testing, including prioritizing devices, automating tests, and providing access to real devices for testing. The document emphasizes the need for organizations to adapt their testing strategies to ensure high-quality mobile applications and websites.
More Related Content
Similar to Mobile Testing in the Cloud
Mobile Testing in the Cloud
- 1. Mobile Testing inthe Cloud Helping QA organizations manage the challenges of a mobile-first world Rachel Obstler May 8, 2014
- 2. Contents The MobileImperative The Challenge of Mobile Quality Solutions Optimizing Your Device Testing Strategy Automation 1
- 3. The Mobile Imperative DOWNLOADEDIN 20131 102BILLION APPS 77BILLION ANTICIPATED REVENUE GENERATED THROUGH MOBILE APPS BY 20171 #1 PRIORITY: MOBILERETAILER TECHNOLOGY PRIORITIES2 2 OF 3 BANKSPREDICT 100% OF THEIR CUSTOMERS WILL USE MOBILE SERVICES BY 20173 1. Gartner; 2. Forrester, State of Retailing Online, 2014; 3. Metaforic UK 2
- 4. The Mobile Imperative 1.Econsultancy; 2. Forbes; 3. ZDNet ONLY 16%CONSUMERS SURVEYED WILLING TO GIVE A PROBLEMATIC MOBILE APP MORE THAN ONE ATTEMPT1 USERS EXPECT A MOBILE APP TO BE RESPONSIVE IN 3 SECONDS 3 A MOBILE FRIENDLY SITE MAKES 2 OF 3 USERSMORE LIKELY TO BUY A COMPANY’S PRODUCT OR SERVICE2 3
- 5. Survey: Quality Expectations ofMobile Websites & Apps Almost half of the respondents think mobile quality expectations are the same as desktop, and 35% think they are higher. 10% 49% 36% 5% Quality expectations for mobile applications and websites are lower than desktop Quality expectations for mobile applications and websites are the same as desktop Quality expectations for mobile applications and websites are higher than desktop Not sure Now when thinking about quality expectations of mobile websites and applications, which of the following is true? (1,314 respondents) 4
- 6. Challenge: Mobile =Devices Device platforms, fragmentation, and growth • Multiple OS’s, form factors and screen resolutions • Frequent device refresh New capabilities to test • Camera, GPS, direction, orientation, voice, etc. Network considerations • Multiple carriers, variable throughput and latency, disconnected use, network switching 5
- 7. Scan & Certify Instrument Test IntegrateObtainInsight Manage Deploy Design & Develop Challenge: Mobile = Faster, More Iterative, Continuous Activity 6
- 8. Challenge: Mobile = Lateto the Game Employee Skillsets HP QC/IBM RQM users HP UFT/QTP users Automation engineers/ programmers Manual testers 7
- 9. Challenge: Mobile =Spread Across the Organization Organizations Multiple teams Geographically diverse employee base IT security requirements 8
- 10. Survey: The MobileTesting Organization is Largely Decentralized How is mobile website and application testing organized within your company? (# of respondents in each column) All Verticals (1590) High Tech/IT/ Software (661) Fin Svcs & Ins (234) Telecom (151) Media (139) Retail* (57) Centralized testing group, mobile-specific 19.7% 20.3% 20.9% 19.9% 22.3% 17.5% Centralized testing group, not mobile-specific 12.6% 12.0% 12.8% 9.9% 18.7% 10.5% Individual QA groups within business units/divisions, mobile-specific 35.8% 38.1% 36.3% 44.4% 36.0% 36.8% Individual QA groups within business units/divisions, not mobile-specific 20.8% 22.5% 20.5% 13.9% 14.4% 17.5% Testing is predominantly outsourced 3.5% 2.6% 3.4% 2.6% 3.6% 8.8% Currently not testing any mobile apps or websites 7.5% 4.5% 6.0% 9.3% 5.0% 8.8% More testing groups are distributed, rather than in one centralized group. Individual and also centralized groups are more often mobile-specific. * Retail sample size <100. 9
- 11. Survey: Challenges ofTesting Mobile Websites and Applications Regarding challenges encountered when testing mobile applications and websites, please rate the following challenges on a 1 to 10 scale, where 1 is ‘Not at all challenging’ and 10 is ‘Extremely challenging’. (1,314 respondents) 1 (Not Challenging) 2 3 4 5 6 7 8 9 10 (Extremely Challenging) Availability of mobile testing experts 4.7% 4.3% 5.8% 7.2% 17.4% 11.3% 15.4% 17.0% 8.7% 8.2% Implementing the right testing method/process for mobile 3.5% 3.2% 5.3% 7.4% 17.7% 11.5% 18.0% 18.3% 8.4% 6.5% Availability of proper testing tools 2.7% 3.2% 5.3% 5.4% 16.3% 11.6% 16.4% 16.6% 11.3% 11.3% Access to mobile devices 6.5% 6.3% 4.7% 4.5% 14.2% 10.0% 13.5% 16.7% 9.9% 13.6% Having enough time to test 4.2% 3.4% 5.6% 5.9% 16.7% 12.3% 14.9% 14.9% 10.2% 11.9% 10
- 12. Survey: Important FunctionalTesting Features Q A Within functional testing, what are the main priorities for QA? Easy access to a variety of device models was rated the most important functional testing feature across all verticals. 11
- 13. Optimize Your Device TestingStrategy Determine your priority devices Provide easy access to real devices Enable local and remote employees Provide a secure, enterprise-grade test environment 12
- 14. Use Data toDetermine Priority Devices 13 Internal apps Customer apps Android Screen Size and Density Source: Google Android OS Market Share Source: Google
- 15. Real Devices inthe cloudYour computer, connected to the internet Your key presses and mouse clicks sent to the device The device’s screen sent back to your computer Provide Easy Access to Devices Using a SaaS- based Real Device Solution 14
- 16. Different screen resolution / screensize Unreadable text, blurred images, misalignment of screen elements, and items that fall off the screen. Android Customizations On-screen and physical controls that function differently across devices Customized handling of inputs and events (e.g. Samsung Swype vs. default Android keyboard). Memory / CPU Low or insufficient memory or processing power. Impacts of other services running on the device. Mobile device characteristics that impact the quality of your application or website, but are not easily verifiable using an emulated phone or browser, include: Test on Real Devices 15
- 17. Devices Shared Devices 24X7access to hundreds of smart devices Use for compatibility testing, “untrusted device” testing Private Devices 24X7 access to your devices Enables geographically diverse team Inside corporate firewall or external Local Devices Plug your smart devices on hand directly into your local computer Make use of existing assets, great for local teams Enable Local and Remote Employees by Using a Combination of Local and Remote Devices 16
- 18. Deploy an Enterprise-gradeProduct 17 Access and Project Management Multiple levels of user permissions Control access to all assets, including devices, scripts and test results Create different groups to manage access Security Password protection features (enforce change; salted; disable save) SSL communication LDAP integration
- 19. 0 Automate! 18 Determine targettest cases to automate Support agile processes (continuous integration) Find a cross-device testing tool Make use of existing (desktop) automation tools and processes where possible Provide multiple scripting options because your team is not one size fits all
- 20. Test Automation isNot Yet Widely Used for Mobile Web What percentage of your mobile web regression tests is currently automated? (# of respondents in each column) All Verticals (1176) High Tech/IT/ Software (520) Fin Svcs & Ins (183) Telecom (106) Media (96) Retail* (45) 0% 28.8% 27.1% 26.8% 17.9% 28.1% 35.6% 1-25% 28.7% 31.3% 25.1% 33.0% 29.2% 24.4% 26-50% 17.3% 19.0% 19.7% 15.1% 14.6% 17.8% 51-75% 9.8% 11.0% 8.2% 15.1% 10.4% 2.2% >75% 4.9% 3.1% 5.5% 5.7% 4.2% 11.1% Don't know 10.5% 8.5% 14.8% 13.2% 13.5% 8.9% Majority or organization have less than 25% of mobile web regression tests automated. * Retail vertical sample size <100. 19
- 21. Mobile Apps –Same Story What percentage of your mobile native applications regression tests is currently automated? (# of respondents in each column) All Verticals (1172) High Tech/IT/ Software (519) Fin Svcs & Ins (183) Telecom (105) Media (95) Retail (45) 0% 30.4% 28.3% 27.9% 20.0% 32.6% 46.7% 1-25% 27.2% 29.7% 26.2% 34.3% 22.1% 17.8% 26-50% 15.4% 18.9% 15.8% 11.4% 8.4% 11.1% 51-75% 9.2% 8.7% 10.4% 13.3% 9.5% 6.7% >75% 4.6% 4.2% 3.8% 5.7% 6.3% 8.9% Don't know 13.1% 10.2% 15.8% 15.2% 21.1% 8.9% * Retail vertical sample size <100. 20
- 22. Determine Your Targetsfor Automation Goals Test Cases Frequency Methodology Basic acceptance, build acceptance Functional testing across all existing areas of product Compatibility across devices Very high value to automate! High value to automate Medium value to automate # of Devices Optimal Test Strategy Automated Testing Automated Testing Manual or Automated Testing Smoke Testing Regression Testing Compatibility Testing Deep testing of new areas of functionality Low value to automate Manual Testing New Feature Testing Not automated Exploratory Testing Exploration of functionality from customer viewpoint Manual Testing 21
- 23.
- 24. Customer Case Study:Speed Time to Market Each build of the native mobile retail investment app required a Build Acceptance Test with 300 hundred test cases across two devices (one Android and one iOS) Run manually, tests were taking two QA engineers two weeks (30 test cases per engineer per day) Engineering did not receive any build feedback until two weeks had passed QA team had QTP skills, and managed the process using QC Automated test cases using DeviceAnywhere and QTP 300 test cases executed across two devices (in parallel) now take 24 hours to run, vastly reducing time to market Manual testers freed up to test new or more complex features All results saved in QC BACKGROUND RESULT: Automation on real devices resulted in 10X Reduction in BAT time! CUSTOMER Director of QA at a major financial institution 23
- 25. Device Fragmentation Lessup front scripting cost Reduce ongoing script maintenance Object-Level Scripting Important Cross-Device Testing 24
- 26. Customer Case Study:Improve Product Quality The current mobile test process for the consumer mobile banking app involved outsourcing testing to a manual test house. The available budget afforded them enough person/hours to complete their 900 regression test cases on only one device Results and issues were managed using unwieldy word documents As usage picked up, issues in the field highlighted the need to ensure quality across a variety of mobile platforms and devices BACKGROUND RESULTS CUSTOMER VP of Engineering at a major bank Now the same 900 test cases are run across 20 devices, improving test coverage and quality All 20 devices tested in half the time it took previously to test one device, improving time to market Test results are automatically available online with screenshots (no more word docs, improved process) 25
- 27. Make Use ofExisting Tools Where Possible ALM Tools (HP ALM, IBM Rational) Automation Tools such as HP QTP Open Source Tools like Selenium, Robotium Other tools 26 Survey your organization: What desktop test tools are currently being used?
- 28. Survey: Important FunctionalTesting Features by Vertical Integration with ALM (Application Lifecycle Management) tools were also highly rated for financial services companies. Having an ‘integration with open source tools’ (such as Selenium, Robotium, etc.) was rated very high for retail companies as they traditionally have been more e-commerce focused and would often be developing automation using open source web-based tools. Were there any other vertical-specific challenges that also rated high? Q 27
- 29. Beginning and AdvancedScripting Options Enable All Employees 28 UI-Based Scripting For complex logic and large test suite design and execution Integration with Leading Test Tools Allow employees with existing skillsets to easily extend to mobile Java API Enables your QA engineers with programming skills to write test in Java Script Recording Point and click recorder enables anyone to immediately create reusable test scripts Automation Engineers UFT/QTP users Programmers New Scripters
- 30. Challenges 29 Mobile =Devices Mobile = Faster, More Iterative, Continuous Activity Mobile = Late to the Game Mobile = Spread Across the Organization
- 31. Solutions Challenge Solution(s) /Strategies to Address Mobile = Devices Use data to determine priority devices Provide access to real devices in the cloud Automate using a cross-device testing tool Mobile = Faster, More Iterative, Continuous Activity Determine high-value test cases to automate Continuous integration Automate using a cross–device testing tool Mobile = Late to the Game Make use of existing test tools where possible Provide a variety of scripting options for differing skill sets Mobile = Spread Across the Organization Provide access to local and remote real devices Enterprise-grade product to manage multiple teams 30
- 32. Make every digitalinteraction count MyKeynote® TESTANALYZE Mobile App Testing Mobile Web Testing Load Testing Data Visualization Competitive Intelligence Expert Analysis Mobile App Monitoring Website Monitoring Mobile Web Monitoring MONITOR 31
- 33. Keynote: Most ExperiencedCloud-based Mobile, App, Web & Network Performance Company on the Planet 1995: Founded 500 employees 4,000+ customers 7,000+ measurement devices in ~300 locations – most scaled on-demand infrastructure in world 700,000,000 performance measurements daily 730 Carrier Networks monitored in ~300 locations in 200 countries 1,500+ devices in 15 global locations Forbes ‘One of the Best 100 Companies in America’ 32 Sao Paolo Moscow Beijing Paris New York
- 34. Integrating the World’sLargest Global Testing & Monitoring Network and the World’s Largest Real Device Cloud 33
- 35. Enterprise Solutions ona Global Scale 3,000 customers in 130 countries Business operations in EMEA, APAC, Americas Global infrastructure Web performance testing & monitoring agents in 100 locations Mobile testing and monitoring agents in 250 locations 7,000 devices on network 270B measurements per year 34
- 36.
Editor's Notes
- #2 Today, organizations are rapidly deploying mobile versions of their customer-facing and internal applications. With the prevalence of more agile-based approaches and the challenge of an ever-increasing diversity of devices and OS versions, testers are being asked to accomplish more testing in less time. Rachel Obstler shares how leading enterprises are improving the efficiency of their mobile testing using automation, and how they identify the right processes and tools for the job. Sharing some fascinating statistics from their recent mobile quality survey of more than 69,000 mobile app developers and QA organizations in the top US enterprises, Rachel dives into the challenges identified in the survey and shares how to improve your testing process through optimizing your device testing strategy, and automating your mobile tests.
- #3 So before I start, let’s do a show of hands for everyone in the audience that is responsible for testing mobile applications or mobile websites. It’s about x%. In a year, almost all of you if not all will be able to raise your hands.
- #4 What is the mobile imperative?We’ve reached the tipping point for mobile. End users are demanding access via mobile.Across industries, mobile is no longer an optional touchpoint for consumer engagement. It is required.Banking metric: this says that mobile will become the primary interaction medium. Everyone will interact via mobile. Some will interact via desktop. This is a complete flip.Not limited to consumer apps– huge growth in Enterprise apps as well:54% increase Q4 over Q3-2013. (Good Technologies)
- #5 And it’s not good enough to simply “be there”. The imperative facing companies is to capitalize on the mobile opportunity quickly AND with high levels of quality and performance. The desktop Web experience has raised expectations with both consumers and employees. You simply must deliver an awesome experience.So the mobile imperative is that you must not only provide access via mobile, but must do so with quality.
- #6 These industry stats are reflected in a recent survey we conducted in partnership with Software Quality Engineering, largely of QA and development professionals. You’ll see more of the key data points from the survey in this presentation.Almost half of the respondents said that the quality expectations for mobile websites and apps are the same as desktop, and 35% actually said they were higher! With stats that say 80% of apps on phones are deleted within one year, every moment of interaction counts.
- #7 But providing a quality mobile app or website is easier said than done. Different challenges to quality are inherent with mobile than what we’ve seen in the desktop world.Dozens of popular devices launched each yearIncreasing fragmentation as device vendors attempt to differentiate themselvesDevices refresh quickly – there are new ones to support all the time. Camera, GPS, direction, orientation, voice, etc.Multiple carriers, variable throughput and latency, disconnected use, network switching say from wifi to cellular and back.Smaller screen real estate, so the user experience is much more critical. Things can’t fall off the screen, or not be selectable. Most of the time, you only get one change to get it right.
- #8 Mobile development lifecycle is different:Faster cyclesMore iterativeContinuous Activity – How fast can you go around the circle
- #9 Another challenge: Mobile is late to the game – many test organizations were set up for the desktop, and are organized around desktop tools and processes. So it’s important to be able to make use of existing skillsets as well as bring new tools and capabilities specific to mobile to the mix. And lastly, having tools that can be used across the organization, regardless of the mobile products being tested and the group doing the testing is vital.
- #10 Last challengeThis one is not unique to mobile. But it’s still a challenge to be addressed; multiple teams, global organizations
- #11 Challenges are reflected in the recent survey we did.Centralized vs. not centralizedAnd within each, when are groups mobile-specific.
- #12 And this question that asked generally about challenges of testing mobile websites and applications really reflects the challenges of devices, skillsets, organizations, and speed requirements. Availability of mobile testing expertsHaving the right tools, process, often a reflection on the complexity added by devices. All related to mobile being late to the game.Time – impact of increasingly agile processes.
- #13 We also see those challenges reflected in the features that are viewed as the most important in terms of functional testing. Namely, easy access to device models. Also automation, which helps speed testing time. A little farther down, but you see support for agile processes also.
- #14 So what can you do about these challenges?First talk about device strategy, and then automation. For device strategy, you can (read bullets)
- #15 Optimize your test resources by prioritizing your devices.Most organizations have this data on usage. Maybe the marketing team holds it; maybe ecommerce, or product management. Ask them for it! They will appreciate that you are aligning your testing plans with actual device usage patterns.If you can’t get it, look on the internet. Google posts lots of data on OS market share, and other items.
- #16 These SaaS solutions are great for a number of reasons:Let another company worry about keeping up with the latest models. Better process – you need to take a screen shot, you can since you are using a computerBetter ergonomically – you can sit hunched over a device for hours on end. Small screen, bad ergonomics.
- #17 Last slide assumes that testing should be done on real devices. But why?A browser emulator cannot accurately render the elements on the page. Often the emulator is not the right browser version, or can only emulate the browser; not exact. Most new devices use a touchscreen for input, but others still have keyboard-only input or keyboards in addition to touchscreens. Applications have to be designed to support multiple keyboard configurations. Potential issues can include missing input keys, pop-up touchscreen keyboards that obscure areas of the application, or touchscreen control bars that are misaligned. Upshot is: if you don’t test on a real device, you 1) don’t know what it looks like, 2) don’t know if it works and 3) don’t know how it performs. But you know it compiles, you know if feature function.Apps with heavy graphics or fast action may not perform within acceptable standards.Now this doesn’t suggest that testing should ONLY be done on real devices – in fact emulators are very valuable testing tools. But typically for earlier in the test cycle.
- #18 Use a test system that enables this. You have local devices; you should be able to use them in the same system as your remote devices.Think about which devices you will use a lot. You may want to dedicate them to a team. Are you planning on automation? We’ll talk about that in the next section more, but having your own dedicated devices are important when you are planning to automate; you can easily be running tests on them for hours on end.
- #19 make sure your test system, SaaS devices, dedicated devices, etc. has what your enterprise needs if you have:Multiple teams – you need separate work spaces for devices, assets, etc.Security requirements. We work with many banks, healthcare organizations, government, etc. Where QA really plays a role of governance, it’s going to be important that their test data and process stays secure, and in fact, their IT department will dictate that it must.So to summarize the device strategy:You can’t test on everything so prioritize using dataFind a tool that provides easy access to test devices so you don’t need to keep up with them and buy themMake sure the tool is flexible enough to support local, remote, private and public devicesAnd make sure it can support your organizations needs around security and project management
- #20 Now on to automation. The rapid speed of releases and the number of platforms that you must test across means that you must look at automating as much as your repetitive tests as possible. The QA groups is being asked to do more (higher quality) with less.I’m going to talk about 5 automation strategies that everyone should take into account when implementing an automation program.
- #21 First sharing some more data from the survey – if you have not yet started or do not have much automation, you are not alone.
- #23 When looking at the testing process, testing of your mobile products is not a one-size-fits-all; depending on the stage you are in, you will be covering more or fewer test cases, at different frequencies, across different numbers of devices. These needs will dictate whether it makes sense to test manually or automate. Higher frequency – more valueThis is the same in the desktop world too.
- #24 Important to have a tool that supports continuous integration
- #26 It’s also important to have a tool that enables you to write a script that works across devices. With newer technologies available today, this is entirely possible.
- #28 Illustration. Important to ask what tools your company is using. If you have a large organization, and like many orgs are making a centralized decision on tooling for many different groups, you could have a variety of tools being used across your organization. We’ve run into many orgs that use for instance HP, but it’s not used uniformly across groups in almost every case.To the extent possible, if you can extend existing tools or processes, you’ll reduce the learning curve. But there is always a tradeoff – tools have to support mobile testing.
- #29 In our survey, we saw different verticals with different tools they typically used.
- #30 For enterprises, QA groups are not one size fits all. Some groups have developers (can program test scripts). Some are used to using QTP. Some are new, and have no preexisting skills.Support all or if not as many of these use cases as possible. This will ensure your process gets adopted, and that you;ve minimized training costs as well as time to get your teams transitioning to mobile up to speed. Choose a flexible tools that support this.
- #31 Reminder – these are the challenges
- #32 And here are the strategies and solutions that I talked about, and which challenges they address
- #33 Because at the end, one error, something not working can make the difference between a successful digital interaction with your customer; say when they are about to hit the Buy button, and them leaving never to return. Keynote’s product suite is designed so that you can ensure every one of your digital interactions counts.
- #34 100+ Locations Web Performance Testing & Monitoring Agents250+ Locations Mobile Monitoring Agents & Network Operator Probes
- #35 Only Keynote has the scale of infrastructure to be able to deliver solutions like Mobile App Monitoring in an effective and affordable way. You can take advantage of our on-demand real device cloud for instant insight, or choose a custom solution with private instances to meet your specific needs. With a continuously evolving network, you can trust Keynote to be your partner over the long haul.