nt Session 
 
Presented by: 
Allan Wagner 
 
 
Brought to you by: 
 
 
340 Corporate Way, Suite   Orange Park, FL 32073 
888‐2
T10 
Concurre
4/8/2014   
2:00 PM 
 
 
 
 
“Continuous Testing Through  
Service Virtualization” 
 
 
IBM 
 
 
 
 
 
 
300,
68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 
Allan Wagner
IBM
 
IBM-certified IT Specialist Allan (Al) Wagner is currently a technical manager and
evangelist with IBM, driving thought leadership, strategic initiatives, and tangible
solutions with a specific focus on quality management, test automation, and service
virtualization. In his fifteen years of practical IT field experience, Al has assisted,
supported, mentored, and enabled teams with their IT application infrastructure,
development, implementation, and operations challenges. A frequent conference
speaker on software quality principles and techniques, Al also has authored or
coauthored the IBM Service Virtualization for Dummies book and numerous papers on
quality management and test automation.
1
Continuous Testing
through
Service Virtualization
@Al_Wagner
awagner@ca.ibm.com
Technical Marketing Manager
IBM
Session T10
Test Techniques
Tuesday, April 8th, 2:00pm
1
Business stakeholder expectations
Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seemed to have enough
time.
• Software quality suffered.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.
2
2
Business stakeholder expectations
Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seems to have enough
time.
• Software quality suffered.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.
What are organizations “actually“
doing to deliver higher quality
software to market faster?
3
Adding more Testers is not a sustainable model
Labor costs continue to increase
Testers
Testers
Testers
Development
Team
Testers
3
4
Adopting technology only increases testing complexity
Test environment setup is harder and more expensive than ever before!
external
users
internal
users
Application
Layer
Mainframe
ERP
Systems
Custom
Applications
Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
5
hybrid
Network Latency
Security/Privacy
Availability
Cost
Data Integrity
Adopting technology only increases testing complexity
Test environment setup is harder and more expensive than ever before!
external
users
internal
users
private
Business Partner
Services
public
External 3rd Party
Services
Application
Layer
Mainframe
ERP
Systems
Custom
Applications
Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
Now, there are
many new
things testers
also need to
consider.
4
6
Faster and more frequent builds are not enough
The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver
code
Nightly build(s)
Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” nightly builds are piling up
7
Faster and more frequent builds are not enough
The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver
code
Nightly build(s)
Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” days to install & configure
“N” nightly builds are piling up
Quality
GateBVT
Multiple
Test Environments
FVT
Perf Test
Security Test
SIT
UAT
I just spent 2 weeks
setting this up &
the build is no good?
When agility stops at
testing, the entire
organization hits the wall
5
8
Bottom line – current approaches are not helping!
Today’s testing challenges still remain
Development VelocityTesting Labor Test Lab Complexity
Outsourcing labor is no longer
a sustainable model as global
wages are increasing
Product and application
complexity are driving
test lab cost increases
Test teams can no longer
keep up with agile
development
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
Web/
Internet
Routing
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
ServicesArchives
Business
Partners
Messaging
Services
Traditional
Testing
Other costs
Testing costs
9
What Testers really need!
Validating today’s complex applications
Production-like test environments which are:
 realistic & complete
 available on demand
 inexpensive to setup
Heterogeneous
Environments
Public
Cloud
Private
Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services
Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Other costs
Testing costs
Automated tests which are:
 able to run unattended validating application
integration, functionality, security & performance
 capable of validating business flows across
multiple channels
 durable, maintainable & resilient to change
Orchestration from build to production
deployment which is:
 reliable
 repeatable
 traceable
Better
Faster
Cheaper
6
10
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Monitoring
Collaborative
Development
Continuous
Release and
Deployment
Continuous
Customer
Feedback
and
Optimization
Continuous
Business
Planning
Monitor
and Optimize
Develop
and Test
Release
and Deploy
Plan
and Measure
DevOps –
Continuous
innovation,
feedback and
improvements
Enables clients to seize market opportunities
and reduce time to customer feedback
Increasing adoption of DevOps
Extending Lean and Agile practices across the entire software value chain
Continuous
Testing
11
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
7
12
Continuous Deployment and Release
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Dev UT
At least 1/3 of is testing!
13
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Test
scripts
Test
data
Test
configurations
Test
reports
Version
Control
Orchestration & Automation
Dev UT
At least 1/3 of is testing!
8
14
Reality check
Which answer would you pick?
What percentage of your time are you spending setting up, maintaining, and
re-cycling test environments?
1. 0% to 20%
2. 21% to 30%
3. 31% to 40%
4. 41% to 50%
5. More than 50%
11%
18%
14%
43%
14%
Responses from GRUC Webcast – October 22nd, 2013
15
Solving the Test Environment dilemma
Making production-like test environments a reality
 Traditional test environments are both underutilized and insufficient
– Environments are expensive: hardware, software, and labor
– Tester and developer time wasted waiting for appropriate environment
 Agile test environments require four enabling capabilities
Dynamic Infrastructure
Test Data
Management
Deployment
Automation
Test Environments
DevOps Foundation
Service
Virtualization
9
16
Reality check
Which answer would you pick?
How does your organization deal with testing bottlenecks due to unavailable or
missing software and services?
1. Manually write simulation stubs to emulate missing functionality.
2. Use commercial service virtualization solution.
3. Defer testing until later when the code is available to test.
4. De-scope testing of that functionality.
5. None of the above.
9%
9%
56%
6%
19%
Responses from GRUC Webcast – October 22nd, 2013
17
What is Service Virtualization?
Making the unavailable available for testing
 Virtual components simulate the behavior
of a service or application during testing
– System dependencies are a key challenge in
setting up test environments
– Unavailable/inaccessible services slow down
the testing process
– Costly 3rd party access fees can significantly
impact testing availability
 Virtual components run on commodity
hardware, private cloud, public cloud
 Each developer and tester can easily have
their own test environment
 Developers and testers can continue to use
current testing procedures and tools
 Streamline creation of production-like
environments and negative testing
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components
10
18
Virtual components simulate
the “real world” behavior
Category Detail
Simple Hard-coded response returned for given input
Non-deterministic One-of-n hard-coded responses
Data driven
Input and/or output data specified in external
data source (Excel, file, database)
Data model, stateful
Input and/or output data kept in data model with
complex relationships. Supports CRUD and
other stateful behavior
Behavioral
Extends model-driven to provide pre-packaged
functionality, e.g. shopping basket, real-time
data feed, trading exchange, order matching
What are virtual components?
Simulating application functionality, behavior, & performance
19
Test Environment
Volume Constraints
Volume Constraints
2hr. access window
Maintenance Constraints
Database / JDBC
Server/ Web Services
Mainframe / MQ/CICS
Smart Device / SD-IO
MQ/CICS
Web Services
Smart Devices
JDBC
Virtual Environment
Capture
& Model
System Under Test
Listen / Capture
Store
Test Case
Repository
Creating virtual components
Listen and capture
11
20
Build virtual components
from the design specs
HelloService
Turn specifications into virtual components
Creating virtual components
Author from scratch using design specifications
21
Continuous Testing reduces risk
Actual Service/App
Virtual Service/App
 Service Virtualization is an enabler for
continuous testing
 Services, applications, systems are
introduced into the continuous integration
cycle in a prioritized, controlled fashion.
Controlled integration helps isolate defects
for faster resolution.
Service virtualization enables
continuous integration testing
Using virtual components
Enables testers to continuously test what they want, when they want, how they want
12
22
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
23
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
13
24
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
 Begin end to end testing
earlier and continuously.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
25
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
 Begin end to end testing
earlier and continuously.
 Introduce the real
implementations as they
become available.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
14
26
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
new
27
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
App Layer
new
15
28
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
 Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new
29
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
 Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
 Test applications end to end with
confidence but at a lower cost and
without delay.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new
16
30
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
Testers
31
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications. Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
JSON RESTWSDL
Virtual
Service
Virtual
Service
Virtual
Service
Testers
17
32
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications.
 Record the functionality and
behavior of the application using
virtual services.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Testers
rec
101101100101101100
33
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications.
 Record the functionality and
behavior of the application using
virtual services.
 Execute automated tests at any
layer in the application using virtual
services to simulate functionality
under development.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
WSDL
Virtual
Service
Virtual
Service
Virtual
Service
Testers
101101100 101101100
18
34
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
new
35
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newVirtual
Service
19
36
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
 Continue testing without delay.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newVirtual
Service
37
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
 Continue testing without delay.
 Replace the virtual service with
the updated source code without
having to redeploy your entire
application.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newfixed
20
38
The ROI of Service Virtualization
The results are real!
3 Year Benefits:
• Increased project delivery capacity 100% (40 projects to 80 projects)
• Avoided hiring three additional FTE's, saving almost $600k.
• Reduced incidents found in production from 2.5 to .3 per project, saving $3 million
• Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k
• Reduced consulting and third party testing fees with better test automation, saving $408k
• Eliminated the need for additional hardware and labor to set up test environments, saving $700k
• Generated new revenue for the business – estimated at tens of millions!
A major European bank implemented service virtualization and automated integration
testing as part of an Agile transformation to enable continuous testing. As a result of
IBM capabilities, their system integration test (SIT) process was reduced from 3
weeks down to four hours!
The Total Economic Impact™ of
IBM Rational Service Virtualization
and Test Automation Solutions, a
July 2013 commissioned study
conducted by Forrester Consulting
on behalf of IBM
39
Orchestration
Continuous Testing
InfoSphere Optim
Test Data Management
Rational Test
Virtualization
Server
Rational Test
Workbench
• Capture service
behavior and model
virtual services
• Test services and
applications long
before their user
interfaces becomes
available and do
integration testing
• Performance test,
regression test,
functional test &
integration test
• Virtualize
heterogeneous
hardware, software,
and services to
provide 24x7 testing
capabilities
• Reduce the cost of
traditional testing
environments
• Build Virtual Services
to simulate over 70+
protocols
Rational
Performance
Test Server
• Reuse Rational Test
Workbench scripts to
drive performance
testing
• Combine virtual
services and the real
implementations to
performance test
earlier
• Probe for
identification of
system bottlenecks
• Create referentially
intact, “right-sized”
test databases
• Automate test result
comparisons to
identify hidden errors
• Protect confidential
data used in test,
training &
development
• Refresh test
environments faster
shortening iterative
testing cycles and
accelerating time to
market
IBM’s continuous testing solution
Rational Quality
Manager
• Track quality using a
comprehensive and
customizable test
plan
• Link test cases with
requirements,
defects, development
work items and builds
• Author and execute
automated or manual
test scripts
• Provide up-to-date
project quality status
using customizable
dashboards
Test Data
Management
Service
Virtualization
Test
Automation
with integration to
Collaborative
Development
& Test
Completing the Continuous Testing story
Service Virtualization is the enabler but it is only part of a Continuous Testing solution
21
40
Orchestrate and automate the deployment of applications, middleware
configuration, and database changes into development, test and
production environments.
Databases Internal
Messages
Third-party
Services
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
Test Data
Management
Deploy
Provision
Test
Combining continuous testing withOrchestration from build to production
Reliable, repeatable, & traceable deployment automation
Deployment
Automationintegrated with
Test Environments
Dynamic Infrastructure
 Test using real world
network conditions
Service
Virtualization
Test
Automation
 Deploy and refresh
right size secure data
 Deploy what is ready,
virtualize the rest
 Continuously test in
production-like env.
Network
Virtualization
41
Continuing on your Service Virtualization journey
Increasing awareness and accelerating learning
Get your copy today!
Service Virtualization
For Dummies
ibm.co/ServiceVirtualizationForDummies
Visit
Service Virtualization
on ibm.com
ibm.co/servicevirtualization
22
42
Continuous testing as our planet gets smarter
If you think testing is hard now, just wait
intelligent interconnectedinstrumented
We are already on the road to the Internet of Things
Sooner than later, someone is going to ask you to test it!
43
23
44

Continuous Testing through Service Virtualization

  • 1.
  • 2.
    Allan Wagner IBM   IBM-certified ITSpecialist Allan (Al) Wagner is currently a technical manager and evangelist with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific focus on quality management, test automation, and service virtualization. In his fifteen years of practical IT field experience, Al has assisted, supported, mentored, and enabled teams with their IT application infrastructure, development, implementation, and operations challenges. A frequent conference speaker on software quality principles and techniques, Al also has authored or coauthored the IBM Service Virtualization for Dummies book and numerous papers on quality management and test automation.
  • 3.
    1 Continuous Testing through Service Virtualization @Al_Wagner awagner@ca.ibm.com TechnicalMarketing Manager IBM Session T10 Test Techniques Tuesday, April 8th, 2:00pm 1 Business stakeholder expectations Better – Cheaper - Faster In the past • Organizations looked to find a balance between speed, cost & risk. • Development teams were not as agile as they are today; testing never seemed to have enough time. • Software quality suffered. Today • Organizations are asking teams to release higher quality software sooner. • Development teams are becoming more and more agile; testing still never seems to have enough time. • Software quality continues to suffer.
  • 4.
    2 2 Business stakeholder expectations Better– Cheaper - Faster In the past • Organizations looked to find a balance between speed, cost & risk. • Development teams were not as agile as they are today; testing never seems to have enough time. • Software quality suffered. Today • Organizations are asking teams to release higher quality software sooner. • Development teams are becoming more and more agile; testing still never seems to have enough time. • Software quality continues to suffer. What are organizations “actually“ doing to deliver higher quality software to market faster? 3 Adding more Testers is not a sustainable model Labor costs continue to increase Testers Testers Testers Development Team Testers
  • 5.
    3 4 Adopting technology onlyincreases testing complexity Test environment setup is harder and more expensive than ever before! external users internal users Application Layer Mainframe ERP Systems Custom Applications Databases Enterprise Service Bus Functionality Usability Performance/Scalability Reliability Consumability Before, testers designed tests to measure quality based on a number of attributes. 5 hybrid Network Latency Security/Privacy Availability Cost Data Integrity Adopting technology only increases testing complexity Test environment setup is harder and more expensive than ever before! external users internal users private Business Partner Services public External 3rd Party Services Application Layer Mainframe ERP Systems Custom Applications Databases Enterprise Service Bus Functionality Usability Performance/Scalability Reliability Consumability Before, testers designed tests to measure quality based on a number of attributes. Now, there are many new things testers also need to consider.
  • 6.
    4 6 Faster and morefrequent builds are not enough The transformation to Agile isn’t benefitting Testers Tester Setup test environment Developer Write & deliver code Nightly build(s) Compile, unit test, publish Developer Developer Developer Developer Developer “N” nightly builds are piling up 7 Faster and more frequent builds are not enough The transformation to Agile isn’t benefitting Testers Tester Setup test environment Developer Write & deliver code Nightly build(s) Compile, unit test, publish Developer Developer Developer Developer Developer “N” days to install & configure “N” nightly builds are piling up Quality GateBVT Multiple Test Environments FVT Perf Test Security Test SIT UAT I just spent 2 weeks setting this up & the build is no good? When agility stops at testing, the entire organization hits the wall
  • 7.
    5 8 Bottom line –current approaches are not helping! Today’s testing challenges still remain Development VelocityTesting Labor Test Lab Complexity Outsourcing labor is no longer a sustainable model as global wages are increasing Product and application complexity are driving test lab cost increases Test teams can no longer keep up with agile development Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration Web/ Internet Routing Service Third-party Services Portals Content Providers EJB Shared ServicesArchives Business Partners Messaging Services Traditional Testing Other costs Testing costs 9 What Testers really need! Validating today’s complex applications Production-like test environments which are:  realistic & complete  available on demand  inexpensive to setup Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration App Under TestRouting Service Third-party Services Portals Content Providers EJB Shared Services Archives Business Partners Messaging Services Other costs Testing costs Automated tests which are:  able to run unattended validating application integration, functionality, security & performance  capable of validating business flows across multiple channels  durable, maintainable & resilient to change Orchestration from build to production deployment which is:  reliable  repeatable  traceable Better Faster Cheaper
  • 8.
    6 10 Accelerate software delivery – forfaster time to value Balance speed, cost, quality and risk – for increased capacity to innovate Reduce time to customer feedback – for improved customer experience Continuous Monitoring Collaborative Development Continuous Release and Deployment Continuous Customer Feedback and Optimization Continuous Business Planning Monitor and Optimize Develop and Test Release and Deploy Plan and Measure DevOps – Continuous innovation, feedback and improvements Enables clients to seize market opportunities and reduce time to customer feedback Increasing adoption of DevOps Extending Lean and Agile practices across the entire software value chain Continuous Testing 11 All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers
  • 9.
    7 12 Continuous Deployment andRelease All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers Collaborative Development The goal: create the build package Continuous Monitoring The goal: production deployment Continuous Testing simultaneously across multiple test environments Everything between dev & prod is a test stage Quality Gate Security FVT Performance SIT UAT Quality Gate BVT Dev UT At least 1/3 of is testing! 13 All the world’s a “test” stage, and all Testers merely players; They have their exits and their entrances; [play on Shakespeare] The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers Collaborative Development The goal: create the build package Continuous Monitoring The goal: production deployment Continuous Testing simultaneously across multiple test environments Everything between dev & prod is a test stage Quality Gate Security FVT Performance SIT UAT Quality Gate BVT Test scripts Test data Test configurations Test reports Version Control Orchestration & Automation Dev UT At least 1/3 of is testing!
  • 10.
    8 14 Reality check Which answerwould you pick? What percentage of your time are you spending setting up, maintaining, and re-cycling test environments? 1. 0% to 20% 2. 21% to 30% 3. 31% to 40% 4. 41% to 50% 5. More than 50% 11% 18% 14% 43% 14% Responses from GRUC Webcast – October 22nd, 2013 15 Solving the Test Environment dilemma Making production-like test environments a reality  Traditional test environments are both underutilized and insufficient – Environments are expensive: hardware, software, and labor – Tester and developer time wasted waiting for appropriate environment  Agile test environments require four enabling capabilities Dynamic Infrastructure Test Data Management Deployment Automation Test Environments DevOps Foundation Service Virtualization
  • 11.
    9 16 Reality check Which answerwould you pick? How does your organization deal with testing bottlenecks due to unavailable or missing software and services? 1. Manually write simulation stubs to emulate missing functionality. 2. Use commercial service virtualization solution. 3. Defer testing until later when the code is available to test. 4. De-scope testing of that functionality. 5. None of the above. 9% 9% 56% 6% 19% Responses from GRUC Webcast – October 22nd, 2013 17 What is Service Virtualization? Making the unavailable available for testing  Virtual components simulate the behavior of a service or application during testing – System dependencies are a key challenge in setting up test environments – Unavailable/inaccessible services slow down the testing process – Costly 3rd party access fees can significantly impact testing availability  Virtual components run on commodity hardware, private cloud, public cloud  Each developer and tester can easily have their own test environment  Developers and testers can continue to use current testing procedures and tools  Streamline creation of production-like environments and negative testing Heterogeneous Environments Public Cloud Private Cloud Data Warehouse Mainframe Enterprise Service Bus Directory Identity File systems Collaboration App Under TestRouting Service Third-party Services Portals Content Providers EJB Shared Services Archives Business Partners Messaging Services Databases Mainframe applications App Under Test Third-party Services Packaged apps, messaging services, etc. Virtual Components
  • 12.
    10 18 Virtual components simulate the“real world” behavior Category Detail Simple Hard-coded response returned for given input Non-deterministic One-of-n hard-coded responses Data driven Input and/or output data specified in external data source (Excel, file, database) Data model, stateful Input and/or output data kept in data model with complex relationships. Supports CRUD and other stateful behavior Behavioral Extends model-driven to provide pre-packaged functionality, e.g. shopping basket, real-time data feed, trading exchange, order matching What are virtual components? Simulating application functionality, behavior, & performance 19 Test Environment Volume Constraints Volume Constraints 2hr. access window Maintenance Constraints Database / JDBC Server/ Web Services Mainframe / MQ/CICS Smart Device / SD-IO MQ/CICS Web Services Smart Devices JDBC Virtual Environment Capture & Model System Under Test Listen / Capture Store Test Case Repository Creating virtual components Listen and capture
  • 13.
    11 20 Build virtual components fromthe design specs HelloService Turn specifications into virtual components Creating virtual components Author from scratch using design specifications 21 Continuous Testing reduces risk Actual Service/App Virtual Service/App  Service Virtualization is an enabler for continuous testing  Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion. Controlled integration helps isolate defects for faster resolution. Service virtualization enables continuous integration testing Using virtual components Enables testers to continuously test what they want, when they want, how they want
  • 14.
    12 22 Avoiding the bigbang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application 23 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services  Simulate the behavior and performance of the dependent components. Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service
  • 15.
    13 24 Avoiding the bigbang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services  Simulate the behavior and performance of the dependent components.  Begin end to end testing earlier and continuously. Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service Virtual Service 25 Avoiding the big bang Control integration Scenario: Testers may often wait or defer testing certain areas of functionality until all of the components are available and deployed as a complete application. This approach results in major defects being discovered late in the process and could put the release date at risk. Service virtualization puts control in the hands of the tester. Testers Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services  Simulate the behavior and performance of the dependent components.  Begin end to end testing earlier and continuously.  Introduce the real implementations as they become available. Mainframe Database ERP System Custom Application Virtual Service
  • 16.
    14 26 Stand up testenvironments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer new 27 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays.  Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application Virtual Service App Layer new
  • 17.
    15 28 Stand up testenvironments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays.  Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees.  Quickly stand up test labs using virtual services to simulate dependencies which have not changed. Testers Business Partner Services 3rd Party Services Virtual Service Virtual Service App Layer Virtual Service new 29 Stand up test environments faster Deploy what has changed, virtualize the rest Scenario: The team is making changes to an application component adding integration with an existing 3rd party service to implement new required functionality. To validate this minor change, the time to stand up a complete test lab combined with the 3rd party service access fees adds substantial project cost – cost which is unnecessary. Service virtualization lowers the cost of testing and eliminates delays.  Create and deploy a virtual component to simulate the 3rd party service functionality and performance without the fees.  Quickly stand up test labs using virtual services to simulate dependencies which have not changed.  Test applications end to end with confidence but at a lower cost and without delay. Testers Business Partner Services 3rd Party Services Virtual Service Virtual Service App Layer Virtual Service new
  • 18.
    16 30 Author automated scriptsearlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application Testers 31 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes.  Create virtual components from development specifications. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application JSON RESTWSDL Virtual Service Virtual Service Virtual Service Testers
  • 19.
    17 32 Author automated scriptsearlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes.  Create virtual components from development specifications.  Record the functionality and behavior of the application using virtual services. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application Virtual Service Virtual Service Virtual Service Testers rec 101101100101101100 33 Author automated scripts earlier Increasing Tester productivity – author automated scripts earlier, execute tests sooner Scenario: While teams may want to automate testing as soon as possible, creation of automated test scripts is often delayed until everything is available in a test lab – typically late in the development cycle or perhaps not until after the actual release of the application software. Service virtualization makes the unavailable available for testing purposes.  Create virtual components from development specifications.  Record the functionality and behavior of the application using virtual services.  Execute automated tests at any layer in the application using virtual services to simulate functionality under development. Application Layer Enterprise Service Bus Business Partner Services 3rd Party Services Mainframe Database ERP System Custom Application WSDL Virtual Service Virtual Service Virtual Service Testers 101101100 101101100
  • 20.
    18 34 Blocking defects -they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer new 35 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity.  Deploy a virtual service to simulate the required functionality and behavior replacing the defective component. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newVirtual Service
  • 21.
    19 36 Blocking defects -they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity.  Deploy a virtual service to simulate the required functionality and behavior replacing the defective component.  Continue testing without delay. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newVirtual Service 37 Blocking defects - they won’t slow me down Reducing testing down time Scenario: As defects are discovered, some may actually block the testing effort from proceeding. Testers may be forced to wait for the fix before being able to continue and development priorities may result in the fix being delayed. This creates a testing bottleneck! Service virtualization eliminates bottlenecks and increases productivity.  Deploy a virtual service to simulate the required functionality and behavior replacing the defective component.  Continue testing without delay.  Replace the virtual service with the updated source code without having to redeploy your entire application. Testers Business Partner Services 3rd Party Services Enterprise Service Bus Mainframe Database ERP System Custom Application App Layer newfixed
  • 22.
    20 38 The ROI ofService Virtualization The results are real! 3 Year Benefits: • Increased project delivery capacity 100% (40 projects to 80 projects) • Avoided hiring three additional FTE's, saving almost $600k. • Reduced incidents found in production from 2.5 to .3 per project, saving $3 million • Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k • Reduced consulting and third party testing fees with better test automation, saving $408k • Eliminated the need for additional hardware and labor to set up test environments, saving $700k • Generated new revenue for the business – estimated at tens of millions! A major European bank implemented service virtualization and automated integration testing as part of an Agile transformation to enable continuous testing. As a result of IBM capabilities, their system integration test (SIT) process was reduced from 3 weeks down to four hours! The Total Economic Impact™ of IBM Rational Service Virtualization and Test Automation Solutions, a July 2013 commissioned study conducted by Forrester Consulting on behalf of IBM 39 Orchestration Continuous Testing InfoSphere Optim Test Data Management Rational Test Virtualization Server Rational Test Workbench • Capture service behavior and model virtual services • Test services and applications long before their user interfaces becomes available and do integration testing • Performance test, regression test, functional test & integration test • Virtualize heterogeneous hardware, software, and services to provide 24x7 testing capabilities • Reduce the cost of traditional testing environments • Build Virtual Services to simulate over 70+ protocols Rational Performance Test Server • Reuse Rational Test Workbench scripts to drive performance testing • Combine virtual services and the real implementations to performance test earlier • Probe for identification of system bottlenecks • Create referentially intact, “right-sized” test databases • Automate test result comparisons to identify hidden errors • Protect confidential data used in test, training & development • Refresh test environments faster shortening iterative testing cycles and accelerating time to market IBM’s continuous testing solution Rational Quality Manager • Track quality using a comprehensive and customizable test plan • Link test cases with requirements, defects, development work items and builds • Author and execute automated or manual test scripts • Provide up-to-date project quality status using customizable dashboards Test Data Management Service Virtualization Test Automation with integration to Collaborative Development & Test Completing the Continuous Testing story Service Virtualization is the enabler but it is only part of a Continuous Testing solution
  • 23.
    21 40 Orchestrate and automatethe deployment of applications, middleware configuration, and database changes into development, test and production environments. Databases Internal Messages Third-party Services virtual components Simultaneously test across multiple test stages Dev QA Test Data Management Deploy Provision Test Combining continuous testing withOrchestration from build to production Reliable, repeatable, & traceable deployment automation Deployment Automationintegrated with Test Environments Dynamic Infrastructure  Test using real world network conditions Service Virtualization Test Automation  Deploy and refresh right size secure data  Deploy what is ready, virtualize the rest  Continuously test in production-like env. Network Virtualization 41 Continuing on your Service Virtualization journey Increasing awareness and accelerating learning Get your copy today! Service Virtualization For Dummies ibm.co/ServiceVirtualizationForDummies Visit Service Virtualization on ibm.com ibm.co/servicevirtualization
  • 24.
    22 42 Continuous testing asour planet gets smarter If you think testing is hard now, just wait intelligent interconnectedinstrumented We are already on the road to the Internet of Things Sooner than later, someone is going to ask you to test it! 43
  • 25.