Black box software testing

01-06-2015
Black Box Software
Testing
1
By Muhammad Soban & Muhammad Umair

 Black box testing is a software testing techniques in which
functionality of the software under test (SUT) is tested
without looking at the internal code structure,
implementation details and knowledge of internal paths
of the software.
 This type of testing is based entirely on the software
requirements and specifications.
Introduction Black Box Testing
2

 In Black Box Testing we just focus on inputs and output of
the software system.
 The above Black Box can be any software system you
want to test. For example : an operating system like
Windows, a website like Google ,a DBMS like Oracle or
even your own custom application.
Black Box Testing
3

 Black box testing tends to find different kinds of errors than
white box testing
 Missing functions
 Usability problems
 Performance problems
 Concurrency and timing errors
 Initialization and termination errors
 Unlike white box testing, black box testing tends to be
applied later in the development process
Black Box Testing
4

Equivalence
Partitioning
Boundary
Value Analysis
Error Guessing
Black Box Testing Techniques
5

 Typically the universe of all possible test cases is so large
that you cannot try them all
 You have to select a relatively small number of test cases
to actually run
 Which test cases should you choose?
 Equivalence partitioning helps answer this question
Equivalence Partitioning
6

 Partition the test cases into "equivalence classes"
 Each equivalence class contains a set of "equivalent" test
cases
 Two test cases are considered to be equivalent if we expect
the program to process them both in the same way (i.e., follow
the same path through the code)
 If you expect the program to process two test cases in the
same way, only test one of them, thus reducing the number of
test cases you have to run
7

 First-level partitioning: Valid vs. Invalid test cases
8
Valid Invalid

 Partition valid and invalid test cases into equivalence
classes
9

 If an oracle is available, the test values in each
equivalence class can be randomly generated. This is
more useful than always testing the same static values.
 Oracle: something that can tell you whether a test passed or
failed
 Test multiple values in each equivalence class. Often
you’re not sure if you have defined the equivalence
classes correctly or completely, and testing multiple values
in each class is more in-depth than relying on a single
value.
10

 Create a test case for at least one value from each
equivalence class
11

 When designing test cases, you may use different
definitions of “equivalence”, each of which will partition
the test case space differently
 Example: int Add(n1, n2, n3, …)
 Equivalence Definition 1: partition test cases by the number of inputs (1, 2,
3, etc.)
 Equivalence Definition 2: partition test cases by the number signs they
contain (positive, negative, both)
 Equivalence Definition 3: partition test cases by the magnitude of
operands (large numbers, small numbers, both)
 Etc.
12

Equivalence Partitioning Examples
13
Input Valid Equivalence Classes Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
0
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
555-5555
(555)555-5555
555-555-5555
200 <= Area code <= 999
Invalid format 5555555,
(555)(555)5555, etc.
Area code < 200 or > 999
Area code with non-numeric
characters

 When choosing values from an equivalence class to test, use the
values that are most likely to cause the program to fail
 Errors tend to occur at the boundaries of equivalence classes rather
than at the "center"
 If (200 < areaCode && areaCode < 999) { // valid area code }
 Wrong!
 If (200 <= areaCode && areaCode <= 999) { // valid area code }
 Testing area codes 200 and 999 would catch this error, but a center value like 770
would not
 In addition to testing center values, we should also test boundary
values
 Right on a boundary
 Very close to a boundary on either side
Boundary Value Analysis
14

 Create test cases to test boundaries of equivalence
classes
Boundary Value Analysis
15

Boundary Value Analysis Examples
16
Input Boundary Cases
A number N such that:
-99 <= N <= 99
-100, -99, -98
-10, -9
-1, 0, 1
9, 10
98, 99, 100
Area code: [200, 999] Area code: 199, 200, 201
Area code: 998, 999, 1000

 Numeric values are often entered as strings which are
then converted to numbers internally [int x = atoi(str);]
 This conversion requires the program to distinguish
between digits and non-digits
 A boundary case to consider: Will the program accept /
and : as digits?
Boundary Value Analysis Examples
17
Char
ASCII
/ 0 1 2 3 4 5 6 7 8 9 :
47 48 49 50 51 52 53 54 55 56 57 58

 Based on intuition, guess what kinds of inputs might cause the
program to fail
 Create some test cases based on your guesses
 Intuition will often lead you toward boundary cases, but not always
 Some special cases aren't boundary values, but are mishandled by
many programs
 Try exiting the program while it's still starting up
 Try loading a corrupted file
 Try strange but legal URLs: hTtP://Www.bYu.EDU/
Error Guessing
18

 Functional testing
 This black box testing type is related to functional requirements of
a system e.g. login system working properly. It is done by
software testers.
 Non-functional testing
 This type of black box testing is not related to testing of a specific
functionality , but non-functional requirements such as
scalability, usability, performance e.g. system under Low memory
conditions
Types of Black Box Testing
19

Types of Functional Black Box Testing
20

 Individual components are tested to ensure that they
operate correctly
 The size of a single module is small enough that we can
locate an error fairly easily
 It is usually performed by the developer.
Unit Testing(White Box Testing)
21

 Incorrect initialization
 Precision inaccuracy
 Improper loop termination
 Improperly modified loop variables
 Incorrect arithmetic precedence
 Comparison of different data types
 Incorrect logical operators or precedence
 Incorrect comparison of variables
 Incorrect symbolic representation of an expression
Test cases in unit testing should uncover
errors
22

23
 A Stub is a small program routine
that substitutes for a longer
program, possibly to be loaded
later or that is located remotely
 A Driver is nothing more than a “main
program” that accepts test-case data,
passes such data to the component (to
be tested), and prints relevant results
 Drivers and stubs represent
overhead. That is, both are
software that must be written but
that is not delivered with the final
software product.
Unit Test Procedure

 Unit-tested modules are combined into subsystems, which
are then tested.
 The goal here is to see if the modules can be integrated
properly
Integration Testing
24

Integration Testing: Incremental
Approach
25
 First combine only two components together and test them.
Remove the errors if they are there, otherwise combine another
component to it and then test again, and so on until the whole
system is developed

Integration Testing: Top-Down
26
 Modules are integrated by moving
downward through the control hierarchy
beginning with the main control module.
 Depth-first integration:
M1, M2, and M5 would be integrated first.
Next, M8 or M6 would be integrated. Then,
the central and right-hand control paths
are built.
 Breadth-first integration:
M2, M3, and M4 would be integrated first.
The next control level M5, M6, and so on
follows.

Integration Testing: Bottom-Up
27
 Begins construction and testing with
the components at the lowest level in
the program structure.
1) Low-level components are combined
into clusters (sometimes called builds)
that perform specific software sub-
functions.
2) A driver (a control program for
testing) is written to coordinate
test case input and output.
3) The cluster is tested.
4) Drivers are removed and clusters are
combined moving upward in the
program structure.

Test
Case
ID
Test Case Objective Test Case Description Expected Result
1 Check the interface link
between the Login and
Mailbox module
Enter login
credentials and click
on the Login button
To be directed to
the Mail Box
2 Check the interface link
between the Mailbox and
Delete Mails Module
From Mail box select
the an email and
click delete button
Selected email
should appear in
the Deleted/Trash
folder
Integration Test Case:
28

 System testing is the testing of a complete and fully
integrated software product
 The testing process is concerned with finding errors that
result from unexpected interactions between subsystems
and system components
 It is also concerned with validating that the system meets
its functional and non-functional requirements
System Testing
29

 Alpha testing refers to the system testing carried out by the
test team within the development organization
 Beta testing is the system testing performed by a selected
group of friendly customers
 Acceptance testing is the system testing performed by the
customer to determine whether to accept or reject the
delivery of the system
Three main kinds of system testing
30

Installation
Testing
Load Testing
Interoperability
Testing
Types of Non-Functional Black Box Testing
31

 Check that software application is
successfully installed and it is
working as expected after
installation
 Make sure that users do not have
any trouble when installing your
software
Installation Testing
32

 Required disk space is1MB, then make sure exactly 1MB
 Test disk space requirement on different file system format. Like FAT16 will
require more space than efficient NTFS or FAT32 file systems
 Try to automate the routine to test the number of files to be written on disk
 Verify registry changes on successful installation
 Forcefully break the installation process in between and check system
recovers to its original state without any issues
Installation Testing Tips With Some Broad
Test Cases
33

 If test cases are distributed from Internet, test cases should
be included for
 Bad network speed and broken connection
 Firewall and security related
 Size and approximate time taken
 Concurrent installation/downloads
Installation Testing Tips With Some Broad
Test Cases
34

 Determines a system's performance under real-life load
conditions
 Load testing identifies the bottlenecks in the system under
various workloads and checks how the system reacts
when the load is gradually increased
Load Test
35

 The maximum operating capacity of an application
 Determine whether current infrastructure is sufficient to run the
application
 Sustainability of application with respect to peak user load
 Number of concurrent users that an application can support, and
scalability to allow more users to access it
 Load testing is commonly used for the Client/Server, Web based
applications - both Intranet and Internet
Load testing usually identifies
36

 Load Runner
 Web Load
 Astra Load Test
 Rad view's Web Load
 Studio, Rational Site Load
 Silk Performer
Load Testing Tools
37

 Interoperability testing involves testing whether a given
software program or technology is compatible with others
and promotes cross-use functionality
 For example, major studies have been conducted to
determine how devices used for heart transplants interact
with various microwave technologies. In software
development, interoperability testing examines how
applications function with other applications or services
Interoperability Testing
38

 Stress testing
 Usability testing
 Security testing
 Recovery testing
 Reliability Testing
 Scalability testing
 Availability Testing
 Performance testing
 Compatibility testing
 Configuration Testing
 Documentation testing
 Maintainability Testing
 Localization testing and Internationalization testing
Other Non Functional Black Box Testing
39

(Dijkstra et al., 1972)
Testing can only show the
presence of errors, not their
absence
40

Black box software testing

In this document