Download to read offline
![Chapter 1 ■ Background
10
This chart was created in response to the Fukushima Daiichi nuclear disaster of 2011, and sought to clear up
misinformation and misunderstanding of comparisons being made around the disaster. It did this by demonstrating the
differences in scale for the amount of radiation from sources such as other people or a banana, up to what a fatal dose of
radiation ultimately would be—how all that compared to spending just ten minutes near the Chernobyl meltdown.
Over the last quarter of a century, Edward Tufte, author and professor emeritus at Yale University, has been
working to raise the bar of information graphics. He published groundbreaking books detailing the history of data
visualization, tracing its roots even further back than Playfair, to the beginnings of cartography. Among his principles
is the idea to maximize the amount of information included in each graphic—both by increasing the amount of
variables or data points in a chart and by eliminating the use of what he has coined chartjunk. Chartjunk, according to
Tufte, is anything included in a graph that is not information, including ornamentation or thick, gaudy arrows.
Tufte also invented the sparkline, a time series chart with all axes removed and only the trendline remaining to
show historic variations of a data point without concern for exact context. Sparklines are intended to be small enough
to place in line with a body of text, similar in size to the surrounding characters, and to show the recent or historic
trend of whatever the context of the text is.
Why Data Visualization?
In William Playfair’s introduction to the Commercial and Political Atlas, he rationalizes that just as algebra is the
abbreviated shorthand for arithmetic, so are charts a way to “abbreviate and facilitate the modes of conveying
information from one person to another.” Almost 300 years later, this principle remains the same.
Data visualizations are a universal way to present complex and varied amounts of information, as we saw in our
opening example with the quarterly earnings report. They are also powerful ways to tell a story with data.
Imagine you have your Apache logs in front of you, with thousands of lines all resembling the following:
127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET / HTTP/1.1 200 468 - Mozilla/5.0 (X11; U;
Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty)
127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET /favicon.ico HTTP/1.1 200 766 - Mozilla/5.0
(X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty)
Among other things, we see IP address, date, requested resource, and client user agent. Now imagine this
repeated thousands of times—so many times that your eyes kind of glaze over because each line so closely resembles
the ones around it that it’s hard to discern where each line ends, let alone what cumulative trends exist within.
By using some analysis and visualization tools such as R, or even a commercial product such as Splunk, we can
artfully pull out all kinds of meaningful and interesting stories out of this log, from how often certain HTTP errors occur
and for which resources, to what our most widely used URLs are, to what the geographic distribution of our user base is.
This is just our Apache access log. Imagine casting a wider net, pulling in release information, bugs and
production incidents. What insights we could gather about what we do: from how our velocity impacts our defect
density to how our bugs are distributed across our feature sets. And what better way to communicate those findings
and tell those stories than through a universally digestible medium, like data visualizations?
The point of this book is to explore how we as developers can leverage this practice and medium as part of
continual improvement—both to identify and quantify our successes and opportunities for improvements, and more
effectively communicate our learning and our progress.
Tools
There are a number of excellent tools, environments, and libraries that we can use both to analyze and visualize our
data. The next two sections describe them.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-17-2048.jpg)
![Chapter 1 ■ Background
15
Creating data visualizations involves four core steps:
1. Identify a problem.
2. Gather the data.
3. Analyze the data.
4. Visualize the data.
Let’s walk through each step in the process and re-create one of the previous charts to demonstrate the process.
Identify a Problem
The very first step is to identify a problem we want to solve. This can be almost anything—from something as
profound and wide-reaching as figuring out why your bug backlog doesn’t seem to go down and stay down, to seeing
what feature releases over a given period in time caused the most production incidents, and why.
For our example, let’s re-create Figure 1-5 and try to quantify the interest in data visualization over time as
represented by the number of New York Times articles on the subject.
Gather Data
We have an idea of what we want to investigate, so let’s dig in. If you are trying to solve a problem or tell a story around
your own product, you would of course start with your own data—maybe your Apache logs, maybe your bug backlog,
maybe exports from your project tracking software.
Note
■
■ If you are focusing on gathering metrics around your product and you don’t already have data handy, you need to
invest in instrumentation.There are many ways to do this, usually by putting logging in your code.At the very least, you want to
log error states and monitor those, but you may want to expand the scope of what you track to include for
debugging purposes
while still respecting both your user’s privacy and your company’s privacy policy. In my book, Pro JavaScript
Performance:
Monitoring and Visualization, I explore ways to track and visualize web and runtime performance.
One important aspect of data gathering is deciding which format your data should be in (if you're lucky) or discovering
which format your data is available in. We’ll next be looking at some of the common data formats in use today.
JSON is an acronym that stands for JavaScript Object Notation. As you probably know, it is essentially a way to
send data as serialized JavaScript objects. We format JSON as follows:
[object]{
[attribute]: [value],
[method] : function(){},
[array]: [item, item]
}
Another way to transfer data is in XML format. XML has an expected syntax, in which elements can have attributes,
which have values, values are always in quotes, and every element must have a closing element. XML looks like this:
parent attribute=value
child attribute=valuenode data/child
/parent
Generally we can expect APIs to return XML or JSON to us, and our preference is usually JSON because as we can
see it is a much more lightweight option just in sheer amount of characters used.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-22-2048.jpg)
![Chapter 1 ■ Background
17
byline: By AUTHOR,
date: 20121011,
title: TITLE,
url: http://www.nytimes.com/foo.html
}, {
body: BODY COPY,
byline: By AUTHOR,
date: 20121021,
title: TITLE,
url: http://www.nytimes.com/bar.html
}
],
tokens: [
JavaScript
],
total: 2
}
Looking at the high-level JSON structure, we see an attribute named offset, an array named results, an array
named tokens, and another attribute named total. The offset variable is for pagination (what page full of results
we are starting with). The total variable is just what it sounds like: the number of results that are returned for our
query. It’s the results array that we really care about; it is an array of objects, each of which corresponds to an article.
The article objects have attributes named body, byline, date, title, and url.
We now have data that we can begin to look at. That takes us to our next step in the process, analyzing our data.
DATA SCRUBBING
There is often a hidden step here, one that anyone who’s dealt with data knows about: scrubbing the data. Often
the data is either not formatted exactly as we need it or, in even worse cases, it is dirty or incomplete.
In the best-case scenario in which your data just needs to be reformatted or even concatenated, go ahead and do
that, but be sure to not lose the integrity of the data.
Dirty data has fields out of order, fields with obviously bad information in them—think strings in ZIP codes—or
gaps in the data. If your data is dirty, you have several choices:
You could drop the rows in question, but that can harm the integrity of the data—a good example
•
is if you are creating a histogram removing rows could change the distribution and change what
your results will be.
The better alternative is to reach out to whoever administers the source of your data and try and
•
get a better version if it exists.
Whatever the case, if data is dirty or it just needs to be reformatted to be able to be imported into R, expect to
have to scrub your data at some point before you begin your analysis.
Analyze Data
Having data is great, but what does it mean? We determine it through analysis.
Analysis is the most crucial piece of creating data visualizations. It’s only through analysis that we can understand
our data, and it is only through understanding it that we can craft our story to share with others.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-24-2048.jpg)
![Chapter 1 ■ Background
19
if(length(author) 1){
author - unknown
}
datestamp -paste(month, /, day, /, year, sep=)
datestamp - as.Date(datestamp,%m/%d/%Y)
newrow - data.frame(datestamp, author, title, stringsAsFactors=FALSE, check.rows=FALSE)
df - rbind(df, newrow)
}
rownames(df) - df$datestamp
Note that our example assumes that the data set returned has unique date values. If you get errors with this, you
may need to scrub your returned data set to purge any duplicate rows.
Once our data frame is populated, we can start to do some analysis on the data. Let’s start out by pulling just the
year from every entry, and quickly making a stem and leaf plot to see the shape of the data.
Note
■
■ John Tukey created the stem and leaf plot in his seminal work, Exploratory Data Analysis. Stem and leaf plots
are quick, high-level ways to see the shape of data, much like a histogram. In the stem and leaf plot, we construct the
“stem” column on the left and the “leaf” column on the right. The stem consists of the most significant unique elements
in a result set. The leaf consists of the remainder of the values associated with each stem. In our stem and leaf plot below,
the years are our stem and R shows zeroes for each row associated with a given year. Something else to note is that
often alternating sequential rows are combined into a single row, in the interest of having a more concise visualization.
First, we will create a new variable to hold the year information:
yearlist - as.POSIXlt(df$datestamp)$year+1900
If we inspect this variable, we see that it looks something like this:
yearlist
[1] 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2011 2011 2011 2011 2011 2011
2011 2011 2011 2011 2011 2011 2011 2011 2011 2011
[30] 2011 2011 2011 2011 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2009 2009 2009 2009 2009
2009 2009 2008 2008 2008 2007 2007 2007 2007 2006
[59] 2006 2006 2006 2005 2005 2005 2005 2005 2005 2004 2003 2003 2003 2002 2002 2002 2002 2001 2001
2000 2000 2000 2000 2000 2000 1999 1999 1999 1999
[88] 1999 1999 1998 1998 1998 1997 1997 1996 1996 1995 1995 1995 1993 1993 1993 1993 1992 1991 1991
1991 1990 1990 1990 1990 1989 1989 1989 1988 1988
[117] 1988 1986 1985 1985 1985 1984 1982 1982 1981
That’s great, that’s exactly what we want: a year to represent every article returned. Next let’s create the stem and
leaf plot:
stem(yearlist)
1980 | 0
1982 | 00
1984 | 0000
1986 | 0
1988 | 000000](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-26-2048.jpg)
![Chapter 1 ■ Background
20
1990 | 0000000
1992 | 00000
1994 | 000
1996 | 0000
1998 | 000000000
2000 | 00000000
2002 | 0000000
2004 | 0000000
2006 | 00000000
2008 | 0000000000
2010 | 000000000000000000000000000000
2012 | 0000000000000
Very interesting. We see a gradual build with some dips in the mid-1990s, another gradual build with another dip
in the mid-2000s and a strong explosion since 2010 (the stem and leaf plot groups years together in twos).
Looking at that, my mind starts to envision a story building about a subject growing in popularity. But what
about the authors of these articles? Maybe they are the result of one or two very interested authors that have quite
a bit to say on the subject.
Let’s explore that idea and take a look at the author data that we parsed out. Let’s look at just the unique authors
from our data frame:
length(unique(df$author))
[1] 81
We see that there are 81 unique authors or combination of authors for these articles! Just out of curiosity, let’s take
a look at the breakdown by author for each article. Let’s quickly create a bar chart to see the overall shape of the data
(the bar chart is shown in Figure 1-17):
plot(table(df$author), axes=FALSE)
Figure 1-17. Bar chart of number of articles by author to quickly visualize](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-27-2048.jpg)
![Chapter 2 ■ R Language Primer
29
The Command Line
The R console is where the magic happens! It is a command-line environment where we can run R expressions. The best
way to get up to speed in R is to script in the console, a piece at a time, generally to try out what you’re trying to do, and
tweak it until you get the results that you want. When you finally have a working example, take the code that does what
you want and save it as an R script file.
R script files are just files that contain pure R and can be run in the console using the source command:
source(someRfile.R)
Looking at the preceding code snippet, we assume that the R script lives in the current work directory. The way
we can see what the current work directory is to use the getwd() function:
getwd()
[1] /Users/tomjbarker
We can also set the working directory by using the setwd() function. Note that changes made to the working
directory are not persisted across R sessions unless the session is saved.
setwd(/Users/tomjbarker/Downloads)
getwd()
[1] /Users/tomjbarker/Downloads
Command History
The R console stores commands that you enter and you can cycle through previous commands by pressing the up
arrow. Hit the escape button to return to the command prompt. We can see the history in a separate window pane
by clicking the Show/Hide Command History button at the top of the console. The Show/Hide Command History
button is the rectangle icon with alternating stripes of yellow and green. See Figure 2-5 for the R console with the
command history shown.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-35-2048.jpg)
![Chapter 2 ■ R Language Primer
31
Packages
Speaking of packages, what are they, exactly? Packages are collections of functions, data sets, or objects that can
be imported into the current session or workspace to extend what we can do in R. Anyone can make a package
and distribute it.
To install a package, we simply type this:
install.packages([package name])
For example, if we want to install the ggplot2 package—which is a widely used and very handy charting
package—we simply type this into the console:
install.packages(ggplot2)
We are immediately prompted to choose the mirror location that we want to use, usually the one closest to our
current location. From there, the install begins. We can see the results in Figure 2-7.
Figure 2-6. Help search results window](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-37-2048.jpg)
![Chapter 2 ■ R Language Primer
33
codetools Code Analysis Tools for R
colorspace Color Space Manipulation
compiler The R Compiler Package
datasets The R Datasets Package
dichromat Color schemes for dichromats
digest Create cryptographic hash digests of R objects
foreign Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, dBase,
...
ggplot2 An implementation of the Grammar of Graphics
gpairs gpairs: The Generalized Pairs Plot
graphics The R Graphics Package
grDevices The R Graphics Devices and Support for Colours and Fonts
grid The Grid Graphics Package
gtable Arrange grobs in tables.
KernSmooth Functions for kernel smoothing for Wand Jones (1995)
labeling Axis Labeling
lattice Lattice Graphics
mapdata Extra Map Databases
mapproj Map Projections
maps Draw Geographical Maps
Importing Data
So now our environment is downloaded and installed, and we know how to install any packages that we may need.
Now we can begin using R.
The first thing we’ll normally want to do is import your data. There are several ways to import data, but the most
common way is to use the read() function, which has several flavors:
read.table([file to read])
read.csv([file to read])
To see this in action, let’s first create a text file named temptext.txt that is formatted like so:
134,432,435,313,11
403,200,500,404,33
77,321,90,2002,395
We can read this into a variable that we will name temptxt:
temptxt - read.table(temptext.txt)
Notice that as we are assigning value to this variable, we are not using an equal sign as the assignment operator.
We are instead using an arrow -. That is R’s assignment operator, although it does also support the equal sign if you
are so inclined. But the standard is the arrow, and all examples that we will show in this book will use the arrow.
If we print out the temptxt variable, we see that it is structured as follows:
temptxt
V1
1 134,432,435,313,11
2 403,200,500,404,33
3 77,321,90,2002,395](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-39-2048.jpg)
![Chapter 2 ■ R Language Primer
34
We see that our variable is a table-like structure called a data frame, and R has assigned a column name (V1) and
row IDs to our data structure. More on column names soon.
The read() function has a number of parameters that you can use to refine how the data is imported and
formatted once it is imported.
Using Headers
The header parameter tells R to treat the first line in the external file as containing header information. The first line
then becomes the column names of the data frame.
For example, suppose we have a log file structured like this:
url, day, date, loadtime, bytes, httprequests, loadtime_repeatview
http://apress.com, Sun, 01 Jul 2012 14:01:28 +0000,7042,956680,73,3341
http://apress.com, Sun, 01 Jul 2012 14:01:31 +0000,6932,892902,76,3428
http://apress.com, Sun, 01 Jul 2012 14:01:33 +0000,4157,594908,38,1614
We can load it into a variable named wpo like so:
wpo - read.table(wpo.txt, header=TRUE)
wpo
url day date loadtime bytes httprequests loadtime_repeatview
1 http://apress.com,Sun,1 Jul 2012 14:01:28 +0000,7042,955550,73,3191
2 http://apress.com,Sun,1 Jul 2012 14:01:31 +0000,6932,892442,76,3728
3 http://apress.com,Sun,1 Jul 2012 14:01:33 +0000,4157,614908,38,1514
When we call the colnames() function to see what the column names are for wpo, we see the following:
colnames(wpo)
[1] url day date loadtime
[5] bytes httprequests loadtime_repeatview
Specifying a String Delimiter
The sep attribute tells the read() function what to use as the string delimiter for parsing the columns in the external
data file. In all the examples we’ve looked at so far, commas are our delimiters, but we could use instead pipes | or any
other character that we want.
Say, for example, that our previous temptxt example used pipes; we would just update the code to be as follows:
134|432|435|313|11
403|200|500|404|33
77|321|90|2002|395
temptxt - read.table(temptext.txt, sep=|)
temptxt
V1 V2 V3 V4 V5
1 134 432 435 313 11
2 403 200 500 404 33
3 77 321 90 2002 395
Oh, notice that? We actually got distinct column names this time (V1, V2, V3, V4, V5). Before, we didn’t specify a
delimiter, so R assumed that each row was one big blob of text and lumped it into a single column (V1).](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-40-2048.jpg)
![Chapter 2 ■ R Language Primer
36
Data types in R are called modes, and can be the following:
numeric
•
character
•
logical
•
complex
•
raw
•
list
•
We can use the mode() function to check the mode of a variable.
Character and numeric modes correspond to string and number (both integer and float) data types. Logical
modes are Boolean values.
n - 122132
mode(n)
[1] numeric
c - test text
mode(c)
[1] character
l - TRUE
mode(l)
[1] logical
We can perform string concatenation using the paste() function. We can use the substr() function to pull
characters out of strings. Let’s look at some examples in code.
Usually, I keep a list of directories that I either read data from or write charts to. Then when I want to reference
a new data file that exists in the data directory, I will just append the new file name to the data directory:
dataDirectory - /Users/tomjbarker/org/data/
buglist - paste(dataDirectory, bugs.txt, sep=)
buglist
[1] /Users/tomjbarker/org/data/bugs.txt
The paste() function takes N amount of strings and concatenates them together. It accepts an argument named
sep that allows us to specify a string that we can use to be a delimiter between joined strings. We don’t want anything
separating our joined strings that we pass in an empty string.
If we want to pull characters from a string, we use the substr() function. The substr() function takes a string to
parse, a starting location, and a stopping location. It returns all the character inclusively from the starting location up
to the ending location. (Remember that in R, lists are not 0-based like most other languages, but instead have
a starting index of 1.)
substr(test, 1,2)
[1] te
In the preceding example, we pass in the string “test” and tell the substr() function to return the first and
second characters.
Complex mode is for complex numbers. The raw mode is to store raw byte data.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-42-2048.jpg)
![Chapter 2 ■ R Language Primer
37
List data types or modes can be one of three classes: vectors, matrices, or data frames. If we call mode() for vectors
or matrices, they return the mode of the data that they contain; class() returns the class. If we call mode() on a data
frame, it returns the type list:
v - c(1:10)
mode(v)
[1] numeric
m - matrix(c(1:10), byrow=TRUE)
mode(m)
[1] numeric
class(m)
[1] matrix
d - data.frame(c(1:10))
mode(d)
[1] list
class(d)
[1] data.frame
Note that we just typed 1:10 rather than the whole sequence of numbers between 1 and 10:
v - c(1:10)
Vectors are single-dimensional arrays that can hold only values of a single mode at a time. It’s when we get to
data frames and matrices that R really starts to get interesting. The next two sections cover those classes.
Data Frames
We saw at the beginning of this chapter that the read() function takes in external data and saves it as a data frame.
Data frames are like arrays in most other loosely typed languages: they are containers that hold different types of data,
referenced by index. The main thing to realize, though, is that data frames see the data that they contain as rows, columns,
and combinations of the two.
For example, think of a data frame as formatted as follows:
col col col col col
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
If we try to reference the first index in the preceding data frame as we traditionally would with an array, say
dataframe[1], R would instead return the first column of data, not the first item. So data frames are referenced by their
column and row. So dataframe[1] returns the first column and dataframe[,2] returns the first row.
Let’s demonstrate this in code.
First let’s create some vectors using the combine function, c(). Remember that vectors are collections of data all
of the same type. The combine function takes a series of values and combines them into vectors.
col1 - c(1,2,3,4,5,6,7,8)
col2 - c(1,2,3,4,5,6,7,8)
col3 - c(1,2,3,4,5,6,7,8)
col4 - c(1,2,3,4,5,6,7,8)](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-43-2048.jpg)
![Chapter 2 ■ R Language Primer
38
Then let’s combine these vectors into a data frame:
df - data.frame(col1,col2,col3,col4)
Now let’s print the data frame to see the contents and the structure of it:
df
col1 col2 col3 col4
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
5 5 5 5 5
6 6 6 6 6
7 7 7 7 7
8 8 8 8 8
Notice that it took each vector and made each one a column. Also notice that each row has an ID; by default,
it is a number, but we can override that.
If we reference the first index, we see that the data frame returns the first column:
df[1]
col1
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
If we put a comma in front of that 1, we reference the first row:
df[,1]
[1] 1 2 3 4 5 6 7 8
So accessing contents of a data frame is done by specifying [column, row].
Matrices work much the same way.
Matrices
Matrices are just like data frames in that they contain rows and columns and can be referenced by either. The core
difference between the two is that data frames can hold different data types but matrices can hold only one type of data.
This presents a philosophical difference. Usually you use data frames to hold data read in externally, like from a
flat file or a database because those are generally of mixed type. You normally store data in matrices that you want to
apply functions to (more on applying functions to lists in a little bit).](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-44-2048.jpg)
![Chapter 2 ■ R Language Primer
39
To create a matrix, we must use the matrix() function, pass in a vector, and tell the function how to distribute
the vector:
The
• nrow parameter specifies how many rows the matrix should have
The
• ncol parameter specifies the number of columns.
The
• byrow parameter tells R that the contents of the vector should be distributed by iterating
across rows if TRUE or by columns if FALSE.
content - c(1,2,3,4,5,6,7,8,9,10)
m1 - matrix(content, nrow=2, ncol=5, byrow=TRUE)
m1
[,1] [,2] [,3] [,4] [,5]
[1,] 1 2 3 4 5
[2,] 6 7 8 9 10
Notice that in the previous example that the m1 matrix is filled in horizontally, row by row. In the following
example, the m1 matrix is filled in vertically by column:
content - c(1,2,3,4,5,6,7,8,9,10)
m1 - matrix(content, nrow=2, ncol=5, byrow=FALSE)
m1
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
Remember that instead of manually typing out all the numbers in the previous content vector, if the numbers are
a sequence we can just type this:
content - (1:10)
We reference the content in matrices with the square bracket, specifying the row and column, respectively.
m1[1,4]
[1] 7
We can convert a data frame to a matrix if the data frame contains only a single type of data. To do this we use the
as.matrix() function. Often times we will do this when passing a data frame to a plotting function to draw a chart.
barplot(as.matrix(df))
Below we create a data frame called df. We populate the data frame with ten consecutive numbers. We then use
as.matrix() to convert df into a matrix and save the result into a new variable called m:
df - data.frame(1:10)
df
X1.10
1 1
2 2
3 3](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-45-2048.jpg)
![Chapter 2 ■ R Language Primer
40
4 4
5 5
6 6
7 7
8 8
9 9
10 10
class(df)
[1] data.frame
m - as.matrix(df)
class(m)
[1] matrix
Keep in mind that because they are all the same data type, matrices require less overhead and are intrinsically
more efficient than data frames. If we compare the size of our matrix m and our data frame df, we see that with just ten
items there is a size difference.
object.size(m)
312 bytes
object.size(df)
440 bytes
With that said, if we increase the scale of this, the increase in efficiency does not equally scale. Compare the following:
big_df - data.frame(1:40000000)
big_m - matrix(1:40000000)
object.size(big_m)
160000112 bytes
object.size(big_df)
160000400 bytes
We can see that the first example with the small data set showed that the matrix was 30 percent smaller in size
than the data frame, but at the larger scale in the second example the matrix was only .00018 percent smaller than
the data frame.
Adding Lists
When combining or adding to data frames or matrices, you generally add either by the row or the column using
rbind() or cbind().
To demonstrate this, let’s add a new row to our data frame df. We’ll pass df into rbind() along with the new row
to add to df. The new row contains just one element, the number 11:
df - rbind(df, 11)
df
X1.10
1 1
2 2
3 3
4 4
5 5
6 6](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-46-2048.jpg)
![Chapter 2 ■ R Language Primer
41
7 7
8 8
9 9
10 10
11 11
Now let’s add a new column to our matrix m. To do this, we simply pass m into cbind() as the first parameter;
the second parameter is a new matrix that will be appended to the new column.
m - rbind(m, 11)
m - cbind(m, matrix(c(50:60), byrow=FALSE))
m
X1.10
[1,] 1 50
[2,] 2 51
[3,] 3 52
[4,] 4 53
[5,] 5 54
[6,] 6 55
[7,] 7 56
[8,] 8 57
[9,] 9 58
[10,] 10 59
[11,] 11 60
What about vectors, you may ask? Well, let’s look at adding to our content vector. We simply use the combine
function to combine the current vector with a new vector:
content - c(1,2,3,4,5,6,7,8,9,10)
content - c(content, c(11:20))
content
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Looping Through Lists
As developers who generally work in procedural languages, or at least came up the ranks using procedural languages
(though in recent years functional programming paradigms have become much more mainstream), we’re most
likely used to looping through our arrays when we want to process the data within them. This is in contrast to purely
functional languages where we would instead apply a function to our lists, like the map() function. R supports both
paradigms. Let’s first look at how to loop through our lists.
The most useful loop that R supports is the for in loop. The basic structure of a for in loop can be seen here:.
for(i in 1:5){print(i)}
[1] 1
[1] 2
[1] 3
[1] 4
[1] 5](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-47-2048.jpg)
![Chapter 2 ■ R Language Primer
42
The variable i increments in value each step through the iteration. We can use the for in loop to step through
lists. We can specify a particular column to iterate through, like the following, in which we loop through the X1.10
column of the data frame df.
for(n in df$X1.10){ print(n)}
[1] 1
[1] 2
[1] 3
[1] 4
[1] 5
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
Note that we are accessing the columns of data frames via the dollar sign operator. The general pattern is
[data frame]$[column name].
Applying Functions to Lists
But the way that R really wants to be used is to apply functions to the contents of lists (see Figure 2-8).
function
element
element
element
element
Figure 2-8. Apply a function to list elements
We do this in R with the apply() function.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-48-2048.jpg)
![Chapter 2 ■ R Language Primer
43
The apply() function takes several parameters:
First is our list.
•
Next a number vector to indicate how we apply the function through the list (
• 1 is for rows, 2 is
for columns, and c[1,2] indicates both rows and columns).
Finally is the function to apply to the list:
•
apply([list], [how to apply function], [function to apply])
Let’s look at an example. Let’s make a new matrix that we’ll call m. The matrix m will have ten columns and four rows:
m - matrix(c(1:40), byrow=FALSE, ncol=10)
m
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 5 9 13 17 21 25 29 33 37
[2,] 2 6 10 14 18 22 26 30 34 38
[3,] 3 7 11 15 19 23 27 31 35 39
[4,] 4 8 12 16 20 24 28 32 36 40
Now say we wanted to increment every number in the m matrix. We could simply use apply() as follows:
apply(m, 2, function(x) x - x + 1)
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 2 6 10 14 18 22 26 30 34 38
[2,] 3 7 11 15 19 23 27 31 35 39
[3,] 4 8 12 16 20 24 28 32 36 40
[4,] 5 9 13 17 21 25 29 33 37 41
Do you see what we did there? We passed in m, we specified that we wanted to apply the function across the
columns, and finally we passed in an anonymous function. The function accepts a parameter that we called x.
The parameter x is a reference to the current matrix element. From there, we just increment the value of x by 1.
OK, say we wanted to do something slightly more interesting, such as zeroing out all the even numbers in the
matrix. We could do the following:
apply(m,c(1,2),function(x){if((x %% 2) == 0) x - 0 else x - x})
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 5 9 13 17 21 25 29 33 37
[2,] 0 0 0 0 0 0 0 0 0 0
[3,] 3 7 11 15 19 23 27 31 35 39
[4,] 0 0 0 0 0 0 0 0 0 0
For the sake of clarity let’s break out that function that we are applying. We simply check to see whether the
current element is even by checking to see whether it has a remainder when divided by two. If it is, we set it to zero;
if it isn’t, we set it to itself:
function(x){
if((x %% 2) == 0)
x - 0
else
x - x
}](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-49-2048.jpg)
{
[body of function]
}
Something interesting that R allows is the ... argument (sometimes called the dots argument). This allows us to
pass in a variable number of parameters into a function. Within the function, we can convert the ... argument into a list
and iterate over the list to retrieve the values within:
offset - function (...){
for(i in list(...)){
print(i)
}
}
offset(23,11)
[1] 23
[1] 11
We can even store values of different data types (modes) in the ... argument:
offset(test value, 12, 100, 19ANM)
[1] test value
[1] 12
[1] 100
[1] 19ANM
R uses lexical scoping. This means that when we call a function and try to reference variables that are not defined
inside the local scope of the function, the R interpreter looks for those variables in the workspace or scope in which the
function was created. If the R interpreter cannot find those variables in that scope, it looks in the parent of that scope.
If we create a function A within function B, the creation scope of function A is function B. For example, see the
following code snippet:
x - 10
wrapper - function(y){
x - 99
c- function(y){
print(x + y)
}
return(c)
}
t - wrapper()
t(1)
[1] 100
x
[1] 10](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-50-2048.jpg)
![Chapter 3 ■ A Deeper Dive into R
48
There also exists the concept of polymorphism, in which we can change functionality via the inheritance chain.
Specifically, we would inherit a function from a base class but override it, keep the signature (the function name, the
type and amount of parameters it accepts, and the type of data that it returns) the same, but change what the function
does. Compare overriding a function to the contrasting concept of overloading a function, in which the function
would have the same name but a different signature and functionality.
S3 Classes
S3, so called because it was first implemented in version 3 of the S language, uses a concept called generic functions.
Everything in R is an object, and each object has a string property called class that signifies what the object is. There
is no validation around it, and we can overwrite the class property ad hoc. That’s the main problem with S3—the
lack of validation. If you ever had an esoteric error message returned when trying to use a function, you probably
experienced the repercussions of this lack of validation firsthand. The error message was probably generated not from
R detecting that an incorrect type had been passed in, but from the function trying to execute with what was passed in
and failing at some step along the way.
See the following code, in which we create a matrix and change its class to be a vector:
m - matrix(c(1:10), nrow=2)
m
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
class(m) - vector
m
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
attr(,class)
[1] vector
BaseUser
login()
createPlaylist
extends extends
implements
User
login()
createPlaylist()
SuperUser
login()
createPlaylist()
editPermissions()
IUser
login()
createPlaylist()
Figure 3-2. An IUser interface implemented by a superclass BaseUser that the subclasses User and SuperUser extend](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-53-2048.jpg)
![Chapter 3 ■ A Deeper Dive into R
49
Generic functions are objects that check the class property of objects passed into them and exhibit different
behavior based on that attribute. It’s a nice way to implement polymorphism. We can see the methods that a generic
function uses by passing the generic function to the methods() function. The following code shows the methods of the
plot() generic function:
methods(plot)
[1] plot.acf* plot.data.frame* plot.decomposed.ts* plot.default plot.dendrogram*
[6] plot.density plot.ecdf plot.factor* plot.formula* plot.function
[11] plot.hclust* plot.histogram* plot.HoltWinters* plot.isoreg* plot.lm
[16] plot.medpolish* plot.mlm plot.ppr* plot.prcomp* plot.princomp*
[21] plot.profile.nls* plot.spec plot.stepfun plot.stl* plot.table*
[26] plot.ts plot.tskernel* plot.TukeyHSD
Non-visible functions are asterisked
Notice that within the generic plot() function is a myriad of methods to handle all the different types of data
that could be passed to it, such as plot.data.frame for when we pass a data frame to plot(); or if we want to plot
a TukeyHSD object plot(), plot.TukeyHSD is ready for us.
Note
■
■ Type ?TukeyHSD for more information on this object.
Now that you know how S3 object-oriented concepts work in R, let’s see how to create our own custom S3 objects
and generic functions.
An S3 class is a list of properties and functions with an attribute named class. The class attribute tells generic
functions how to treat objects that implement a particular class. Let’s create an example using the UserClass idea
from Figure 3-2:
tom - list(userid = tbarker, password = password123, playlist=c(12,332,45))
class(tom) - user
We can inspect our new object by using the attributes() function, which tells us the properties that the object
has as well as its class:
attributes(tom)
$names
[1] userid password playlist
$class
[1] user
Now to create generic functions that we can use with our new class. Start by creating a function that will handle only
our user object; then generalize it so any class can use it. It will be the createPlaylist() function and it will accept the
user on which to perform the operation and a playlist to set. The syntax for this is [function name].[class name].
Note that we access the properties of S3 objects using the dollar sign:
createPlaylist.user - function(user, playlist=NULL){
user$playlist - playlist
return(user)
}](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-54-2048.jpg)
![To His Majesty the King of Prussia.[59]
Berlin, October, 28th, 1842
Your Majesty,
In the memorable words your Majesty was pleased to address to me, you
mentioned that it was intended to add a certain number of able singers to
the existing Royal Church choirs, to form a nucleus for these choirs, as well
as for any amateurs of singing who might subsequently wish to join them,
serving as a rallying-point and example, and in this manner gradually to
elevate and to ennoble church music, and to ensure its greater development.
Also, in order to support the singing of the congregation by instruments,
which produce the most solemn and noble effects,—as your Majesty may
remember, during the celebration of the Jubilee in the Nicolai Church,—it is
proposed that a small number of instrumentalists (probably selected from
the members of the Royal Orchestra) should be engaged, who are also
intended to form the basis for subsequent grand performances of oratorios,
etc.
The direction of a musical choir of this instructive nature, a genuine
Royal Orchestra, your Majesty expressed your intention to entrust to me,
but, till its formation, to grant me entire freedom of choice with regard to
my place of residence.
The execution of this plan will fulfil to the utmost all my wishes as to
public musical efficiency; I can never cease to be grateful to your Majesty
for it, and I do not doubt that the organization of such an institution could
be effected here without any serious difficulties.
But I would request your Majesty not to devolve this organization on me
personally, but merely to permit me to co-operate with my opinion and
advice, which I shall always be gladly prepared to give. Until however, to
use your Majesty’s own expression, the instrument is ready on which I am
hereafter to play, I wish to make use of the freedom of action so graciously
accorded me, and shortly to return to Leipzig, for the direction of the Town
Hall concerts. The orders which your Majesty was pleased to give me, I
shall there with the utmost zeal and to the best of my abilities carry into
execution; at the same time I entreat your Majesty, as I am engaged in no](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-58-2048.jpg)
![account, I begged to relinquish one-half of my salary, so long as I was not
really engaged in active work. This proposal he accepted, and I am now
here again with my wife and child. I have been obliged definitively to
decline the offers of the King of Saxony; but in order to do so in the most
respectful manner, I went to Dresden a few days after my return here,
thanked the King once more verbally, and entreated him not the less to
bestow the twenty thousand thalers (which an old Leipziger bequeathed in
his will to the King for the establishment of an Academy of Art) to found a
school for music in Leipzig, to which he graciously acceded. The official
announcement came the day before yesterday. This music school is to be
organized next winter, at least in its chief features; when it is established, I
may well say that I have been the means of procuring a durable benefit for
music here. If they begin anything solid in Berlin, I can settle there with a
clear conscience; if they allow the matter to stand over, it is probable that I
may go on with my half-salary and my situation here for more than a year,
and my duties be confined, as now, to executing particular commands of the
King,—for instance, I am to supply him with music for the “Midsummer
Night’s Dream,” the “Storm,” and “Œdipus Coloneus.”
Such then is the desired conclusion of this long, long transaction.
Forgive all these details, but I wished to inform you minutely of every
particular.
A request occurs to me which I long ago intended to have made to you.
In Switzerland I saw my former guide, Michael, whom, on my previous
mountain-expeditions, I always found to be an excellent, honest, obliging
fellow, and on this occasion I met with him again, married to a charming
pretty woman; he has children, and is no longer a guide, but established as
landlord of the ‘Krone.’ During our first visit to Meiringen this summer, we
lived at the Hôtel de Reichenbach, but the second time we were at the
‘Krone,’ and quite delighted with the cleanliness, and neatness, and the civil
behaviour of all the people in the house. It is a most genuine Swiss village
inn, taken in its best sense. Now Michael’s greatest wish is to be named
among the inns at Meiringen, in the new edition of Murray’s ‘Switzerland,’
and I promised to endeavour to effect this for him.[60] Is it in your power to
get this done? The first inn there is the ‘Wilde Mann,’ the second the
‘Reichenbach,’ and the third undoubtedly the ‘Krone;’ and if Murray
recommends it as such, I am convinced it will do him credit. He might also
mention that it is most beautifully situated, with a full view of the](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-62-2048.jpg)
![longer stay is unfortunately impossible; still you and I can have some
political gossip together, and be inseparable during my stay.
The King having on this occasion conferred on me a new title,[61] almost
embarrasses me; I am unwilling to be of the number of those in the present
day, who possess a greater number of decorations than they have written
good compositions, and yet it seems rather like it; at all events, I really have
no idea what return I can possibly make for all this, still, as I have not in
any way sought it, I may be excused. To refuse such a thing is out of the
question, and there is no one who does not rejoice in being over-estimated,
because on some other occasion the balance is sure to be made even by
depreciation.—Ever your
Felix.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-67-2048.jpg)
![To His Mother.
Leipzig, December 11th, 1842.
Dearest Mother,
On the 21st or 22nd, we give a concert here for the King, who has sworn
death and destruction to all the hares in the country round. In this concert
we mean to sing for his benefit (how touching!) the partridge and hare hunt
out of the “Seasons.” My “Walpurgis Nacht” is to appear once more in the
second part, in a somewhat different garb indeed from the former one,
which was somewhat too richly endowed with trombones, and rather poor
in the vocal parts; but to effect this, I have been obliged to re-write the
whole score from A to Z, and to add two new arias, not to mention the rest
of the clipping and cutting. If I don’t like it now, I solemnly vow to give it
up for the rest of my life. I think of bringing with me to Berlin a movement
from the “Midsummer Night’s Dream,” and one from “Œdipus.” The music
school here, please God! will make a beginning next February; Hauptmann,
David, Schumann and his wife, Becker, Pohlenz, and I, are to be the
teachers at first. It commences with ten sinecures; the rest who may wish to
have instruction, must pay seventy-five thalers a year. Now you know all
that I know, the rest can only be taught by experience and trial.
I wished for you recently at a subscription concert. I think I never played
the Beethoven G major concerto so well,—my old cheval de bataille; the
first cadence especially, and a new return to the solo, pleased me
exceedingly, and apparently the audience still more.
What you write to me about the répertoire of your Berlin concerts, does
not inspire me with any wish to hear more about them. The arrangement of
the “Aufforderung zum Tanz,” and the compositions of English
ambassadors,—these are valuable things! If experiments are to be thus
made and listened to, it would be advisable to be rather more liberal
towards the works of our Fatherland. You will again say that I am cynical;
but many of my ideas are so intimately connected with my life and my
views on art, that you must be indulgent with regard to them.
The monument to old Sebastian Bach is now very handsome.[62]
Bendemann was here the day before yesterday, to inspect it once more. All](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-68-2048.jpg)
![the inner scaffolding had been removed, so the pillars and smaller columns,
and scrolls, and above all the bas-reliefs, and the grand, antiquated old
features sparkled clearly in the sun, and caused me great delight. The whole
structure, with its numerous elegant decorations, is really typical of the old
fellow. It is now covered up again, and will remain so till March, when it is
to be inaugurated on his birthday, by one of his motetts. Cedars are to be
planted round the monument, and a Gothic seat placed in front of it. We are
anxious, however, not to make too much fuss on the subject, and to avoid
the present pompous style of phraseology, and the worship of art and artists,
which is so much the fashion.
Here, the outward aspect of things is now as much too flourishing, as it
formerly was too miserable for artists, which would be very pleasant for us,
but it does harm to the cause. Art is becoming spoiled and sluggish, so we
should rather be grateful to our present enemies than be angry with them. I
also consider it too much good fortune that the King of Prussia has
nominated me General Music Director. This is another new title and new
honour, whereas I really do not know how to do enough to deserve the old
ones.
This is a hallowed day for us all, with its delightful and memorable
recollections;[63] think of me too on this anniversary, as I do of you and of
him, so long as life endures.—Your
Felix.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-69-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, December 22nd, 1842.[64]
My dear Brother,
I wrote to you the day after our arrival here that we were all well, and
living in our sorrow as we best could, dwelling on the happiness we once
possessed. My letter was addressed to Fanny, but written to you all; though
it seems you had not heard of it, and even this trifle shows, what will day by
day be more deeply and painfully felt by us,—that the point of union is now
gone, where even as children we could always meet; and though we were
no longer so in years, we felt that we were still so in feeling. When I wrote
to my Mother, I knew that I wrote to you all, and you knew it too; we are
children no longer, but we have enjoyed what it really is to be so. Now, this
is gone for ever! At such a time, with regard to outward things, we are as if
in a dark room, groping to find the way, hour after hour. Tell me if we
cannot arrange that I should write to one of you by turns once every week,
and get an answer from you, so that we may at least hear of each other
every three weeks, independent of more frequent letters; or say whether any
better arrangement occurs to you. I thank you a thousand times for your
kind question about the house. I had thought of asking you for it, and now
you offer it to me. But before we finally settle this, I should like you to
bring the subject cautiously on the tapis, in the presence of our sisters and
brother-in-law. If you perceive that any unpleasant feeling is awakened in
their minds by such a proposal, when for the first time, in Berlin, I am not
to live under the same roof with them, and if they give any indication of
such a feeling, even by a single word or remark, (you will quickly observe
this, and I rely entirely on you,) then we must give it up. In any other event,
I shall thankfully accept your kindness. My next visit to Berlin will be a
severe trial to me; indeed, all I say and do is a trial to me,—anything, in
short, that is not mere patient endurance. I have, however, begun to work
again, and that is the only thing which occupies me a little. Happily, I have
some half-mechanical work to do,—transcribing, instrumentation, and
similar things. This can be accomplished by a kind of almost animal
instinct, which we can follow, and which does us more good than if we had
it not. But yesterday I was obliged to direct. That was terrible. They told me](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-72-2048.jpg)
![that the first time would be terrible, but sooner or later it must be done. I
thought so too, but I would fain have waited for a few weeks. The first thing
was a song of Rochlitz’s; but when in the rehearsal the alto sang, piano,
“Wie der Hirsch schreit,”[65] I was so overcome, that I was obliged
afterwards to go out of the room, to give free vent to my tears.
To-day, Heaven be praised, I am not required to see or speak to any one,
and my cough is better. Thus time glides on; but what we have once
possessed is not less precious, and what we have now lost not less painful
with time. Farewell, dearest Brother. Continue to love me.—Your
Felix.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-73-2048.jpg)
![heading of the first introduction, 6/4 time, afterwards repeated, may not
give rise to misapprehension? If I am not mistaken it is marked moderato
sostenuto. Instead of this sostenuto, ought it not rather to be printed con
moto, or con molto di moto? That heading would, it seems to me, lead to the
right tempo, if it were 6/8 time instead of 6/4; but in 6/4 time, it is so very
customary to count the separate crotchets slowly and deliberately, that I
think the movement would be taken too slow, which I found to be the case
at the first rehearsal, until I no longer paid any attention to the notes or the
heading, but adhered to the sense alone. As many musicians cling so closely
to such headings, I was resolved at all events to mention to you my doubts
on this subject.
Allow me to thank you once more for your obliging letter, and the
friendly intention which you inform me of in it;[66] but I thank you still
more for the pleasure which you have caused me by the work itself; and
pray believe that no one will follow your future course with warmer
sympathy, or anticipate your future works with more anxiety and hope than
your
Felix Mendelssohn Bartholdy.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-81-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, July 21st, 1843.
Dear Brother,
I had almost hoped to be able to answer your letter in person, for I was
very nearly taking a journey to Berlin again. Herr von Massow has sent me
a communication connected with that tedious everlasting affair, which
irritated me so much that it almost made me ill, and I do not feel right yet.
In my first feeling of anger, I wished to go to Berlin to speak to you and
break off the whole affair; but I prefer writing, and so I am now writing to
you. Instead of receiving the assent to the proposals on which we had
agreed in the interview of the 10th,[67] Herr von Massow sends me a
commission to arrange for orchestra and chorus, without delay, the chorale,
“Herr Gott, Dich loben wir,” the longest chorale and the most tiresome
work which I ever attempted; and the day after I had finished it and sent it
off, I receive an official document which I must sign before the assent of
the King can be solicited; when I had signed it, the others present at that
conference would also subscribe their names. In this deed all the
stipulations are correctly stated, but six or eight additional clauses are
written on the margin, not one syllable of which had ever been named
during the conference, invalidating the whole intention of the above
stipulations, and placing myself and the Institute in the most entire
subservience to Herr von Küstner,—and in short, showing in the clearest
light all the difficulties to which I formerly alluded, and the existence of
which Herr von Massow denied. Among other things, it is said, the
appointment of the orchestra for all church music is to be devolved on the
theatrical music direction; before every concert there must be an
application made to the General Intendancy, whether the day, which
according to our agreement was to be settled once for all at the beginning of
the winter, is to continue the same or be altered, etc.; all things of which not
one syllable had been alluded to in the conference. As I told you, I fretted
myself till I was quite ill about it. Remembering your words, I thought it the
most judicious plan to write direct to the King, and break off the affair.
After two days’ consideration, I did not think I was justified in doing so; I
therefore wrote to Herr von Massow, why and wherefore I could not give](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-85-2048.jpg)
![my signature, requesting him to inform me whether the King intended to
carry out our former agreement. If he did not feel disposed to do so, or if he,
Herr von Massow, considered it necessary to insert new clauses in the
agreement, I should then consider the affair impracticable, and must act
accordingly. In the other view of the case, he knew that I was prepared to
come; I was also to say how far I had got with “Œdipus.” I answered that in
accordance with Tieck’s wish, I had arranged the “Midsummer Night’s
Dream” with music, to be performed in the new palace; that I had also, by
special commission from the King, written choruses,[68] and that I had not
resumed the choruses of “Œdipus” since the previous autumn, because
another Greek piece had been appointed to be performed. I said all this in a
friendly manner, but I do assure you that the affair cost me four most angry,
disturbed, and irksome days. If I could only have spoken to you for a single
hour! I should have been glad to know whether you approved of my course,
that is of my letter, or whether you would have preferred a short letter
resigning the appointment. It is really too provoking that in all and
everything the same spirit prevails; in this case too, all might be smoothed
over and set to rights by a few words, and every moment I expect to hear
them spoken, and then there would be a possibility of something good and
new; but they are not spoken, and they are replaced by a thousand
annoyances, and my head at last is so bewildered that I think I become
almost as perverted and unnatural, as the whole affair is at last likely to turn
out. Forgive me for causing you to have your share of annoyance, but now I
have told you all—and enough. I have not been able to work during these
days. To make up for this, I have done the “Jungfrau” for you in Indian ink;
the mountain I think is excellent, but I have again utterly destroyed the
pines in the foreground. I mean now, too, to resume your sonata.—Your
Felix.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-86-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, August 26th, 1843.
Dear Brother,
I yesterday received a letter from Herr von Massow containing the
intelligence that the King had fully sanctioned the affair of the Wirklich
Geheimrath; I wished to write this to you instantly.[69] To-day I got a
second letter, with the information that the King desires to have three
representations in the New Palace in the second half of September, namely,
1, “Antigone;” 2, “The Midsummer Night’s Dream;” 3, “Athalia” (“Medea”
is to be given between Nos. 1 and 2, and all the four within fourteen days),
and I am invited to Berlin for the purpose. Now I would rather not write, for
I have a frightful quantity of things to do before then, as not one of the
scores is yet fit for the transcriber, and the overture to “Athalia” still
wanting, as well as the instrumentation of the whole, etc. etc. I have written
nevertheless that I would come, and the music should be finished.—Ever
your
Felix.](https://image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-89-2048.jpg)
![Chapter 1 ■ Background
10
This chart was created in response to the Fukushima Daiichi nuclear disaster of 2011, and sought to clear up
misinformation and misunderstanding of comparisons being made around the disaster. It did this by demonstrating the
differences in scale for the amount of radiation from sources such as other people or a banana, up to what a fatal dose of
radiation ultimately would be—how all that compared to spending just ten minutes near the Chernobyl meltdown.
Over the last quarter of a century, Edward Tufte, author and professor emeritus at Yale University, has been
working to raise the bar of information graphics. He published groundbreaking books detailing the history of data
visualization, tracing its roots even further back than Playfair, to the beginnings of cartography. Among his principles
is the idea to maximize the amount of information included in each graphic—both by increasing the amount of
variables or data points in a chart and by eliminating the use of what he has coined chartjunk. Chartjunk, according to
Tufte, is anything included in a graph that is not information, including ornamentation or thick, gaudy arrows.
Tufte also invented the sparkline, a time series chart with all axes removed and only the trendline remaining to
show historic variations of a data point without concern for exact context. Sparklines are intended to be small enough
to place in line with a body of text, similar in size to the surrounding characters, and to show the recent or historic
trend of whatever the context of the text is.
Why Data Visualization?
In William Playfair’s introduction to the Commercial and Political Atlas, he rationalizes that just as algebra is the
abbreviated shorthand for arithmetic, so are charts a way to “abbreviate and facilitate the modes of conveying
information from one person to another.” Almost 300 years later, this principle remains the same.
Data visualizations are a universal way to present complex and varied amounts of information, as we saw in our
opening example with the quarterly earnings report. They are also powerful ways to tell a story with data.
Imagine you have your Apache logs in front of you, with thousands of lines all resembling the following:
127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET / HTTP/1.1 200 468 - Mozilla/5.0 (X11; U;
Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty)
127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET /favicon.ico HTTP/1.1 200 766 - Mozilla/5.0
(X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty)
Among other things, we see IP address, date, requested resource, and client user agent. Now imagine this
repeated thousands of times—so many times that your eyes kind of glaze over because each line so closely resembles
the ones around it that it’s hard to discern where each line ends, let alone what cumulative trends exist within.
By using some analysis and visualization tools such as R, or even a commercial product such as Splunk, we can
artfully pull out all kinds of meaningful and interesting stories out of this log, from how often certain HTTP errors occur
and for which resources, to what our most widely used URLs are, to what the geographic distribution of our user base is.
This is just our Apache access log. Imagine casting a wider net, pulling in release information, bugs and
production incidents. What insights we could gather about what we do: from how our velocity impacts our defect
density to how our bugs are distributed across our feature sets. And what better way to communicate those findings
and tell those stories than through a universally digestible medium, like data visualizations?
The point of this book is to explore how we as developers can leverage this practice and medium as part of
continual improvement—both to identify and quantify our successes and opportunities for improvements, and more
effectively communicate our learning and our progress.
Tools
There are a number of excellent tools, environments, and libraries that we can use both to analyze and visualize our
data. The next two sections describe them.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-17-2048.jpg)
![Chapter 1 ■ Background
15
Creating data visualizations involves four core steps:
1. Identify a problem.
2. Gather the data.
3. Analyze the data.
4. Visualize the data.
Let’s walk through each step in the process and re-create one of the previous charts to demonstrate the process.
Identify a Problem
The very first step is to identify a problem we want to solve. This can be almost anything—from something as
profound and wide-reaching as figuring out why your bug backlog doesn’t seem to go down and stay down, to seeing
what feature releases over a given period in time caused the most production incidents, and why.
For our example, let’s re-create Figure 1-5 and try to quantify the interest in data visualization over time as
represented by the number of New York Times articles on the subject.
Gather Data
We have an idea of what we want to investigate, so let’s dig in. If you are trying to solve a problem or tell a story around
your own product, you would of course start with your own data—maybe your Apache logs, maybe your bug backlog,
maybe exports from your project tracking software.
Note
■
■ If you are focusing on gathering metrics around your product and you don’t already have data handy, you need to
invest in instrumentation.There are many ways to do this, usually by putting logging in your code.At the very least, you want to
log error states and monitor those, but you may want to expand the scope of what you track to include for
debugging purposes
while still respecting both your user’s privacy and your company’s privacy policy. In my book, Pro JavaScript
Performance:
Monitoring and Visualization, I explore ways to track and visualize web and runtime performance.
One important aspect of data gathering is deciding which format your data should be in (if you're lucky) or discovering
which format your data is available in. We’ll next be looking at some of the common data formats in use today.
JSON is an acronym that stands for JavaScript Object Notation. As you probably know, it is essentially a way to
send data as serialized JavaScript objects. We format JSON as follows:
[object]{
[attribute]: [value],
[method] : function(){},
[array]: [item, item]
}
Another way to transfer data is in XML format. XML has an expected syntax, in which elements can have attributes,
which have values, values are always in quotes, and every element must have a closing element. XML looks like this:
parent attribute=value
child attribute=valuenode data/child
/parent
Generally we can expect APIs to return XML or JSON to us, and our preference is usually JSON because as we can
see it is a much more lightweight option just in sheer amount of characters used.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-22-2048.jpg)
![Chapter 1 ■ Background
17
byline: By AUTHOR,
date: 20121011,
title: TITLE,
url: http://www.nytimes.com/foo.html
}, {
body: BODY COPY,
byline: By AUTHOR,
date: 20121021,
title: TITLE,
url: http://www.nytimes.com/bar.html
}
],
tokens: [
JavaScript
],
total: 2
}
Looking at the high-level JSON structure, we see an attribute named offset, an array named results, an array
named tokens, and another attribute named total. The offset variable is for pagination (what page full of results
we are starting with). The total variable is just what it sounds like: the number of results that are returned for our
query. It’s the results array that we really care about; it is an array of objects, each of which corresponds to an article.
The article objects have attributes named body, byline, date, title, and url.
We now have data that we can begin to look at. That takes us to our next step in the process, analyzing our data.
DATA SCRUBBING
There is often a hidden step here, one that anyone who’s dealt with data knows about: scrubbing the data. Often
the data is either not formatted exactly as we need it or, in even worse cases, it is dirty or incomplete.
In the best-case scenario in which your data just needs to be reformatted or even concatenated, go ahead and do
that, but be sure to not lose the integrity of the data.
Dirty data has fields out of order, fields with obviously bad information in them—think strings in ZIP codes—or
gaps in the data. If your data is dirty, you have several choices:
You could drop the rows in question, but that can harm the integrity of the data—a good example
•
is if you are creating a histogram removing rows could change the distribution and change what
your results will be.
The better alternative is to reach out to whoever administers the source of your data and try and
•
get a better version if it exists.
Whatever the case, if data is dirty or it just needs to be reformatted to be able to be imported into R, expect to
have to scrub your data at some point before you begin your analysis.
Analyze Data
Having data is great, but what does it mean? We determine it through analysis.
Analysis is the most crucial piece of creating data visualizations. It’s only through analysis that we can understand
our data, and it is only through understanding it that we can craft our story to share with others.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-24-2048.jpg)
![Chapter 1 ■ Background
19
if(length(author) 1){
author - unknown
}
datestamp -paste(month, /, day, /, year, sep=)
datestamp - as.Date(datestamp,%m/%d/%Y)
newrow - data.frame(datestamp, author, title, stringsAsFactors=FALSE, check.rows=FALSE)
df - rbind(df, newrow)
}
rownames(df) - df$datestamp
Note that our example assumes that the data set returned has unique date values. If you get errors with this, you
may need to scrub your returned data set to purge any duplicate rows.
Once our data frame is populated, we can start to do some analysis on the data. Let’s start out by pulling just the
year from every entry, and quickly making a stem and leaf plot to see the shape of the data.
Note
■
■ John Tukey created the stem and leaf plot in his seminal work, Exploratory Data Analysis. Stem and leaf plots
are quick, high-level ways to see the shape of data, much like a histogram. In the stem and leaf plot, we construct the
“stem” column on the left and the “leaf” column on the right. The stem consists of the most significant unique elements
in a result set. The leaf consists of the remainder of the values associated with each stem. In our stem and leaf plot below,
the years are our stem and R shows zeroes for each row associated with a given year. Something else to note is that
often alternating sequential rows are combined into a single row, in the interest of having a more concise visualization.
First, we will create a new variable to hold the year information:
yearlist - as.POSIXlt(df$datestamp)$year+1900
If we inspect this variable, we see that it looks something like this:
yearlist
[1] 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2011 2011 2011 2011 2011 2011
2011 2011 2011 2011 2011 2011 2011 2011 2011 2011
[30] 2011 2011 2011 2011 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2009 2009 2009 2009 2009
2009 2009 2008 2008 2008 2007 2007 2007 2007 2006
[59] 2006 2006 2006 2005 2005 2005 2005 2005 2005 2004 2003 2003 2003 2002 2002 2002 2002 2001 2001
2000 2000 2000 2000 2000 2000 1999 1999 1999 1999
[88] 1999 1999 1998 1998 1998 1997 1997 1996 1996 1995 1995 1995 1993 1993 1993 1993 1992 1991 1991
1991 1990 1990 1990 1990 1989 1989 1989 1988 1988
[117] 1988 1986 1985 1985 1985 1984 1982 1982 1981
That’s great, that’s exactly what we want: a year to represent every article returned. Next let’s create the stem and
leaf plot:
stem(yearlist)
1980 | 0
1982 | 00
1984 | 0000
1986 | 0
1988 | 000000](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-26-2048.jpg)
![Chapter 1 ■ Background
20
1990 | 0000000
1992 | 00000
1994 | 000
1996 | 0000
1998 | 000000000
2000 | 00000000
2002 | 0000000
2004 | 0000000
2006 | 00000000
2008 | 0000000000
2010 | 000000000000000000000000000000
2012 | 0000000000000
Very interesting. We see a gradual build with some dips in the mid-1990s, another gradual build with another dip
in the mid-2000s and a strong explosion since 2010 (the stem and leaf plot groups years together in twos).
Looking at that, my mind starts to envision a story building about a subject growing in popularity. But what
about the authors of these articles? Maybe they are the result of one or two very interested authors that have quite
a bit to say on the subject.
Let’s explore that idea and take a look at the author data that we parsed out. Let’s look at just the unique authors
from our data frame:
length(unique(df$author))
[1] 81
We see that there are 81 unique authors or combination of authors for these articles! Just out of curiosity, let’s take
a look at the breakdown by author for each article. Let’s quickly create a bar chart to see the overall shape of the data
(the bar chart is shown in Figure 1-17):
plot(table(df$author), axes=FALSE)
Figure 1-17. Bar chart of number of articles by author to quickly visualize](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-27-2048.jpg)
![Chapter 2 ■ R Language Primer
29
The Command Line
The R console is where the magic happens! It is a command-line environment where we can run R expressions. The best
way to get up to speed in R is to script in the console, a piece at a time, generally to try out what you’re trying to do, and
tweak it until you get the results that you want. When you finally have a working example, take the code that does what
you want and save it as an R script file.
R script files are just files that contain pure R and can be run in the console using the source command:
source(someRfile.R)
Looking at the preceding code snippet, we assume that the R script lives in the current work directory. The way
we can see what the current work directory is to use the getwd() function:
getwd()
[1] /Users/tomjbarker
We can also set the working directory by using the setwd() function. Note that changes made to the working
directory are not persisted across R sessions unless the session is saved.
setwd(/Users/tomjbarker/Downloads)
getwd()
[1] /Users/tomjbarker/Downloads
Command History
The R console stores commands that you enter and you can cycle through previous commands by pressing the up
arrow. Hit the escape button to return to the command prompt. We can see the history in a separate window pane
by clicking the Show/Hide Command History button at the top of the console. The Show/Hide Command History
button is the rectangle icon with alternating stripes of yellow and green. See Figure 2-5 for the R console with the
command history shown.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-35-2048.jpg)
![Chapter 2 ■ R Language Primer
31
Packages
Speaking of packages, what are they, exactly? Packages are collections of functions, data sets, or objects that can
be imported into the current session or workspace to extend what we can do in R. Anyone can make a package
and distribute it.
To install a package, we simply type this:
install.packages([package name])
For example, if we want to install the ggplot2 package—which is a widely used and very handy charting
package—we simply type this into the console:
install.packages(ggplot2)
We are immediately prompted to choose the mirror location that we want to use, usually the one closest to our
current location. From there, the install begins. We can see the results in Figure 2-7.
Figure 2-6. Help search results window](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-37-2048.jpg)
![Chapter 2 ■ R Language Primer
33
codetools Code Analysis Tools for R
colorspace Color Space Manipulation
compiler The R Compiler Package
datasets The R Datasets Package
dichromat Color schemes for dichromats
digest Create cryptographic hash digests of R objects
foreign Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, dBase,
...
ggplot2 An implementation of the Grammar of Graphics
gpairs gpairs: The Generalized Pairs Plot
graphics The R Graphics Package
grDevices The R Graphics Devices and Support for Colours and Fonts
grid The Grid Graphics Package
gtable Arrange grobs in tables.
KernSmooth Functions for kernel smoothing for Wand Jones (1995)
labeling Axis Labeling
lattice Lattice Graphics
mapdata Extra Map Databases
mapproj Map Projections
maps Draw Geographical Maps
Importing Data
So now our environment is downloaded and installed, and we know how to install any packages that we may need.
Now we can begin using R.
The first thing we’ll normally want to do is import your data. There are several ways to import data, but the most
common way is to use the read() function, which has several flavors:
read.table([file to read])
read.csv([file to read])
To see this in action, let’s first create a text file named temptext.txt that is formatted like so:
134,432,435,313,11
403,200,500,404,33
77,321,90,2002,395
We can read this into a variable that we will name temptxt:
temptxt - read.table(temptext.txt)
Notice that as we are assigning value to this variable, we are not using an equal sign as the assignment operator.
We are instead using an arrow -. That is R’s assignment operator, although it does also support the equal sign if you
are so inclined. But the standard is the arrow, and all examples that we will show in this book will use the arrow.
If we print out the temptxt variable, we see that it is structured as follows:
temptxt
V1
1 134,432,435,313,11
2 403,200,500,404,33
3 77,321,90,2002,395](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-39-2048.jpg)
![Chapter 2 ■ R Language Primer
34
We see that our variable is a table-like structure called a data frame, and R has assigned a column name (V1) and
row IDs to our data structure. More on column names soon.
The read() function has a number of parameters that you can use to refine how the data is imported and
formatted once it is imported.
Using Headers
The header parameter tells R to treat the first line in the external file as containing header information. The first line
then becomes the column names of the data frame.
For example, suppose we have a log file structured like this:
url, day, date, loadtime, bytes, httprequests, loadtime_repeatview
http://apress.com, Sun, 01 Jul 2012 14:01:28 +0000,7042,956680,73,3341
http://apress.com, Sun, 01 Jul 2012 14:01:31 +0000,6932,892902,76,3428
http://apress.com, Sun, 01 Jul 2012 14:01:33 +0000,4157,594908,38,1614
We can load it into a variable named wpo like so:
wpo - read.table(wpo.txt, header=TRUE)
wpo
url day date loadtime bytes httprequests loadtime_repeatview
1 http://apress.com,Sun,1 Jul 2012 14:01:28 +0000,7042,955550,73,3191
2 http://apress.com,Sun,1 Jul 2012 14:01:31 +0000,6932,892442,76,3728
3 http://apress.com,Sun,1 Jul 2012 14:01:33 +0000,4157,614908,38,1514
When we call the colnames() function to see what the column names are for wpo, we see the following:
colnames(wpo)
[1] url day date loadtime
[5] bytes httprequests loadtime_repeatview
Specifying a String Delimiter
The sep attribute tells the read() function what to use as the string delimiter for parsing the columns in the external
data file. In all the examples we’ve looked at so far, commas are our delimiters, but we could use instead pipes | or any
other character that we want.
Say, for example, that our previous temptxt example used pipes; we would just update the code to be as follows:
134|432|435|313|11
403|200|500|404|33
77|321|90|2002|395
temptxt - read.table(temptext.txt, sep=|)
temptxt
V1 V2 V3 V4 V5
1 134 432 435 313 11
2 403 200 500 404 33
3 77 321 90 2002 395
Oh, notice that? We actually got distinct column names this time (V1, V2, V3, V4, V5). Before, we didn’t specify a
delimiter, so R assumed that each row was one big blob of text and lumped it into a single column (V1).](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-40-2048.jpg)
![Chapter 2 ■ R Language Primer
36
Data types in R are called modes, and can be the following:
numeric
•
character
•
logical
•
complex
•
raw
•
list
•
We can use the mode() function to check the mode of a variable.
Character and numeric modes correspond to string and number (both integer and float) data types. Logical
modes are Boolean values.
n - 122132
mode(n)
[1] numeric
c - test text
mode(c)
[1] character
l - TRUE
mode(l)
[1] logical
We can perform string concatenation using the paste() function. We can use the substr() function to pull
characters out of strings. Let’s look at some examples in code.
Usually, I keep a list of directories that I either read data from or write charts to. Then when I want to reference
a new data file that exists in the data directory, I will just append the new file name to the data directory:
dataDirectory - /Users/tomjbarker/org/data/
buglist - paste(dataDirectory, bugs.txt, sep=)
buglist
[1] /Users/tomjbarker/org/data/bugs.txt
The paste() function takes N amount of strings and concatenates them together. It accepts an argument named
sep that allows us to specify a string that we can use to be a delimiter between joined strings. We don’t want anything
separating our joined strings that we pass in an empty string.
If we want to pull characters from a string, we use the substr() function. The substr() function takes a string to
parse, a starting location, and a stopping location. It returns all the character inclusively from the starting location up
to the ending location. (Remember that in R, lists are not 0-based like most other languages, but instead have
a starting index of 1.)
substr(test, 1,2)
[1] te
In the preceding example, we pass in the string “test” and tell the substr() function to return the first and
second characters.
Complex mode is for complex numbers. The raw mode is to store raw byte data.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-42-2048.jpg)
![Chapter 2 ■ R Language Primer
37
List data types or modes can be one of three classes: vectors, matrices, or data frames. If we call mode() for vectors
or matrices, they return the mode of the data that they contain; class() returns the class. If we call mode() on a data
frame, it returns the type list:
v - c(1:10)
mode(v)
[1] numeric
m - matrix(c(1:10), byrow=TRUE)
mode(m)
[1] numeric
class(m)
[1] matrix
d - data.frame(c(1:10))
mode(d)
[1] list
class(d)
[1] data.frame
Note that we just typed 1:10 rather than the whole sequence of numbers between 1 and 10:
v - c(1:10)
Vectors are single-dimensional arrays that can hold only values of a single mode at a time. It’s when we get to
data frames and matrices that R really starts to get interesting. The next two sections cover those classes.
Data Frames
We saw at the beginning of this chapter that the read() function takes in external data and saves it as a data frame.
Data frames are like arrays in most other loosely typed languages: they are containers that hold different types of data,
referenced by index. The main thing to realize, though, is that data frames see the data that they contain as rows, columns,
and combinations of the two.
For example, think of a data frame as formatted as follows:
col col col col col
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ]
If we try to reference the first index in the preceding data frame as we traditionally would with an array, say
dataframe[1], R would instead return the first column of data, not the first item. So data frames are referenced by their
column and row. So dataframe[1] returns the first column and dataframe[,2] returns the first row.
Let’s demonstrate this in code.
First let’s create some vectors using the combine function, c(). Remember that vectors are collections of data all
of the same type. The combine function takes a series of values and combines them into vectors.
col1 - c(1,2,3,4,5,6,7,8)
col2 - c(1,2,3,4,5,6,7,8)
col3 - c(1,2,3,4,5,6,7,8)
col4 - c(1,2,3,4,5,6,7,8)](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-43-2048.jpg)
![Chapter 2 ■ R Language Primer
38
Then let’s combine these vectors into a data frame:
df - data.frame(col1,col2,col3,col4)
Now let’s print the data frame to see the contents and the structure of it:
df
col1 col2 col3 col4
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
5 5 5 5 5
6 6 6 6 6
7 7 7 7 7
8 8 8 8 8
Notice that it took each vector and made each one a column. Also notice that each row has an ID; by default,
it is a number, but we can override that.
If we reference the first index, we see that the data frame returns the first column:
df[1]
col1
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
If we put a comma in front of that 1, we reference the first row:
df[,1]
[1] 1 2 3 4 5 6 7 8
So accessing contents of a data frame is done by specifying [column, row].
Matrices work much the same way.
Matrices
Matrices are just like data frames in that they contain rows and columns and can be referenced by either. The core
difference between the two is that data frames can hold different data types but matrices can hold only one type of data.
This presents a philosophical difference. Usually you use data frames to hold data read in externally, like from a
flat file or a database because those are generally of mixed type. You normally store data in matrices that you want to
apply functions to (more on applying functions to lists in a little bit).](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-44-2048.jpg)
![Chapter 2 ■ R Language Primer
39
To create a matrix, we must use the matrix() function, pass in a vector, and tell the function how to distribute
the vector:
The
• nrow parameter specifies how many rows the matrix should have
The
• ncol parameter specifies the number of columns.
The
• byrow parameter tells R that the contents of the vector should be distributed by iterating
across rows if TRUE or by columns if FALSE.
content - c(1,2,3,4,5,6,7,8,9,10)
m1 - matrix(content, nrow=2, ncol=5, byrow=TRUE)
m1
[,1] [,2] [,3] [,4] [,5]
[1,] 1 2 3 4 5
[2,] 6 7 8 9 10
Notice that in the previous example that the m1 matrix is filled in horizontally, row by row. In the following
example, the m1 matrix is filled in vertically by column:
content - c(1,2,3,4,5,6,7,8,9,10)
m1 - matrix(content, nrow=2, ncol=5, byrow=FALSE)
m1
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
Remember that instead of manually typing out all the numbers in the previous content vector, if the numbers are
a sequence we can just type this:
content - (1:10)
We reference the content in matrices with the square bracket, specifying the row and column, respectively.
m1[1,4]
[1] 7
We can convert a data frame to a matrix if the data frame contains only a single type of data. To do this we use the
as.matrix() function. Often times we will do this when passing a data frame to a plotting function to draw a chart.
barplot(as.matrix(df))
Below we create a data frame called df. We populate the data frame with ten consecutive numbers. We then use
as.matrix() to convert df into a matrix and save the result into a new variable called m:
df - data.frame(1:10)
df
X1.10
1 1
2 2
3 3](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-45-2048.jpg)
![Chapter 2 ■ R Language Primer
40
4 4
5 5
6 6
7 7
8 8
9 9
10 10
class(df)
[1] data.frame
m - as.matrix(df)
class(m)
[1] matrix
Keep in mind that because they are all the same data type, matrices require less overhead and are intrinsically
more efficient than data frames. If we compare the size of our matrix m and our data frame df, we see that with just ten
items there is a size difference.
object.size(m)
312 bytes
object.size(df)
440 bytes
With that said, if we increase the scale of this, the increase in efficiency does not equally scale. Compare the following:
big_df - data.frame(1:40000000)
big_m - matrix(1:40000000)
object.size(big_m)
160000112 bytes
object.size(big_df)
160000400 bytes
We can see that the first example with the small data set showed that the matrix was 30 percent smaller in size
than the data frame, but at the larger scale in the second example the matrix was only .00018 percent smaller than
the data frame.
Adding Lists
When combining or adding to data frames or matrices, you generally add either by the row or the column using
rbind() or cbind().
To demonstrate this, let’s add a new row to our data frame df. We’ll pass df into rbind() along with the new row
to add to df. The new row contains just one element, the number 11:
df - rbind(df, 11)
df
X1.10
1 1
2 2
3 3
4 4
5 5
6 6](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-46-2048.jpg)
![Chapter 2 ■ R Language Primer
41
7 7
8 8
9 9
10 10
11 11
Now let’s add a new column to our matrix m. To do this, we simply pass m into cbind() as the first parameter;
the second parameter is a new matrix that will be appended to the new column.
m - rbind(m, 11)
m - cbind(m, matrix(c(50:60), byrow=FALSE))
m
X1.10
[1,] 1 50
[2,] 2 51
[3,] 3 52
[4,] 4 53
[5,] 5 54
[6,] 6 55
[7,] 7 56
[8,] 8 57
[9,] 9 58
[10,] 10 59
[11,] 11 60
What about vectors, you may ask? Well, let’s look at adding to our content vector. We simply use the combine
function to combine the current vector with a new vector:
content - c(1,2,3,4,5,6,7,8,9,10)
content - c(content, c(11:20))
content
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Looping Through Lists
As developers who generally work in procedural languages, or at least came up the ranks using procedural languages
(though in recent years functional programming paradigms have become much more mainstream), we’re most
likely used to looping through our arrays when we want to process the data within them. This is in contrast to purely
functional languages where we would instead apply a function to our lists, like the map() function. R supports both
paradigms. Let’s first look at how to loop through our lists.
The most useful loop that R supports is the for in loop. The basic structure of a for in loop can be seen here:.
for(i in 1:5){print(i)}
[1] 1
[1] 2
[1] 3
[1] 4
[1] 5](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-47-2048.jpg)
![Chapter 2 ■ R Language Primer
42
The variable i increments in value each step through the iteration. We can use the for in loop to step through
lists. We can specify a particular column to iterate through, like the following, in which we loop through the X1.10
column of the data frame df.
for(n in df$X1.10){ print(n)}
[1] 1
[1] 2
[1] 3
[1] 4
[1] 5
[1] 6
[1] 7
[1] 8
[1] 9
[1] 10
[1] 11
Note that we are accessing the columns of data frames via the dollar sign operator. The general pattern is
[data frame]$[column name].
Applying Functions to Lists
But the way that R really wants to be used is to apply functions to the contents of lists (see Figure 2-8).
function
element
element
element
element
Figure 2-8. Apply a function to list elements
We do this in R with the apply() function.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-48-2048.jpg)
![Chapter 2 ■ R Language Primer
43
The apply() function takes several parameters:
First is our list.
•
Next a number vector to indicate how we apply the function through the list (
• 1 is for rows, 2 is
for columns, and c[1,2] indicates both rows and columns).
Finally is the function to apply to the list:
•
apply([list], [how to apply function], [function to apply])
Let’s look at an example. Let’s make a new matrix that we’ll call m. The matrix m will have ten columns and four rows:
m - matrix(c(1:40), byrow=FALSE, ncol=10)
m
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 5 9 13 17 21 25 29 33 37
[2,] 2 6 10 14 18 22 26 30 34 38
[3,] 3 7 11 15 19 23 27 31 35 39
[4,] 4 8 12 16 20 24 28 32 36 40
Now say we wanted to increment every number in the m matrix. We could simply use apply() as follows:
apply(m, 2, function(x) x - x + 1)
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 2 6 10 14 18 22 26 30 34 38
[2,] 3 7 11 15 19 23 27 31 35 39
[3,] 4 8 12 16 20 24 28 32 36 40
[4,] 5 9 13 17 21 25 29 33 37 41
Do you see what we did there? We passed in m, we specified that we wanted to apply the function across the
columns, and finally we passed in an anonymous function. The function accepts a parameter that we called x.
The parameter x is a reference to the current matrix element. From there, we just increment the value of x by 1.
OK, say we wanted to do something slightly more interesting, such as zeroing out all the even numbers in the
matrix. We could do the following:
apply(m,c(1,2),function(x){if((x %% 2) == 0) x - 0 else x - x})
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 5 9 13 17 21 25 29 33 37
[2,] 0 0 0 0 0 0 0 0 0 0
[3,] 3 7 11 15 19 23 27 31 35 39
[4,] 0 0 0 0 0 0 0 0 0 0
For the sake of clarity let’s break out that function that we are applying. We simply check to see whether the
current element is even by checking to see whether it has a remainder when divided by two. If it is, we set it to zero;
if it isn’t, we set it to itself:
function(x){
if((x %% 2) == 0)
x - 0
else
x - x
}](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-49-2048.jpg)
{
[body of function]
}
Something interesting that R allows is the ... argument (sometimes called the dots argument). This allows us to
pass in a variable number of parameters into a function. Within the function, we can convert the ... argument into a list
and iterate over the list to retrieve the values within:
offset - function (...){
for(i in list(...)){
print(i)
}
}
offset(23,11)
[1] 23
[1] 11
We can even store values of different data types (modes) in the ... argument:
offset(test value, 12, 100, 19ANM)
[1] test value
[1] 12
[1] 100
[1] 19ANM
R uses lexical scoping. This means that when we call a function and try to reference variables that are not defined
inside the local scope of the function, the R interpreter looks for those variables in the workspace or scope in which the
function was created. If the R interpreter cannot find those variables in that scope, it looks in the parent of that scope.
If we create a function A within function B, the creation scope of function A is function B. For example, see the
following code snippet:
x - 10
wrapper - function(y){
x - 99
c- function(y){
print(x + y)
}
return(c)
}
t - wrapper()
t(1)
[1] 100
x
[1] 10](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-50-2048.jpg)
![Chapter 3 ■ A Deeper Dive into R
48
There also exists the concept of polymorphism, in which we can change functionality via the inheritance chain.
Specifically, we would inherit a function from a base class but override it, keep the signature (the function name, the
type and amount of parameters it accepts, and the type of data that it returns) the same, but change what the function
does. Compare overriding a function to the contrasting concept of overloading a function, in which the function
would have the same name but a different signature and functionality.
S3 Classes
S3, so called because it was first implemented in version 3 of the S language, uses a concept called generic functions.
Everything in R is an object, and each object has a string property called class that signifies what the object is. There
is no validation around it, and we can overwrite the class property ad hoc. That’s the main problem with S3—the
lack of validation. If you ever had an esoteric error message returned when trying to use a function, you probably
experienced the repercussions of this lack of validation firsthand. The error message was probably generated not from
R detecting that an incorrect type had been passed in, but from the function trying to execute with what was passed in
and failing at some step along the way.
See the following code, in which we create a matrix and change its class to be a vector:
m - matrix(c(1:10), nrow=2)
m
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
class(m) - vector
m
[,1] [,2] [,3] [,4] [,5]
[1,] 1 3 5 7 9
[2,] 2 4 6 8 10
attr(,class)
[1] vector
BaseUser
login()
createPlaylist
extends extends
implements
User
login()
createPlaylist()
SuperUser
login()
createPlaylist()
editPermissions()
IUser
login()
createPlaylist()
Figure 3-2. An IUser interface implemented by a superclass BaseUser that the subclasses User and SuperUser extend](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-53-2048.jpg)
![Chapter 3 ■ A Deeper Dive into R
49
Generic functions are objects that check the class property of objects passed into them and exhibit different
behavior based on that attribute. It’s a nice way to implement polymorphism. We can see the methods that a generic
function uses by passing the generic function to the methods() function. The following code shows the methods of the
plot() generic function:
methods(plot)
[1] plot.acf* plot.data.frame* plot.decomposed.ts* plot.default plot.dendrogram*
[6] plot.density plot.ecdf plot.factor* plot.formula* plot.function
[11] plot.hclust* plot.histogram* plot.HoltWinters* plot.isoreg* plot.lm
[16] plot.medpolish* plot.mlm plot.ppr* plot.prcomp* plot.princomp*
[21] plot.profile.nls* plot.spec plot.stepfun plot.stl* plot.table*
[26] plot.ts plot.tskernel* plot.TukeyHSD
Non-visible functions are asterisked
Notice that within the generic plot() function is a myriad of methods to handle all the different types of data
that could be passed to it, such as plot.data.frame for when we pass a data frame to plot(); or if we want to plot
a TukeyHSD object plot(), plot.TukeyHSD is ready for us.
Note
■
■ Type ?TukeyHSD for more information on this object.
Now that you know how S3 object-oriented concepts work in R, let’s see how to create our own custom S3 objects
and generic functions.
An S3 class is a list of properties and functions with an attribute named class. The class attribute tells generic
functions how to treat objects that implement a particular class. Let’s create an example using the UserClass idea
from Figure 3-2:
tom - list(userid = tbarker, password = password123, playlist=c(12,332,45))
class(tom) - user
We can inspect our new object by using the attributes() function, which tells us the properties that the object
has as well as its class:
attributes(tom)
$names
[1] userid password playlist
$class
[1] user
Now to create generic functions that we can use with our new class. Start by creating a function that will handle only
our user object; then generalize it so any class can use it. It will be the createPlaylist() function and it will accept the
user on which to perform the operation and a playlist to set. The syntax for this is [function name].[class name].
Note that we access the properties of S3 objects using the dollar sign:
createPlaylist.user - function(user, playlist=NULL){
user$playlist - playlist
return(user)
}](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-54-2048.jpg)
![To His Majesty the King of Prussia.[59]
Berlin, October, 28th, 1842
Your Majesty,
In the memorable words your Majesty was pleased to address to me, you
mentioned that it was intended to add a certain number of able singers to
the existing Royal Church choirs, to form a nucleus for these choirs, as well
as for any amateurs of singing who might subsequently wish to join them,
serving as a rallying-point and example, and in this manner gradually to
elevate and to ennoble church music, and to ensure its greater development.
Also, in order to support the singing of the congregation by instruments,
which produce the most solemn and noble effects,—as your Majesty may
remember, during the celebration of the Jubilee in the Nicolai Church,—it is
proposed that a small number of instrumentalists (probably selected from
the members of the Royal Orchestra) should be engaged, who are also
intended to form the basis for subsequent grand performances of oratorios,
etc.
The direction of a musical choir of this instructive nature, a genuine
Royal Orchestra, your Majesty expressed your intention to entrust to me,
but, till its formation, to grant me entire freedom of choice with regard to
my place of residence.
The execution of this plan will fulfil to the utmost all my wishes as to
public musical efficiency; I can never cease to be grateful to your Majesty
for it, and I do not doubt that the organization of such an institution could
be effected here without any serious difficulties.
But I would request your Majesty not to devolve this organization on me
personally, but merely to permit me to co-operate with my opinion and
advice, which I shall always be gladly prepared to give. Until however, to
use your Majesty’s own expression, the instrument is ready on which I am
hereafter to play, I wish to make use of the freedom of action so graciously
accorded me, and shortly to return to Leipzig, for the direction of the Town
Hall concerts. The orders which your Majesty was pleased to give me, I
shall there with the utmost zeal and to the best of my abilities carry into
execution; at the same time I entreat your Majesty, as I am engaged in no](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-58-2048.jpg)
![account, I begged to relinquish one-half of my salary, so long as I was not
really engaged in active work. This proposal he accepted, and I am now
here again with my wife and child. I have been obliged definitively to
decline the offers of the King of Saxony; but in order to do so in the most
respectful manner, I went to Dresden a few days after my return here,
thanked the King once more verbally, and entreated him not the less to
bestow the twenty thousand thalers (which an old Leipziger bequeathed in
his will to the King for the establishment of an Academy of Art) to found a
school for music in Leipzig, to which he graciously acceded. The official
announcement came the day before yesterday. This music school is to be
organized next winter, at least in its chief features; when it is established, I
may well say that I have been the means of procuring a durable benefit for
music here. If they begin anything solid in Berlin, I can settle there with a
clear conscience; if they allow the matter to stand over, it is probable that I
may go on with my half-salary and my situation here for more than a year,
and my duties be confined, as now, to executing particular commands of the
King,—for instance, I am to supply him with music for the “Midsummer
Night’s Dream,” the “Storm,” and “Œdipus Coloneus.”
Such then is the desired conclusion of this long, long transaction.
Forgive all these details, but I wished to inform you minutely of every
particular.
A request occurs to me which I long ago intended to have made to you.
In Switzerland I saw my former guide, Michael, whom, on my previous
mountain-expeditions, I always found to be an excellent, honest, obliging
fellow, and on this occasion I met with him again, married to a charming
pretty woman; he has children, and is no longer a guide, but established as
landlord of the ‘Krone.’ During our first visit to Meiringen this summer, we
lived at the Hôtel de Reichenbach, but the second time we were at the
‘Krone,’ and quite delighted with the cleanliness, and neatness, and the civil
behaviour of all the people in the house. It is a most genuine Swiss village
inn, taken in its best sense. Now Michael’s greatest wish is to be named
among the inns at Meiringen, in the new edition of Murray’s ‘Switzerland,’
and I promised to endeavour to effect this for him.[60] Is it in your power to
get this done? The first inn there is the ‘Wilde Mann,’ the second the
‘Reichenbach,’ and the third undoubtedly the ‘Krone;’ and if Murray
recommends it as such, I am convinced it will do him credit. He might also
mention that it is most beautifully situated, with a full view of the](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-62-2048.jpg)
![longer stay is unfortunately impossible; still you and I can have some
political gossip together, and be inseparable during my stay.
The King having on this occasion conferred on me a new title,[61] almost
embarrasses me; I am unwilling to be of the number of those in the present
day, who possess a greater number of decorations than they have written
good compositions, and yet it seems rather like it; at all events, I really have
no idea what return I can possibly make for all this, still, as I have not in
any way sought it, I may be excused. To refuse such a thing is out of the
question, and there is no one who does not rejoice in being over-estimated,
because on some other occasion the balance is sure to be made even by
depreciation.—Ever your
Felix.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-67-2048.jpg)
![To His Mother.
Leipzig, December 11th, 1842.
Dearest Mother,
On the 21st or 22nd, we give a concert here for the King, who has sworn
death and destruction to all the hares in the country round. In this concert
we mean to sing for his benefit (how touching!) the partridge and hare hunt
out of the “Seasons.” My “Walpurgis Nacht” is to appear once more in the
second part, in a somewhat different garb indeed from the former one,
which was somewhat too richly endowed with trombones, and rather poor
in the vocal parts; but to effect this, I have been obliged to re-write the
whole score from A to Z, and to add two new arias, not to mention the rest
of the clipping and cutting. If I don’t like it now, I solemnly vow to give it
up for the rest of my life. I think of bringing with me to Berlin a movement
from the “Midsummer Night’s Dream,” and one from “Œdipus.” The music
school here, please God! will make a beginning next February; Hauptmann,
David, Schumann and his wife, Becker, Pohlenz, and I, are to be the
teachers at first. It commences with ten sinecures; the rest who may wish to
have instruction, must pay seventy-five thalers a year. Now you know all
that I know, the rest can only be taught by experience and trial.
I wished for you recently at a subscription concert. I think I never played
the Beethoven G major concerto so well,—my old cheval de bataille; the
first cadence especially, and a new return to the solo, pleased me
exceedingly, and apparently the audience still more.
What you write to me about the répertoire of your Berlin concerts, does
not inspire me with any wish to hear more about them. The arrangement of
the “Aufforderung zum Tanz,” and the compositions of English
ambassadors,—these are valuable things! If experiments are to be thus
made and listened to, it would be advisable to be rather more liberal
towards the works of our Fatherland. You will again say that I am cynical;
but many of my ideas are so intimately connected with my life and my
views on art, that you must be indulgent with regard to them.
The monument to old Sebastian Bach is now very handsome.[62]
Bendemann was here the day before yesterday, to inspect it once more. All](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-68-2048.jpg)
![the inner scaffolding had been removed, so the pillars and smaller columns,
and scrolls, and above all the bas-reliefs, and the grand, antiquated old
features sparkled clearly in the sun, and caused me great delight. The whole
structure, with its numerous elegant decorations, is really typical of the old
fellow. It is now covered up again, and will remain so till March, when it is
to be inaugurated on his birthday, by one of his motetts. Cedars are to be
planted round the monument, and a Gothic seat placed in front of it. We are
anxious, however, not to make too much fuss on the subject, and to avoid
the present pompous style of phraseology, and the worship of art and artists,
which is so much the fashion.
Here, the outward aspect of things is now as much too flourishing, as it
formerly was too miserable for artists, which would be very pleasant for us,
but it does harm to the cause. Art is becoming spoiled and sluggish, so we
should rather be grateful to our present enemies than be angry with them. I
also consider it too much good fortune that the King of Prussia has
nominated me General Music Director. This is another new title and new
honour, whereas I really do not know how to do enough to deserve the old
ones.
This is a hallowed day for us all, with its delightful and memorable
recollections;[63] think of me too on this anniversary, as I do of you and of
him, so long as life endures.—Your
Felix.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-69-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, December 22nd, 1842.[64]
My dear Brother,
I wrote to you the day after our arrival here that we were all well, and
living in our sorrow as we best could, dwelling on the happiness we once
possessed. My letter was addressed to Fanny, but written to you all; though
it seems you had not heard of it, and even this trifle shows, what will day by
day be more deeply and painfully felt by us,—that the point of union is now
gone, where even as children we could always meet; and though we were
no longer so in years, we felt that we were still so in feeling. When I wrote
to my Mother, I knew that I wrote to you all, and you knew it too; we are
children no longer, but we have enjoyed what it really is to be so. Now, this
is gone for ever! At such a time, with regard to outward things, we are as if
in a dark room, groping to find the way, hour after hour. Tell me if we
cannot arrange that I should write to one of you by turns once every week,
and get an answer from you, so that we may at least hear of each other
every three weeks, independent of more frequent letters; or say whether any
better arrangement occurs to you. I thank you a thousand times for your
kind question about the house. I had thought of asking you for it, and now
you offer it to me. But before we finally settle this, I should like you to
bring the subject cautiously on the tapis, in the presence of our sisters and
brother-in-law. If you perceive that any unpleasant feeling is awakened in
their minds by such a proposal, when for the first time, in Berlin, I am not
to live under the same roof with them, and if they give any indication of
such a feeling, even by a single word or remark, (you will quickly observe
this, and I rely entirely on you,) then we must give it up. In any other event,
I shall thankfully accept your kindness. My next visit to Berlin will be a
severe trial to me; indeed, all I say and do is a trial to me,—anything, in
short, that is not mere patient endurance. I have, however, begun to work
again, and that is the only thing which occupies me a little. Happily, I have
some half-mechanical work to do,—transcribing, instrumentation, and
similar things. This can be accomplished by a kind of almost animal
instinct, which we can follow, and which does us more good than if we had
it not. But yesterday I was obliged to direct. That was terrible. They told me](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-72-2048.jpg)
![that the first time would be terrible, but sooner or later it must be done. I
thought so too, but I would fain have waited for a few weeks. The first thing
was a song of Rochlitz’s; but when in the rehearsal the alto sang, piano,
“Wie der Hirsch schreit,”[65] I was so overcome, that I was obliged
afterwards to go out of the room, to give free vent to my tears.
To-day, Heaven be praised, I am not required to see or speak to any one,
and my cough is better. Thus time glides on; but what we have once
possessed is not less precious, and what we have now lost not less painful
with time. Farewell, dearest Brother. Continue to love me.—Your
Felix.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-73-2048.jpg)
![heading of the first introduction, 6/4 time, afterwards repeated, may not
give rise to misapprehension? If I am not mistaken it is marked moderato
sostenuto. Instead of this sostenuto, ought it not rather to be printed con
moto, or con molto di moto? That heading would, it seems to me, lead to the
right tempo, if it were 6/8 time instead of 6/4; but in 6/4 time, it is so very
customary to count the separate crotchets slowly and deliberately, that I
think the movement would be taken too slow, which I found to be the case
at the first rehearsal, until I no longer paid any attention to the notes or the
heading, but adhered to the sense alone. As many musicians cling so closely
to such headings, I was resolved at all events to mention to you my doubts
on this subject.
Allow me to thank you once more for your obliging letter, and the
friendly intention which you inform me of in it;[66] but I thank you still
more for the pleasure which you have caused me by the work itself; and
pray believe that no one will follow your future course with warmer
sympathy, or anticipate your future works with more anxiety and hope than
your
Felix Mendelssohn Bartholdy.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-81-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, July 21st, 1843.
Dear Brother,
I had almost hoped to be able to answer your letter in person, for I was
very nearly taking a journey to Berlin again. Herr von Massow has sent me
a communication connected with that tedious everlasting affair, which
irritated me so much that it almost made me ill, and I do not feel right yet.
In my first feeling of anger, I wished to go to Berlin to speak to you and
break off the whole affair; but I prefer writing, and so I am now writing to
you. Instead of receiving the assent to the proposals on which we had
agreed in the interview of the 10th,[67] Herr von Massow sends me a
commission to arrange for orchestra and chorus, without delay, the chorale,
“Herr Gott, Dich loben wir,” the longest chorale and the most tiresome
work which I ever attempted; and the day after I had finished it and sent it
off, I receive an official document which I must sign before the assent of
the King can be solicited; when I had signed it, the others present at that
conference would also subscribe their names. In this deed all the
stipulations are correctly stated, but six or eight additional clauses are
written on the margin, not one syllable of which had ever been named
during the conference, invalidating the whole intention of the above
stipulations, and placing myself and the Institute in the most entire
subservience to Herr von Küstner,—and in short, showing in the clearest
light all the difficulties to which I formerly alluded, and the existence of
which Herr von Massow denied. Among other things, it is said, the
appointment of the orchestra for all church music is to be devolved on the
theatrical music direction; before every concert there must be an
application made to the General Intendancy, whether the day, which
according to our agreement was to be settled once for all at the beginning of
the winter, is to continue the same or be altered, etc.; all things of which not
one syllable had been alluded to in the conference. As I told you, I fretted
myself till I was quite ill about it. Remembering your words, I thought it the
most judicious plan to write direct to the King, and break off the affair.
After two days’ consideration, I did not think I was justified in doing so; I
therefore wrote to Herr von Massow, why and wherefore I could not give](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-85-2048.jpg)
![my signature, requesting him to inform me whether the King intended to
carry out our former agreement. If he did not feel disposed to do so, or if he,
Herr von Massow, considered it necessary to insert new clauses in the
agreement, I should then consider the affair impracticable, and must act
accordingly. In the other view of the case, he knew that I was prepared to
come; I was also to say how far I had got with “Œdipus.” I answered that in
accordance with Tieck’s wish, I had arranged the “Midsummer Night’s
Dream” with music, to be performed in the new palace; that I had also, by
special commission from the King, written choruses,[68] and that I had not
resumed the choruses of “Œdipus” since the previous autumn, because
another Greek piece had been appointed to be performed. I said all this in a
friendly manner, but I do assure you that the affair cost me four most angry,
disturbed, and irksome days. If I could only have spoken to you for a single
hour! I should have been glad to know whether you approved of my course,
that is of my letter, or whether you would have preferred a short letter
resigning the appointment. It is really too provoking that in all and
everything the same spirit prevails; in this case too, all might be smoothed
over and set to rights by a few words, and every moment I expect to hear
them spoken, and then there would be a possibility of something good and
new; but they are not spoken, and they are replaced by a thousand
annoyances, and my head at last is so bewildered that I think I become
almost as perverted and unnatural, as the whole affair is at last likely to turn
out. Forgive me for causing you to have your share of annoyance, but now I
have told you all—and enough. I have not been able to work during these
days. To make up for this, I have done the “Jungfrau” for you in Indian ink;
the mountain I think is excellent, but I have again utterly destroyed the
pines in the foreground. I mean now, too, to resume your sonata.—Your
Felix.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-86-2048.jpg)
![To Paul Mendelssohn Bartholdy.
Leipzig, August 26th, 1843.
Dear Brother,
I yesterday received a letter from Herr von Massow containing the
intelligence that the King had fully sanctioned the affair of the Wirklich
Geheimrath; I wished to write this to you instantly.[69] To-day I got a
second letter, with the information that the King desires to have three
representations in the New Palace in the second half of September, namely,
1, “Antigone;” 2, “The Midsummer Night’s Dream;” 3, “Athalia” (“Medea”
is to be given between Nos. 1 and 2, and all the four within fourteen days),
and I am invited to Berlin for the purpose. Now I would rather not write, for
I have a frightful quantity of things to do before then, as not one of the
scores is yet fit for the transcriber, and the overture to “Athalia” still
wanting, as well as the instrumentation of the whole, etc. etc. I have written
nevertheless that I would come, and the music should be finished.—Ever
your
Felix.](https://crownmelresort.com/image.slidesharecdn.com/27770318-250520152431-be4fd80d/75/Pro-Data-Visualization-Using-R-And-Javascript-1st-Ed-Tom-Barker-89-2048.jpg)
![SHS_Core_CAE_Q3_LE1 FOR THIRD [FINAL].pdf](https://cdn.slidesharecdn.com/ss_thumbnails/shscorecaeq3le1final-251116055110-e3081055-thumbnail.jpg?width=640&height=640&fit=bounds)
Pro Data Visualization Using R And Javascript 1st Ed Tom Barker
- 1. Pro Data VisualizationUsing R And Javascript 1st Ed Tom Barker download https://ebookbell.com/product/pro-data-visualization-using-r-and- javascript-1st-ed-tom-barker-55540636 Explore and download more ebooks at ebookbell.com
- 2. Here are somerecommended products that we believe you will be interested in. You can click the link to download. Pro Data Visualization Using R And Javascript Analyze And Visualize Key Data On The Web 2nd Edition Tom Barker https://ebookbell.com/product/pro-data-visualization-using-r-and- javascript-analyze-and-visualize-key-data-on-the-web-2nd-edition-tom- barker-35015364 Pro Data Visualization Using R And Javascript 1st Edition Tom Barker https://ebookbell.com/product/pro-data-visualization-using-r-and- javascript-1st-edition-tom-barker-35189830 Pro Data Visualization Using R And Javascript Tom Barker https://ebookbell.com/product/pro-data-visualization-using-r-and- javascript-tom-barker-43781096 Pro Power Bi Desktop Selfservice Analytics And Data Visualization For The Power User 3rd Edition Adam Aspin https://ebookbell.com/product/pro-power-bi-desktop-selfservice- analytics-and-data-visualization-for-the-power-user-3rd-edition-adam- aspin-11137514
- 3. Pro Power BiDesktop Selfservice Analytics And Data Visualization For The Power User 3rd Edition Adam Aspin https://ebookbell.com/product/pro-power-bi-desktop-selfservice- analytics-and-data-visualization-for-the-power-user-3rd-edition-adam- aspin-62627316 Pro Data Mashup For Power Bi Powering Up With Power Query And The M Language To Find Load And Transform Data 1st Edition Adam Aspin https://ebookbell.com/product/pro-data-mashup-for-power-bi-powering- up-with-power-query-and-the-m-language-to-find-load-and-transform- data-1st-edition-adam-aspin-44868580 Pro Data Backup And Recovery 1st Edition Steven Nelson https://ebookbell.com/product/pro-data-backup-and-recovery-1st- edition-steven-nelson-2095548 Pro Hadoop Data Analytics Designing And Building Big Data Systems Using The Hadoop Ecosystem 1st Ed Kerry Koitzsch https://ebookbell.com/product/pro-hadoop-data-analytics-designing-and- building-big-data-systems-using-the-hadoop-ecosystem-1st-ed-kerry- koitzsch-55285126 Pro Ember Data Getting Ember Data To Work With Your Api 1st Ed David Tang https://ebookbell.com/product/pro-ember-data-getting-ember-data-to- work-with-your-api-1st-ed-david-tang-22417656
- 6. For your convenienceApress has placed some of the front matter material after the index. Please use the Bookmarks and Contents at a Glance links to access them.
- 7. v Contents at aGlance About the Author���������������������������������������������������������������������������������������������������������������xiii About the Technical Reviewer��������������������������������������������������������������������������������������������xv Acknowledgments������������������������������������������������������������������������������������������������������������xvii Chapter 1: Background ■ ■ ������������������������������������������������������������������������������������������������������1 Chapter 2: R Language Primer ■ ■ ����������������������������������������������������������������������������������������25 Chapter 3: A Deeper Dive into R ■ ■ ��������������������������������������������������������������������������������������47 Chapter 4: Data Visualization with D3 ■ ■ �����������������������������������������������������������������������������65 Chapter 5: Visualizing Spatial Data from Access Logs ■ ■ ����������������������������������������������������85 Chapter 6: Visualizing Data Over Time ■ ■ ��������������������������������������������������������������������������111 Chapter 7: Bar Charts ■ ■ ����������������������������������������������������������������������������������������������������133 Chapter 8: Correlation Analysis with Scatter Plots ■ ■ �������������������������������������������������������157 Chapter 9: Visualizing the Balance of Delivery and Quality with ■ ■ Parallel Coordinates������������������������������������������������������������������������������������������������������177 Index���������������������������������������������������������������������������������������������������������������������������������193
- 8. 1 Chapter 1 Background There isa new concept emerging in the field of web development: using data visualizations as communication tools. This concept is something that is already well established in other fields and departments. At the company where you work, your finance department probably uses data visualizations to represent fiscal information both internally and externally; just take a look at the quarterly earnings reports for almost any publicly traded company. They are full of charts to show revenue by quarter, or year over year earnings, or a plethora of other historic financial data. All are designed to show lots and lots of data points, potentially pages and pages of data points, in a single easily digestible graphic. Compare the bar chart in Google’s quarterly earnings report from back in 2007 (see Figure 1-1) to a subset of the data it is based on in tabular format (see Figure 1-2). Figure 1-1. Google Q4 2007 quarterly revenue shown in a bar chart
- 9. Chapter 1 ■Background 2 The bar chart is imminently more readable. We can clearly see by the shape of it that earnings are up and have been steadily going up each quarter. By the color-coding, we can see the sources of the earnings; and with the annotations, we can see both the precise numbers that those color-coding represent and what the year over year percentages are. With the tabular data, you have to read labels on the left, line up the data on the right with those labels, do your own aggregation and comparison, and draw your own conclusions. There is a lot more upfront work needed to take in the tabular data, and there exists the very real possibility of your audience either not understanding the data (thus creating their own incorrect story around the data) or tuning out completely because of the sheer amount of work needed to take in the information. It’s not just the Finance department that uses visualizations to communicate dense amounts of data. Maybe your Operations department uses charts to communicate server uptime, or your Customer Support department uses graphs to show call volume. Whatever the case, it’s about time Engineering and Web Development got on board with this. As a department, group, and industry we have a huge amount of relevant data that is important for us to first be aware of so that we can refine and improve what we do; but also to communicate out to our stakeholders, to demonstrate our successes or validate resource needs, or to plan tactical roadmaps for the coming year. Before we can do this, we need to understand what we are doing. We need to understand what data visualizations are, a general idea of their history, when to use them, and how to use them both technically and ethically. What Is Data Visualization? OK, so what exactly is data visualization? Data visualization is the art and practice of gathering, analyzing, and graphically representing empirical information. They are sometimes called information graphics, or even just charts and graphs. Whatever you call it, the goal of visualizing data is to tell the story in the data. Telling the story is predicated on understanding the data at a very deep level, and gathering insight from comparisons of data points in the numbers. There exists syntax for crafting data visualizations, patterns in the form of charts that have an immediately known context. We devote a chapter to each of the significant chart types later in the book. Time Series Charts Time series charts show changes over time. See Figure 1-3 for a time series chart that shows the weighted popularity of the keyword “Data Visualization” from Google Trends (http://www.google.com/trends/). Figure 1-2. Similar earnings data in tabular form
- 10. Chapter 1 ■Background 3 Note that the vertical y axis shows a sequence of numbers that increment by 20 up to 100. These numbers represent the weighted search volume, where 100 is the peak search volume for our term. On the horizontal x axis, we see years going from 2007 to 2012. The line in the chart represents both axes, the given search volume for each date. From just this small sample size, we can see that the term has more than tripled in popularity, from a low of 29 in the beginning of 2007 up to the ceiling of 100 by the end of 2012. Bar Charts Bar charts show comparisons of data points. See Figure 1-4 for a bar chart that demonstrates the search volume by country for the keyword “Data Visualization,” the data for which is also sourced from Google Trends. Figure 1-3. Time series of weighted trend for the keyword “Data Visualization” from Google Trends Search Volume for Keyword ‘Data Visualization’ by Region from Google Trends Spain France Germany China United Kingdom Netherlands Australia Canada India United States 0 20 40 60 80 100 Figure 1-4. Google Trends breakdown of search volume by region for keyword “Data Visualization”
- 11. Chapter 1 ■Background 4 We can see the names of the countries on the y axis and the normalized search volume, from 0 to 100, on the x axis. Notice, though, that no time measure is given. Does this chart represent data for a day, a month, or a year? Also note that we have no context for what the unit of measure is. I highlight these points not to answer them but to demonstrate the limitations and pitfalls of this particular chart type. We must always be aware that our audience does not bring the same experience and context that we bring, so we must strive to make the stories in our visualizations as self evident as possible. Histograms Histograms are a type of bar chart used to show the distribution of data or how often groups of information appear in the data. See Figure 1-5 for a histogram that shows how many articles the New York Times published each year, from 1980 to 2012, that related in some way to the subject of data visualization. We can see from the chart that the subject has been ramping up in frequency since 2009. 1980 1985 1990 1995 2000 2005 2010 Year Distribution of Articles about Data Visualization by the NY Times Frequency 20 15 10 5 0 Figure 1-5. Histogram showing distribution of NY Times articles about data visualization
- 12. Chapter 1 ■Background 5 In this example, the states with the darker shades indicate a greater interest in the search term. (This data also is derived from Google Trends, for which interest is demonstrated by how frequently the term “Data Visualization” is searched for on Google.) Scatter Plots Like bar charts, scatter plots are used to compare data, but specifically to suggest correlations in the data, or where the data may be dependent or related in some way. See Figure 1-7, in which we use data from Google Correlate, (http://www.google.com/trends/correlate), to look for a relationship between search volume for the keyword “What is Data Visualization” and the keyword “How to Create Data Visualization.” Figure 1-6. Data map of U.S. states by interest in “Data Visualization” (data from Google Trends) Data Maps Data maps are used to show the distribution of information over a spatial region. Figure 1-6 shows a data map used to demonstrate the interest in the search term “Data Visualization” broken out by U.S. states.
- 13. Chapter 1 ■Background 6 This chart suggests a positive correlation in the data, meaning that as one term rises in popularity the other also rises. So what this chart suggests is that as more people find out about data visualization, more people want to learn how to create data visualizations. The important thing to remember about correlation is that it does not suggest a direct cause—correlation is not causation. History If we’re talking about the history of data visualization, the modern conception of data visualization largely started with William Playfair. William Playfair was, among other things, an engineer, an accountant, a banker, and an all-around Renaissance man who single handedly created the time series chart, the bar chart, and the bubble chart. Playfair’s charts were published in the late eighteenth century into the early nineteenth century. He was very aware that his innovations were the first of their kind, at least in the realm of communicating statistical information, and he spent a good amount of space in his books describing how to make the mental leap to seeing bars and lines as representing physical things like money. Playfair is best known for two of his books: the Commercial and Political Atlas and the Statistical Breviary. The Commercial and Political Atlas was published in 1786 and focused on different aspects of economic data from national debt, to trade figures, and even military spending. It also featured the first printed time series graph and bar chart. Figure 1-7. Scatter plot examining the correlation between search volume for terms related to “Data Visualization” , “How to Create” and “What is”
- 14. Chapter 1 ■Background 7 His Statistical Breviary focused on statistical information around the resources of the major European countries of the time and introduced the bubble chart. Playfair had several goals with his charts, among them perhaps stirring controversy, commenting on the diminishing spending power of the working class, and even demonstrating the balance of favor in the import and export figures of the British Empire, but ultimately his most wide-reaching goal was to communicate complex statistical information in an easily digested, universally understood format. Note ■ ■ Both books are back in print relatively recently, thanks to Howard Wainer, Ian Spence, and Cambridge University Press. Playfair had several contemporaries, including Dr. John Snow, who made my personal favorite chart: the cholera map. The cholera map is everything an informational graphic should be: it was simple to read; it was informative; and, most importantly, it solved a real problem. The cholera map is a data map that outlined the location of all the diagnosed cases of cholera in the outbreak of London 1854 (see Figure 1-8). The shaded areas are recorded deaths from cholera, and the shaded circles on the map are water pumps. From careful inspection, the recorded deaths seemed to radiate out from the water pump on Broad Street. Figure 1-8. John Snow’s cholera map
- 15. Chapter 1 ■Background 8 Dr. Snow had the Broad Street water pump closed, and the outbreak ended. Beautiful, concise, and logical. Another historically significant information graphic is the Diagram of the Causes of Mortality in the Army in the East, by Florence Nightingale and William Farr. This chart is shown in Figure 1-9. Figure 1-9. Florence Nightingale and William Farr’s Diagram of the Causes of Mortality in the Army in the East Nightingale and Farr created this chart in 1856 to demonstrate the relative number of preventable deaths and, at a higher level, to improve the sanitary conditions of military installations. Note that the Nightingale and Farr visualization is a stylized pie chart. Pie charts are generally a circle representing the entirety of a given data set with slices of the circle representing percentages of a whole. The usefulness of pie charts is sometimes debated because it can be argued that it is harder to discern the difference in value between angles than it is to determine the length of a bar or the placement of a line against Cartesian coordinates. Nightingale seemingly avoids this pitfall by having not just the angle of the wedge hold value but by also altering the relative size of the slices so they eschew the confines of the containing circle and represent relative value. All the above examples had specific goals or problems that they were trying to solve. Note ■ ■ A rich comprehensive history is beyond the scope of this book, but if you are interested in a thoughtful, incredibly researched analysis, be sure to read Edward Tufte’s The Visual Display of Quantitative Information. Modern Landscape Data visualization is in the midst of a modern revitalization due in large part to the proliferation of cheap storage space to store logs, and free and open source tools to analyze and chart the information in these logs.
- 16. Chapter 1 ■Background 9 From a consumption and appreciation perspective, there are websites that are dedicated to studying and talking about information graphics. There are generalized sites such as FlowingData that both aggregate and discuss data visualizations from around the web, from astrophysics timelines to mock visualizations used on the floor of Congress. The mission statement from the FlowingData About page (http://flowingdata.com/about/) is appropriately the following: “FlowingData explores how designers, statisticians, and computer scientists use data to understand ourselves better—mainly through data visualization.” There are more specialized sites such as quantifiedself.com that are focused on gathering and visualizing information about oneself. There are even web comics about data visualization, the quintessential one being xkcd.com, run by Randall Munroe. One of the most famous and topical visualizations that Randall has created thus far is the Radiation Dose Chart. We can see the Radiation Dose Chart in Figure 1-10 (it is available in high resolution here: http://xkcd.com/radiation/). Figure 1-10. Radiation Dose Chart, by Randall Munroe. Note that the range in scale being represented in this visualization as a single block in one chart is exploded to show an entirely new microcosm of context and information. This pattern is repeated over and over again to show an incredible depth of information
- 17. Chapter 1 ■Background 10 This chart was created in response to the Fukushima Daiichi nuclear disaster of 2011, and sought to clear up misinformation and misunderstanding of comparisons being made around the disaster. It did this by demonstrating the differences in scale for the amount of radiation from sources such as other people or a banana, up to what a fatal dose of radiation ultimately would be—how all that compared to spending just ten minutes near the Chernobyl meltdown. Over the last quarter of a century, Edward Tufte, author and professor emeritus at Yale University, has been working to raise the bar of information graphics. He published groundbreaking books detailing the history of data visualization, tracing its roots even further back than Playfair, to the beginnings of cartography. Among his principles is the idea to maximize the amount of information included in each graphic—both by increasing the amount of variables or data points in a chart and by eliminating the use of what he has coined chartjunk. Chartjunk, according to Tufte, is anything included in a graph that is not information, including ornamentation or thick, gaudy arrows. Tufte also invented the sparkline, a time series chart with all axes removed and only the trendline remaining to show historic variations of a data point without concern for exact context. Sparklines are intended to be small enough to place in line with a body of text, similar in size to the surrounding characters, and to show the recent or historic trend of whatever the context of the text is. Why Data Visualization? In William Playfair’s introduction to the Commercial and Political Atlas, he rationalizes that just as algebra is the abbreviated shorthand for arithmetic, so are charts a way to “abbreviate and facilitate the modes of conveying information from one person to another.” Almost 300 years later, this principle remains the same. Data visualizations are a universal way to present complex and varied amounts of information, as we saw in our opening example with the quarterly earnings report. They are also powerful ways to tell a story with data. Imagine you have your Apache logs in front of you, with thousands of lines all resembling the following: 127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET / HTTP/1.1 200 468 - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty) 127.0.0.1 - - [10/Dec/2012:10:39:11 +0300] GET /favicon.ico HTTP/1.1 200 766 - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty) Among other things, we see IP address, date, requested resource, and client user agent. Now imagine this repeated thousands of times—so many times that your eyes kind of glaze over because each line so closely resembles the ones around it that it’s hard to discern where each line ends, let alone what cumulative trends exist within. By using some analysis and visualization tools such as R, or even a commercial product such as Splunk, we can artfully pull out all kinds of meaningful and interesting stories out of this log, from how often certain HTTP errors occur and for which resources, to what our most widely used URLs are, to what the geographic distribution of our user base is. This is just our Apache access log. Imagine casting a wider net, pulling in release information, bugs and production incidents. What insights we could gather about what we do: from how our velocity impacts our defect density to how our bugs are distributed across our feature sets. And what better way to communicate those findings and tell those stories than through a universally digestible medium, like data visualizations? The point of this book is to explore how we as developers can leverage this practice and medium as part of continual improvement—both to identify and quantify our successes and opportunities for improvements, and more effectively communicate our learning and our progress. Tools There are a number of excellent tools, environments, and libraries that we can use both to analyze and visualize our data. The next two sections describe them.
- 18. Chapter 1 ■Background 11 Languages, Environments, and Libraries The tools that are most relevant to web developers are Splunk, R, and the D3 JavaScript library. See Figure 1-11 for a comparison of interest over time for them (from Google Trends). Figure 1-11. Google Trends analysis of interest over time in Splunk, R, and D3 From the figure we can see that R has had a steady consistent amount of interest since 200; Splunk had an introduction to the chart around 2005, had a spike of interest around 2006, and had steady growth since then. As for D3, we see it just start to peak around 2011 when it was introduced and its predecessor Protovis was sunsetted. Let’s start with the tool of choice for many developers, scientists, and statisticians: the R language. We have a deep dive into the R environment and language in the next chapter, but for now it’s enough to know that it is an open source environment and language used for statistical analysis and graphical display. It is powerful, fun to use, and, best of all, it is free. Splunk has seen a tremendous steady growth in interest over the last few years—and for good reason. It is easy to use once it’s set up, scales wonderfully, supports multiple concurrent users, and puts data reporting at the fingertips of everyone. You simply set it up to consume your log files; then you can go into the Splunk dashboard and run reports on key values within those logs. Splunk creates visualizations as part of its reporting capabilities, as well as alerting. While Splunk is a commercial product, it also offers a free version, available here: http://www.splunk.com/download. D3 is a JavaScript library that allows us to craft interactive visualizations. It is the official follow-up to Protovis. Protovis was a JavaScript library created in 2009 by Stanford University’s Stanford Visualization Group. Protovis was sunsetted in 2011, and the creators unveiled D3. We explore the D3 library at length in Chapter 4. Analysis Tools Aside from the previously mentioned languages and environments, there are a number of analysis tools available online. A great hosted tool for analysis and research is Google Trends. Google Trends allows you to compare trends on search terms. It provides all kinds of great statistical information around those trends, including comparing their relative search volume (see Figure 1-12), the geographic area those trends are coming from (see Figure 1-13), and related keywords.
- 19. Chapter 1 ■Background 12 Figure 1-13. Google Trends data map showing geographic location where interest in the key words is originating Figure 1-12. Google Trends for the terms “data scientist” and “computer scientist” over time; note the interest in the term “data scientist” growing rapidly from 2011 on to match the interest in the term “computer scientist”
- 20. Chapter 1 ■Background 13 Another great tool for analysis is Wolfram|Alpha (http://wolframalpha.com). See Figure 1-14 for a screenshot of the Wolfram|Alpha homepage. Figure 1-14. Home page for Wolfram|Alpha Wolfram|Alpha is not a search engine. Search engines spider and index content. Wolfram|Alpha is instead a Question Answering (QA) engine that parses human readable sentences with natural language processing and responds with computed results. Say, for example, you want to search for the speed of light. You might go to the Wolfram|Alpha site and type in “What is the speed of light?” Remember that it uses natural language processing to parse your search query, not the keyword lookup. The results of this query can be seen in Figure 1-15. Wolfram|Alpha essentially looks up all the data it has around the speed of light and presents it in a structured, categorized fashion. You can also export the raw data for each result.
- 21. Chapter 1 ■Background 14 Figure 1-15. Wolfram|Alpha results for query What is the speed of light Process Overview So we understand what data visualization is, have a high-level understanding of the history of it and an idea of the current landscape. We’re beginning to get an inkling about how we can start to use this in our world. We know some of the tools that are available to us to facilitate the analysis and creation of our charts. Now let’s look at the process involved.
- 22. Chapter 1 ■Background 15 Creating data visualizations involves four core steps: 1. Identify a problem. 2. Gather the data. 3. Analyze the data. 4. Visualize the data. Let’s walk through each step in the process and re-create one of the previous charts to demonstrate the process. Identify a Problem The very first step is to identify a problem we want to solve. This can be almost anything—from something as profound and wide-reaching as figuring out why your bug backlog doesn’t seem to go down and stay down, to seeing what feature releases over a given period in time caused the most production incidents, and why. For our example, let’s re-create Figure 1-5 and try to quantify the interest in data visualization over time as represented by the number of New York Times articles on the subject. Gather Data We have an idea of what we want to investigate, so let’s dig in. If you are trying to solve a problem or tell a story around your own product, you would of course start with your own data—maybe your Apache logs, maybe your bug backlog, maybe exports from your project tracking software. Note ■ ■ If you are focusing on gathering metrics around your product and you don’t already have data handy, you need to invest in instrumentation.There are many ways to do this, usually by putting logging in your code.At the very least, you want to log error states and monitor those, but you may want to expand the scope of what you track to include for debugging purposes while still respecting both your user’s privacy and your company’s privacy policy. In my book, Pro JavaScript Performance: Monitoring and Visualization, I explore ways to track and visualize web and runtime performance. One important aspect of data gathering is deciding which format your data should be in (if you're lucky) or discovering which format your data is available in. We’ll next be looking at some of the common data formats in use today. JSON is an acronym that stands for JavaScript Object Notation. As you probably know, it is essentially a way to send data as serialized JavaScript objects. We format JSON as follows: [object]{ [attribute]: [value], [method] : function(){}, [array]: [item, item] } Another way to transfer data is in XML format. XML has an expected syntax, in which elements can have attributes, which have values, values are always in quotes, and every element must have a closing element. XML looks like this: parent attribute=value child attribute=valuenode data/child /parent Generally we can expect APIs to return XML or JSON to us, and our preference is usually JSON because as we can see it is a much more lightweight option just in sheer amount of characters used.
- 23. Chapter 1 ■Background 16 But if we are exporting data from an application, it most likely will be in the form of a comma separated value file, or CSV. A CSV is exactly what it sounds like: values separated by commas or some other sort of delimiter: value1,value2,value3 value4,value5,value6 For our example, we’ll use the New York Times API Tool, available at http://prototype.nytimes.com/gst/ apitool/index.html. The API Tool exposes all the APIs that the New York Times makes available, including the Article Search API, the Campaign Finance API, and the Movie Review API. All we need to do is select the Article Search API from the drop-down menu, type in our search query or the phrase that we want to search for, and click “Make Request” . This queries the API and returns the data to us, formatted as JSON. We can see the results in Figure 1-16. Figure 1-16. The NY Times API Tool We can then copy and paste the returned JSON data to our own file or we could go the extra step to get an API key so that we can query the API from our own applications. For the sake of our example, we will save the JSON data to a file that we will name jsNYTimesData. The contents of the file will be structured like so: { offset: 0, results: [ { body: BODY COPY,
- 24. Chapter 1 ■Background 17 byline: By AUTHOR, date: 20121011, title: TITLE, url: http://www.nytimes.com/foo.html }, { body: BODY COPY, byline: By AUTHOR, date: 20121021, title: TITLE, url: http://www.nytimes.com/bar.html } ], tokens: [ JavaScript ], total: 2 } Looking at the high-level JSON structure, we see an attribute named offset, an array named results, an array named tokens, and another attribute named total. The offset variable is for pagination (what page full of results we are starting with). The total variable is just what it sounds like: the number of results that are returned for our query. It’s the results array that we really care about; it is an array of objects, each of which corresponds to an article. The article objects have attributes named body, byline, date, title, and url. We now have data that we can begin to look at. That takes us to our next step in the process, analyzing our data. DATA SCRUBBING There is often a hidden step here, one that anyone who’s dealt with data knows about: scrubbing the data. Often the data is either not formatted exactly as we need it or, in even worse cases, it is dirty or incomplete. In the best-case scenario in which your data just needs to be reformatted or even concatenated, go ahead and do that, but be sure to not lose the integrity of the data. Dirty data has fields out of order, fields with obviously bad information in them—think strings in ZIP codes—or gaps in the data. If your data is dirty, you have several choices: You could drop the rows in question, but that can harm the integrity of the data—a good example • is if you are creating a histogram removing rows could change the distribution and change what your results will be. The better alternative is to reach out to whoever administers the source of your data and try and • get a better version if it exists. Whatever the case, if data is dirty or it just needs to be reformatted to be able to be imported into R, expect to have to scrub your data at some point before you begin your analysis. Analyze Data Having data is great, but what does it mean? We determine it through analysis. Analysis is the most crucial piece of creating data visualizations. It’s only through analysis that we can understand our data, and it is only through understanding it that we can craft our story to share with others.
- 25. Chapter 1 ■Background 18 To begin analysis, let’s import our data into R. Don’t worry if you aren’t completely fluent in R; we do a deep dive into the language in the next chapter. If you aren’t familiar with R yet, don’t worry about coding along with the following examples: just follow along to get an idea of what is happening and return to these examples after reading Chapters 3 and 4. Because our data is JSON, let’s use an R package called rjson. This will allow us to read in and parse JSON with the fromJSON() function: library(rjson) json_data - fromJSON(paste(readLines(jsNYTimesData.txt), collapse=)) This is great, except the data is read in as pure text, including the date information. We can’t extract information from text because obviously text has no contextual meaning outside of being raw characters. So we need to iterate through the data and parse it to more meaningful types. Let's create a data frame (an array-like data type specific to R that we talk about next chapter), loop through our json_data object; and parse year, month, and day parts out of the date attribute. Let’s also parse the author name out of the byline, and check to make sure that if the author’s name isn’t present we substitute the empty value with the string “unknown”. df - data.frame() for(n in json_data$results){ year -substr(n$date, 0, 4) month - substr(n$date, 5, 6) day - substr(n$date, 7, 8) author - substr(n$byline, 4, 30) title - n$title if(length(author) 1){ author - unknown } Next, we can reassemble the date into a MM/DD/YYYY formatted string and convert it to a date object: datestamp -paste(month, /, day, /, year, sep=) datestamp - as.Date(datestamp,%m/%d/%Y) And finally before we leave the loop, we should add this newly parsed author and date information to a temporary row and add that row to our new data frame. newrow - data.frame(datestamp, author, title, stringsAsFactors=FALSE, check.rows=FALSE) df - rbind(df, newrow) } rownames(df) - df$datestamp Our complete loop should look like the following: df - data.frame() for(n in json_data$results){ year -substr(n$date, 0, 4) month - substr(n$date, 5, 6) day - substr(n$date, 7, 8) author - substr(n$byline, 4, 30) title - n$title
- 26. Chapter 1 ■Background 19 if(length(author) 1){ author - unknown } datestamp -paste(month, /, day, /, year, sep=) datestamp - as.Date(datestamp,%m/%d/%Y) newrow - data.frame(datestamp, author, title, stringsAsFactors=FALSE, check.rows=FALSE) df - rbind(df, newrow) } rownames(df) - df$datestamp Note that our example assumes that the data set returned has unique date values. If you get errors with this, you may need to scrub your returned data set to purge any duplicate rows. Once our data frame is populated, we can start to do some analysis on the data. Let’s start out by pulling just the year from every entry, and quickly making a stem and leaf plot to see the shape of the data. Note ■ ■ John Tukey created the stem and leaf plot in his seminal work, Exploratory Data Analysis. Stem and leaf plots are quick, high-level ways to see the shape of data, much like a histogram. In the stem and leaf plot, we construct the “stem” column on the left and the “leaf” column on the right. The stem consists of the most significant unique elements in a result set. The leaf consists of the remainder of the values associated with each stem. In our stem and leaf plot below, the years are our stem and R shows zeroes for each row associated with a given year. Something else to note is that often alternating sequential rows are combined into a single row, in the interest of having a more concise visualization. First, we will create a new variable to hold the year information: yearlist - as.POSIXlt(df$datestamp)$year+1900 If we inspect this variable, we see that it looks something like this: yearlist [1] 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2012 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 2011 [30] 2011 2011 2011 2011 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2009 2009 2009 2009 2009 2009 2009 2008 2008 2008 2007 2007 2007 2007 2006 [59] 2006 2006 2006 2005 2005 2005 2005 2005 2005 2004 2003 2003 2003 2002 2002 2002 2002 2001 2001 2000 2000 2000 2000 2000 2000 1999 1999 1999 1999 [88] 1999 1999 1998 1998 1998 1997 1997 1996 1996 1995 1995 1995 1993 1993 1993 1993 1992 1991 1991 1991 1990 1990 1990 1990 1989 1989 1989 1988 1988 [117] 1988 1986 1985 1985 1985 1984 1982 1982 1981 That’s great, that’s exactly what we want: a year to represent every article returned. Next let’s create the stem and leaf plot: stem(yearlist) 1980 | 0 1982 | 00 1984 | 0000 1986 | 0 1988 | 000000
- 27. Chapter 1 ■Background 20 1990 | 0000000 1992 | 00000 1994 | 000 1996 | 0000 1998 | 000000000 2000 | 00000000 2002 | 0000000 2004 | 0000000 2006 | 00000000 2008 | 0000000000 2010 | 000000000000000000000000000000 2012 | 0000000000000 Very interesting. We see a gradual build with some dips in the mid-1990s, another gradual build with another dip in the mid-2000s and a strong explosion since 2010 (the stem and leaf plot groups years together in twos). Looking at that, my mind starts to envision a story building about a subject growing in popularity. But what about the authors of these articles? Maybe they are the result of one or two very interested authors that have quite a bit to say on the subject. Let’s explore that idea and take a look at the author data that we parsed out. Let’s look at just the unique authors from our data frame: length(unique(df$author)) [1] 81 We see that there are 81 unique authors or combination of authors for these articles! Just out of curiosity, let’s take a look at the breakdown by author for each article. Let’s quickly create a bar chart to see the overall shape of the data (the bar chart is shown in Figure 1-17): plot(table(df$author), axes=FALSE) Figure 1-17. Bar chart of number of articles by author to quickly visualize
- 28. Chapter 1 ■Background 21 We remove the x and y axes to allow ourselves to focus just on the shape of the data without worrying too much about the granular details. From the shape, we can see a large number of bars with the same value; these are authors who have written a single article. The higher bars are authors who have written multiple articles. Essentially each bar is a unique author, and the height of the bar indicates the number of articles they have written. We can see that although there are roughly five standout contributors, most authors have average one article. Note that we just created several visualizations as part of our analysis. The two steps aren’t mutually exclusive; we often times create quick visualizations to facilitate our own understanding of the data. It’s the intention with which they are created that make them part of the analysis phase. These visualizations are intended to improve our own understanding of the data so that we can accurately tell the story in the data. What we’ve seen in this particular data set tells a story of a subject growing in popularity, demonstrated by the increasing number of articles by a variety of authors. Let’s now prepare it for mass consumption. Note ■ ■ We are not fabricating or inventing this story. Like information archaeologists, we are sifting through the raw data to uncover the story. Visualize Data Once we’ve analyzed the data and understand it (and I mean really understand the data to the point where we are conversant in all the granular details around it), and once we’ve seen the story that the data has within, it is time to share that story. For the current example, we’ve already crafted a stem and leaf plot as well as a bar chart as part of our analysis. However, stem and leaf plots are great for analyzing data, but not so great for messaging out about the findings. It is not immediately obvious what the context of the numbers in a stem and leaf plot represents. And the bar chart we created supported the main thesis of the story instead of communicating that thesis. Since we want to demonstrate the distribution of articles by year, let’s instead use a histogram to tell the story: hist(yearlist) See Figure 1-18 for what this call to the hist() function generates.
- 29. Chapter 1 ■Background 22 This is a good start, but let’s refine this further. Let’s color in the bars, give the chart a meaningful title, and strictly define the range of years. hist(yearlist, breaks=(1981:2012), freq=TRUE, col=#CCCCCC, main=Distribution of Articles about Data Visualizationnby the NY Times, xlab = Year) This produces the histogram that we see in Figure 1-5. Ethics of Data Visualization Remember Figure 1-3 from the beginning of this chapter where we looked at the weighted popularity of the search term “Data Visualization”? By constraining the data to 2006 to 2012, we told a story of a keyword growing in popularity, almost doubling in popularity over a six-year period. But what if we included more data points in our sample and extended our view to include 2004? See Figure 1-19 for this expanded time series chart. 1980 1985 1990 1995 2000 2005 2010 2015 yearlist Histogram of yearlist Frequency 30 25 20 15 10 5 0 Figure 1-18. Histogram of yearlist
- 30. Chapter 1 ■Background 23 This expanded chart tells a different story: one that describes a dip in popularity between 2005 and 2009. This expanded chart also demonstrates how easy it is to misrepresent the truth intentionally or unintentionally with data visualizations. Cite Sources When Playfair first published his Commercial and Political Atlas, one of the biggest biases he had to battle was the inherent distrust his peers had of charts to accurately represent data. He tried to overcome this by including data tables in the first two editions of the book. Similarly, we should always include our sources when distributing our charts so that our audience can go back and independently verify the data if they want to. This is important because we are trying to share information, not hoard it, and we should encourage others to inspect the data for themselves and be excited about the results. Be Aware of Visual Cues A side effect of using charts to function as visual shorthand is that we bring our own perspective and context to play when we view charts. We are used to certain things, such as the color red being used to signify danger or flagging for attention, or the color green signifying safety. These color connotations are part of a branch of color theory called color harmony, and it’s worth at least being aware of what your color choices could be implying. When in doubt, get a second opinion. When creating our graphics, we can often get married to a certain layout or chart choice. This is natural because we have spent time invested in analyzing and crafting the chart. A fresh, objective set of eyes should point out unintentional meanings or overly complex designs, and make for a more crisp visualization. Summary This chapter took a look at some introductory concepts about data visualization, from conducting data gathering and exploration, to looking at the charts that make up the visual patterns that define how we communicate with data. We looked a little at the history of data visualization, from the early beginnings with William Playfair and Florence Nightingale to modern examples such as xkcd.com. While we saw a little bit of code in this chapter, in the next chapter we start to dig in to the tactics of learning R and getting our hands dirty reading in data, shaping data, and crafting our own visualizations. Figure 1-19. Google Trends time series chart with expanded time range. Note that the additional data points give a greater context and tell a different story
- 31. 25 Chapter 2 R LanguagePrimer In the last chapter, we defined what data visualizations are, looked at a little bit of the history of the medium, and explored the process for creating them. This chapter takes a deeper dive into one of the most important tools for creating data visualizations: R. When creating data visualizations, R is an integral tool for both analyzing data and creating visualizations. We will use R extensively through the rest of this book, so we had better level set first. R is both an environment and a language to run statistical computations and produce data graphics. It was created by Ross Ihaka and Robert Gentleman in 1993 while at University of Auckland. The R environment is the runtime environment that you develop and run R in. The R language is the programming language that you develop in. R is the successor to the S language, a statistical programming language that came out of Bell Labs in 1976. Getting to Know the R Console Let’s start by downloading and installing R. R is available from the R Foundation at http://www.r-project.org/. See Figure 2-1 for a screenshot of the R Foundation homepage.
- 32. Chapter 2 ■R Language Primer 26 It is available as a precompiled binary from the Comprehensive R Archive Network (CRAN) website: http://cran.r-project.org/ (see Figure 2-2). We just select our operating system and what version of R we want, and we can begin to download. Figure 2-1. Homepage of the R Foundation
- 33. Chapter 2 ■R Language Primer 27 Once the download is complete, we can run through the installer. See Figure 2-3 for a screenshot of the R installer for the Mac OS. Figure 2-2. The CRAN website
- 34. Chapter 2 ■R Language Primer 28 Once we finish the installation we can launch the R application, and we are presented with the R console, as shown in Figure 2-4. Figure 2-3. R installation on a Mac Figure 2-4. The R console
- 35. Chapter 2 ■R Language Primer 29 The Command Line The R console is where the magic happens! It is a command-line environment where we can run R expressions. The best way to get up to speed in R is to script in the console, a piece at a time, generally to try out what you’re trying to do, and tweak it until you get the results that you want. When you finally have a working example, take the code that does what you want and save it as an R script file. R script files are just files that contain pure R and can be run in the console using the source command: source(someRfile.R) Looking at the preceding code snippet, we assume that the R script lives in the current work directory. The way we can see what the current work directory is to use the getwd() function: getwd() [1] /Users/tomjbarker We can also set the working directory by using the setwd() function. Note that changes made to the working directory are not persisted across R sessions unless the session is saved. setwd(/Users/tomjbarker/Downloads) getwd() [1] /Users/tomjbarker/Downloads Command History The R console stores commands that you enter and you can cycle through previous commands by pressing the up arrow. Hit the escape button to return to the command prompt. We can see the history in a separate window pane by clicking the Show/Hide Command History button at the top of the console. The Show/Hide Command History button is the rectangle icon with alternating stripes of yellow and green. See Figure 2-5 for the R console with the command history shown.
- 36. Chapter 2 ■R Language Primer 30 Accessing Documentation To read the R documentation around a specific function or keyword, you simply type a question mark before the keyword: ?setwd If you want to search the documentation for a specific word or phrase, you can type two question marks before the search query: ??working directory This code launches a window that shows search results (see Figure 2-6). The search result window has a row for each topic that contains the search phrase and has the name of the help topic, the package that the functionality that the help topic talks about is in, and a short description for the help topic. Figure 2-5. R console with command history shown
- 37. Chapter 2 ■R Language Primer 31 Packages Speaking of packages, what are they, exactly? Packages are collections of functions, data sets, or objects that can be imported into the current session or workspace to extend what we can do in R. Anyone can make a package and distribute it. To install a package, we simply type this: install.packages([package name]) For example, if we want to install the ggplot2 package—which is a widely used and very handy charting package—we simply type this into the console: install.packages(ggplot2) We are immediately prompted to choose the mirror location that we want to use, usually the one closest to our current location. From there, the install begins. We can see the results in Figure 2-7. Figure 2-6. Help search results window
- 38. Chapter 2 ■R Language Primer 32 The zipped-up package is downloaded and exploded into our R installation. If want to use a package that we have installed, we must first include it in our workspace. To do this we use the library() function: library(ggplot2) A list of packages available at the CRAN can be found here: http://cran.r-project.org/web/packages/ available_packages_by_name.html. To see a list of packages already installed, we can simply call the library() function with no parameter (depending on your install and your environment, your list of packages may vary): library() Packages in library '/Library/Frameworks/R.framework/Versions/2.15/Resources/library': barcode Barcode distribution plots base The R Base Package boot Bootstrap Functions (originally by Angelo Canty for S) class Functions for Classification cluster Cluster Analysis Extended Rousseeuw et al. Figure 2-7. Installing the ggplot2 package
- 39. Chapter 2 ■R Language Primer 33 codetools Code Analysis Tools for R colorspace Color Space Manipulation compiler The R Compiler Package datasets The R Datasets Package dichromat Color schemes for dichromats digest Create cryptographic hash digests of R objects foreign Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, dBase, ... ggplot2 An implementation of the Grammar of Graphics gpairs gpairs: The Generalized Pairs Plot graphics The R Graphics Package grDevices The R Graphics Devices and Support for Colours and Fonts grid The Grid Graphics Package gtable Arrange grobs in tables. KernSmooth Functions for kernel smoothing for Wand Jones (1995) labeling Axis Labeling lattice Lattice Graphics mapdata Extra Map Databases mapproj Map Projections maps Draw Geographical Maps Importing Data So now our environment is downloaded and installed, and we know how to install any packages that we may need. Now we can begin using R. The first thing we’ll normally want to do is import your data. There are several ways to import data, but the most common way is to use the read() function, which has several flavors: read.table([file to read]) read.csv([file to read]) To see this in action, let’s first create a text file named temptext.txt that is formatted like so: 134,432,435,313,11 403,200,500,404,33 77,321,90,2002,395 We can read this into a variable that we will name temptxt: temptxt - read.table(temptext.txt) Notice that as we are assigning value to this variable, we are not using an equal sign as the assignment operator. We are instead using an arrow -. That is R’s assignment operator, although it does also support the equal sign if you are so inclined. But the standard is the arrow, and all examples that we will show in this book will use the arrow. If we print out the temptxt variable, we see that it is structured as follows: temptxt V1 1 134,432,435,313,11 2 403,200,500,404,33 3 77,321,90,2002,395
- 40. Chapter 2 ■R Language Primer 34 We see that our variable is a table-like structure called a data frame, and R has assigned a column name (V1) and row IDs to our data structure. More on column names soon. The read() function has a number of parameters that you can use to refine how the data is imported and formatted once it is imported. Using Headers The header parameter tells R to treat the first line in the external file as containing header information. The first line then becomes the column names of the data frame. For example, suppose we have a log file structured like this: url, day, date, loadtime, bytes, httprequests, loadtime_repeatview http://apress.com, Sun, 01 Jul 2012 14:01:28 +0000,7042,956680,73,3341 http://apress.com, Sun, 01 Jul 2012 14:01:31 +0000,6932,892902,76,3428 http://apress.com, Sun, 01 Jul 2012 14:01:33 +0000,4157,594908,38,1614 We can load it into a variable named wpo like so: wpo - read.table(wpo.txt, header=TRUE) wpo url day date loadtime bytes httprequests loadtime_repeatview 1 http://apress.com,Sun,1 Jul 2012 14:01:28 +0000,7042,955550,73,3191 2 http://apress.com,Sun,1 Jul 2012 14:01:31 +0000,6932,892442,76,3728 3 http://apress.com,Sun,1 Jul 2012 14:01:33 +0000,4157,614908,38,1514 When we call the colnames() function to see what the column names are for wpo, we see the following: colnames(wpo) [1] url day date loadtime [5] bytes httprequests loadtime_repeatview Specifying a String Delimiter The sep attribute tells the read() function what to use as the string delimiter for parsing the columns in the external data file. In all the examples we’ve looked at so far, commas are our delimiters, but we could use instead pipes | or any other character that we want. Say, for example, that our previous temptxt example used pipes; we would just update the code to be as follows: 134|432|435|313|11 403|200|500|404|33 77|321|90|2002|395 temptxt - read.table(temptext.txt, sep=|) temptxt V1 V2 V3 V4 V5 1 134 432 435 313 11 2 403 200 500 404 33 3 77 321 90 2002 395 Oh, notice that? We actually got distinct column names this time (V1, V2, V3, V4, V5). Before, we didn’t specify a delimiter, so R assumed that each row was one big blob of text and lumped it into a single column (V1).
- 41. Chapter 2 ■R Language Primer 35 Specifying Row Identifiers The row.names attribute allows us to specify identifiers for our rows. By default, as we’ve seen in the previous examples, R uses incrementing numbers as row IDs. Keep in mind that the row names need to be unique for each row. With that in mind, let’s take a look at importing some different log data, which has performance metrics for unique URLs: url, day, date, loadtime, bytes, httprequests, loadtime_repeatview http://apress.com, Sun, 01 Jul 2012 14:01:28 +0000,7042,956680,73,3341 http://google.com, Sun, 01 Jul 2012 14:01:31 +0000,6932,892902,76,3428 http://apple.com, Sun, 01 Jul 2012 14:01:33 +0000,4157,594908,38,1614 When we read it in, we’ll be sure to specify that the data in the url column should be used as the row name for the data frame. wpo - read.table(wpo.txt, header=TRUE, sep=,, row.names=url) wpo day date loadtime bytes httprequests loadtime_repeatview http://apress.com Sun 01 Jul 2012 14:01:28 +0000 7042 956680 73 3341 http://google.com Sun 01 Jul 2012 14:01:31 +0000 6932 892902 76 3428 http://apple.com Sun 01 Jul 2012 14:01:33 +0000 4157 594908 38 1614 Using Custom Column Names And there we go. But what if we want to have column names, but the first line in our file is not header information? We can use the col.names parameter to specify a vector that we can use as column names. Let’s take a look. In this example, we’ll use the pipe separated text file used previously. 134|432|435|313|11 403|200|500|404|33 77|321|90|2002|395 First, we’ll create a vector named columnNames that will hold the strings that we will use as the column names: columnNames - c(resource_id, dns_lookup, cache_load, file_size, server_response) Then we’ll read in the data, passing in our vector to the col.names parameter. resource_log - read.table(temptext.txt, sep=|, col.names=columnNames) resource_log resource_id dns_lookup cache_load file_size server_response 1 134 432 435 313 11 2 403 200 500 404 33 3 77 321 90 2002 395 Data Structures and Data Types In the previous examples, we touched on a lot of concepts; we created variables, including vectors and data frames; but we didn’t talk much about what they are. Let’s take a step back and look at the data types that R supports and how to use them.
- 42. Chapter 2 ■R Language Primer 36 Data types in R are called modes, and can be the following: numeric • character • logical • complex • raw • list • We can use the mode() function to check the mode of a variable. Character and numeric modes correspond to string and number (both integer and float) data types. Logical modes are Boolean values. n - 122132 mode(n) [1] numeric c - test text mode(c) [1] character l - TRUE mode(l) [1] logical We can perform string concatenation using the paste() function. We can use the substr() function to pull characters out of strings. Let’s look at some examples in code. Usually, I keep a list of directories that I either read data from or write charts to. Then when I want to reference a new data file that exists in the data directory, I will just append the new file name to the data directory: dataDirectory - /Users/tomjbarker/org/data/ buglist - paste(dataDirectory, bugs.txt, sep=) buglist [1] /Users/tomjbarker/org/data/bugs.txt The paste() function takes N amount of strings and concatenates them together. It accepts an argument named sep that allows us to specify a string that we can use to be a delimiter between joined strings. We don’t want anything separating our joined strings that we pass in an empty string. If we want to pull characters from a string, we use the substr() function. The substr() function takes a string to parse, a starting location, and a stopping location. It returns all the character inclusively from the starting location up to the ending location. (Remember that in R, lists are not 0-based like most other languages, but instead have a starting index of 1.) substr(test, 1,2) [1] te In the preceding example, we pass in the string “test” and tell the substr() function to return the first and second characters. Complex mode is for complex numbers. The raw mode is to store raw byte data.
- 43. Chapter 2 ■R Language Primer 37 List data types or modes can be one of three classes: vectors, matrices, or data frames. If we call mode() for vectors or matrices, they return the mode of the data that they contain; class() returns the class. If we call mode() on a data frame, it returns the type list: v - c(1:10) mode(v) [1] numeric m - matrix(c(1:10), byrow=TRUE) mode(m) [1] numeric class(m) [1] matrix d - data.frame(c(1:10)) mode(d) [1] list class(d) [1] data.frame Note that we just typed 1:10 rather than the whole sequence of numbers between 1 and 10: v - c(1:10) Vectors are single-dimensional arrays that can hold only values of a single mode at a time. It’s when we get to data frames and matrices that R really starts to get interesting. The next two sections cover those classes. Data Frames We saw at the beginning of this chapter that the read() function takes in external data and saves it as a data frame. Data frames are like arrays in most other loosely typed languages: they are containers that hold different types of data, referenced by index. The main thing to realize, though, is that data frames see the data that they contain as rows, columns, and combinations of the two. For example, think of a data frame as formatted as follows: col col col col col row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] row [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] If we try to reference the first index in the preceding data frame as we traditionally would with an array, say dataframe[1], R would instead return the first column of data, not the first item. So data frames are referenced by their column and row. So dataframe[1] returns the first column and dataframe[,2] returns the first row. Let’s demonstrate this in code. First let’s create some vectors using the combine function, c(). Remember that vectors are collections of data all of the same type. The combine function takes a series of values and combines them into vectors. col1 - c(1,2,3,4,5,6,7,8) col2 - c(1,2,3,4,5,6,7,8) col3 - c(1,2,3,4,5,6,7,8) col4 - c(1,2,3,4,5,6,7,8)
- 44. Chapter 2 ■R Language Primer 38 Then let’s combine these vectors into a data frame: df - data.frame(col1,col2,col3,col4) Now let’s print the data frame to see the contents and the structure of it: df col1 col2 col3 col4 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 6 6 6 6 6 7 7 7 7 7 8 8 8 8 8 Notice that it took each vector and made each one a column. Also notice that each row has an ID; by default, it is a number, but we can override that. If we reference the first index, we see that the data frame returns the first column: df[1] col1 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 If we put a comma in front of that 1, we reference the first row: df[,1] [1] 1 2 3 4 5 6 7 8 So accessing contents of a data frame is done by specifying [column, row]. Matrices work much the same way. Matrices Matrices are just like data frames in that they contain rows and columns and can be referenced by either. The core difference between the two is that data frames can hold different data types but matrices can hold only one type of data. This presents a philosophical difference. Usually you use data frames to hold data read in externally, like from a flat file or a database because those are generally of mixed type. You normally store data in matrices that you want to apply functions to (more on applying functions to lists in a little bit).
- 45. Chapter 2 ■R Language Primer 39 To create a matrix, we must use the matrix() function, pass in a vector, and tell the function how to distribute the vector: The • nrow parameter specifies how many rows the matrix should have The • ncol parameter specifies the number of columns. The • byrow parameter tells R that the contents of the vector should be distributed by iterating across rows if TRUE or by columns if FALSE. content - c(1,2,3,4,5,6,7,8,9,10) m1 - matrix(content, nrow=2, ncol=5, byrow=TRUE) m1 [,1] [,2] [,3] [,4] [,5] [1,] 1 2 3 4 5 [2,] 6 7 8 9 10 Notice that in the previous example that the m1 matrix is filled in horizontally, row by row. In the following example, the m1 matrix is filled in vertically by column: content - c(1,2,3,4,5,6,7,8,9,10) m1 - matrix(content, nrow=2, ncol=5, byrow=FALSE) m1 [,1] [,2] [,3] [,4] [,5] [1,] 1 3 5 7 9 [2,] 2 4 6 8 10 Remember that instead of manually typing out all the numbers in the previous content vector, if the numbers are a sequence we can just type this: content - (1:10) We reference the content in matrices with the square bracket, specifying the row and column, respectively. m1[1,4] [1] 7 We can convert a data frame to a matrix if the data frame contains only a single type of data. To do this we use the as.matrix() function. Often times we will do this when passing a data frame to a plotting function to draw a chart. barplot(as.matrix(df)) Below we create a data frame called df. We populate the data frame with ten consecutive numbers. We then use as.matrix() to convert df into a matrix and save the result into a new variable called m: df - data.frame(1:10) df X1.10 1 1 2 2 3 3
- 46. Chapter 2 ■R Language Primer 40 4 4 5 5 6 6 7 7 8 8 9 9 10 10 class(df) [1] data.frame m - as.matrix(df) class(m) [1] matrix Keep in mind that because they are all the same data type, matrices require less overhead and are intrinsically more efficient than data frames. If we compare the size of our matrix m and our data frame df, we see that with just ten items there is a size difference. object.size(m) 312 bytes object.size(df) 440 bytes With that said, if we increase the scale of this, the increase in efficiency does not equally scale. Compare the following: big_df - data.frame(1:40000000) big_m - matrix(1:40000000) object.size(big_m) 160000112 bytes object.size(big_df) 160000400 bytes We can see that the first example with the small data set showed that the matrix was 30 percent smaller in size than the data frame, but at the larger scale in the second example the matrix was only .00018 percent smaller than the data frame. Adding Lists When combining or adding to data frames or matrices, you generally add either by the row or the column using rbind() or cbind(). To demonstrate this, let’s add a new row to our data frame df. We’ll pass df into rbind() along with the new row to add to df. The new row contains just one element, the number 11: df - rbind(df, 11) df X1.10 1 1 2 2 3 3 4 4 5 5 6 6
- 47. Chapter 2 ■R Language Primer 41 7 7 8 8 9 9 10 10 11 11 Now let’s add a new column to our matrix m. To do this, we simply pass m into cbind() as the first parameter; the second parameter is a new matrix that will be appended to the new column. m - rbind(m, 11) m - cbind(m, matrix(c(50:60), byrow=FALSE)) m X1.10 [1,] 1 50 [2,] 2 51 [3,] 3 52 [4,] 4 53 [5,] 5 54 [6,] 6 55 [7,] 7 56 [8,] 8 57 [9,] 9 58 [10,] 10 59 [11,] 11 60 What about vectors, you may ask? Well, let’s look at adding to our content vector. We simply use the combine function to combine the current vector with a new vector: content - c(1,2,3,4,5,6,7,8,9,10) content - c(content, c(11:20)) content [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Looping Through Lists As developers who generally work in procedural languages, or at least came up the ranks using procedural languages (though in recent years functional programming paradigms have become much more mainstream), we’re most likely used to looping through our arrays when we want to process the data within them. This is in contrast to purely functional languages where we would instead apply a function to our lists, like the map() function. R supports both paradigms. Let’s first look at how to loop through our lists. The most useful loop that R supports is the for in loop. The basic structure of a for in loop can be seen here:. for(i in 1:5){print(i)} [1] 1 [1] 2 [1] 3 [1] 4 [1] 5
- 48. Chapter 2 ■R Language Primer 42 The variable i increments in value each step through the iteration. We can use the for in loop to step through lists. We can specify a particular column to iterate through, like the following, in which we loop through the X1.10 column of the data frame df. for(n in df$X1.10){ print(n)} [1] 1 [1] 2 [1] 3 [1] 4 [1] 5 [1] 6 [1] 7 [1] 8 [1] 9 [1] 10 [1] 11 Note that we are accessing the columns of data frames via the dollar sign operator. The general pattern is [data frame]$[column name]. Applying Functions to Lists But the way that R really wants to be used is to apply functions to the contents of lists (see Figure 2-8). function element element element element Figure 2-8. Apply a function to list elements We do this in R with the apply() function.
- 49. Chapter 2 ■R Language Primer 43 The apply() function takes several parameters: First is our list. • Next a number vector to indicate how we apply the function through the list ( • 1 is for rows, 2 is for columns, and c[1,2] indicates both rows and columns). Finally is the function to apply to the list: • apply([list], [how to apply function], [function to apply]) Let’s look at an example. Let’s make a new matrix that we’ll call m. The matrix m will have ten columns and four rows: m - matrix(c(1:40), byrow=FALSE, ncol=10) m [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [1,] 1 5 9 13 17 21 25 29 33 37 [2,] 2 6 10 14 18 22 26 30 34 38 [3,] 3 7 11 15 19 23 27 31 35 39 [4,] 4 8 12 16 20 24 28 32 36 40 Now say we wanted to increment every number in the m matrix. We could simply use apply() as follows: apply(m, 2, function(x) x - x + 1) [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [1,] 2 6 10 14 18 22 26 30 34 38 [2,] 3 7 11 15 19 23 27 31 35 39 [3,] 4 8 12 16 20 24 28 32 36 40 [4,] 5 9 13 17 21 25 29 33 37 41 Do you see what we did there? We passed in m, we specified that we wanted to apply the function across the columns, and finally we passed in an anonymous function. The function accepts a parameter that we called x. The parameter x is a reference to the current matrix element. From there, we just increment the value of x by 1. OK, say we wanted to do something slightly more interesting, such as zeroing out all the even numbers in the matrix. We could do the following: apply(m,c(1,2),function(x){if((x %% 2) == 0) x - 0 else x - x}) [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [1,] 1 5 9 13 17 21 25 29 33 37 [2,] 0 0 0 0 0 0 0 0 0 0 [3,] 3 7 11 15 19 23 27 31 35 39 [4,] 0 0 0 0 0 0 0 0 0 0 For the sake of clarity let’s break out that function that we are applying. We simply check to see whether the current element is even by checking to see whether it has a remainder when divided by two. If it is, we set it to zero; if it isn’t, we set it to itself: function(x){ if((x %% 2) == 0) x - 0 else x - x }
- 50. Chapter 2 ■R Language Primer 44 Functions Speaking of functions, the syntax for creating functions in R is much like most other languages. We use the function keyword, give the function a name, have open and closed parentheses where we specify arguments, and wrap the body of the function in curly braces: function [function name]([argument]) { [body of function] } Something interesting that R allows is the ... argument (sometimes called the dots argument). This allows us to pass in a variable number of parameters into a function. Within the function, we can convert the ... argument into a list and iterate over the list to retrieve the values within: offset - function (...){ for(i in list(...)){ print(i) } } offset(23,11) [1] 23 [1] 11 We can even store values of different data types (modes) in the ... argument: offset(test value, 12, 100, 19ANM) [1] test value [1] 12 [1] 100 [1] 19ANM R uses lexical scoping. This means that when we call a function and try to reference variables that are not defined inside the local scope of the function, the R interpreter looks for those variables in the workspace or scope in which the function was created. If the R interpreter cannot find those variables in that scope, it looks in the parent of that scope. If we create a function A within function B, the creation scope of function A is function B. For example, see the following code snippet: x - 10 wrapper - function(y){ x - 99 c- function(y){ print(x + y) } return(c) } t - wrapper() t(1) [1] 100 x [1] 10
- 51. Chapter 2 ■R Language Primer 45 We created a variable x in the global space and gave it a value of 10. We created a function, named it wrapper, and had it accept an argument named y. Within the wrapper() function, we created another variable named x and gave it a value of 99. We also created a function named c. The function wrapper() passes the argument y into the function c(), and the c() function outputs the value of x added to y. Finally, the wrapper() function returns the c() function. We created a variable t and set it to the returned value of the wrapper() function, which is the function c(). When we run the t() function and pass in a value of 1, we see that it outputs 100 because it is referencing the variable x from the function wrapper(). Being able to reach into the scope of a function that has executed is called a closure. But, you may ask, how can we be sure that we are executing the returned function and not re-running wrapper() each time? R has a very nice feature where if you type in the name of a function without the parentheses, the interpreter will output the body of the function. When we do this, we are in fact referencing the returned function and using a closure to reference the x variable: t function(y){ print(x + y) } environment: 0x17f1d4c4 Summary In this chapter, we downloaded and installed R. We explored the command line, went over data types, and got up and running importing into the R environment data for analysis. We looked at lists, how to create them, add to them, loop through them, and to apply functions to elements in a list. We looked at functions, talked about lexical scope, and saw how to create closures in R. Next chapter we’ll take a deeper dive into R, look at objects, get our feet wet with statistical analysis in R, and explore creating R markdown documents for distribution over the web.
- 52. 47 Chapter 3 A DeeperDive into R The last chapter explored some introductory concepts in R, from using the console to importing data. We installed packages and discussed data types, including different list types. We finished up by talking about functions and creating closures. This chapter will look at object-oriented concepts in R, explore concepts in statistical analysis, and finally see how R can be incorporated into R Markdown for real time distribution. Object-Oriented Programming in R R supports two different systems for creating objects: the S3 and S4 methods. S3 is the default way that objects are handled in R. We’ve been using and making S3 objects with everything that we’ve done so far. S4 is a newer way to create objects in R that has more built-in validation, but more overhead. Let’s take a look at both methods. Okay, so traditional, class-based, object-oriented design is characterized by creating classes that are the blueprint for instantiated objects (see Figure 3-1). class matrix m1 m2 object object Figure 3-1. The matrix class is used to create the variables m1 and m2, both matrices At a very high level, in traditional object-oriented languages, classes can extend other classes to inherit the parent class’ behavior, and classes can also implement interfaces, which are contracts defining what the public signature of the object should be. See Figure 3-2 for an example of this, in which we create an IUser interface that describes what the public interface should be for any user type class, and a BaseUser class that implements the interface and provides a base functionality. In some languages, we might make BaseUser an abstract class, a class that can be extended but not directly instantiated. The User and SuperUser classes extend BaseClass and customize the existing functionality for their own purposes.
- 53. Chapter 3 ■A Deeper Dive into R 48 There also exists the concept of polymorphism, in which we can change functionality via the inheritance chain. Specifically, we would inherit a function from a base class but override it, keep the signature (the function name, the type and amount of parameters it accepts, and the type of data that it returns) the same, but change what the function does. Compare overriding a function to the contrasting concept of overloading a function, in which the function would have the same name but a different signature and functionality. S3 Classes S3, so called because it was first implemented in version 3 of the S language, uses a concept called generic functions. Everything in R is an object, and each object has a string property called class that signifies what the object is. There is no validation around it, and we can overwrite the class property ad hoc. That’s the main problem with S3—the lack of validation. If you ever had an esoteric error message returned when trying to use a function, you probably experienced the repercussions of this lack of validation firsthand. The error message was probably generated not from R detecting that an incorrect type had been passed in, but from the function trying to execute with what was passed in and failing at some step along the way. See the following code, in which we create a matrix and change its class to be a vector: m - matrix(c(1:10), nrow=2) m [,1] [,2] [,3] [,4] [,5] [1,] 1 3 5 7 9 [2,] 2 4 6 8 10 class(m) - vector m [,1] [,2] [,3] [,4] [,5] [1,] 1 3 5 7 9 [2,] 2 4 6 8 10 attr(,class) [1] vector BaseUser login() createPlaylist extends extends implements User login() createPlaylist() SuperUser login() createPlaylist() editPermissions() IUser login() createPlaylist() Figure 3-2. An IUser interface implemented by a superclass BaseUser that the subclasses User and SuperUser extend
- 54. Chapter 3 ■A Deeper Dive into R 49 Generic functions are objects that check the class property of objects passed into them and exhibit different behavior based on that attribute. It’s a nice way to implement polymorphism. We can see the methods that a generic function uses by passing the generic function to the methods() function. The following code shows the methods of the plot() generic function: methods(plot) [1] plot.acf* plot.data.frame* plot.decomposed.ts* plot.default plot.dendrogram* [6] plot.density plot.ecdf plot.factor* plot.formula* plot.function [11] plot.hclust* plot.histogram* plot.HoltWinters* plot.isoreg* plot.lm [16] plot.medpolish* plot.mlm plot.ppr* plot.prcomp* plot.princomp* [21] plot.profile.nls* plot.spec plot.stepfun plot.stl* plot.table* [26] plot.ts plot.tskernel* plot.TukeyHSD Non-visible functions are asterisked Notice that within the generic plot() function is a myriad of methods to handle all the different types of data that could be passed to it, such as plot.data.frame for when we pass a data frame to plot(); or if we want to plot a TukeyHSD object plot(), plot.TukeyHSD is ready for us. Note ■ ■ Type ?TukeyHSD for more information on this object. Now that you know how S3 object-oriented concepts work in R, let’s see how to create our own custom S3 objects and generic functions. An S3 class is a list of properties and functions with an attribute named class. The class attribute tells generic functions how to treat objects that implement a particular class. Let’s create an example using the UserClass idea from Figure 3-2: tom - list(userid = tbarker, password = password123, playlist=c(12,332,45)) class(tom) - user We can inspect our new object by using the attributes() function, which tells us the properties that the object has as well as its class: attributes(tom) $names [1] userid password playlist $class [1] user Now to create generic functions that we can use with our new class. Start by creating a function that will handle only our user object; then generalize it so any class can use it. It will be the createPlaylist() function and it will accept the user on which to perform the operation and a playlist to set. The syntax for this is [function name].[class name]. Note that we access the properties of S3 objects using the dollar sign: createPlaylist.user - function(user, playlist=NULL){ user$playlist - playlist return(user) }
- 55. Random documents withunrelated content Scribd suggests to you:
- 56. To Wirklich GeheimrathHerr von Massow. Berlin, October 23rd 1842. Your Excellency, Permit me respectfully to ask whether you will be so good as to assist in procuring me an audience of his Majesty, to place before him my present position here, and my wishes with regard to it. Your Excellency is aware that I am not so situated as to be able to accept the proposal of Herr Eichhorn to place myself at the head of the whole of the Evangelical Church music here. As I already told the Minister (and your Excellency quite agreed to this in our last conversation), such a situation, if considered practically, must either consist of a general superintendence of all the present organists, choristers, school-masters, etc., or of the improvement and practice of the singing choirs in one or more cathedrals. Neither of these, however, is the kind of work which I particularly desire. Moreover, the first of these functions is superfluous if such places are properly filled; and the second, to be really effectually carried out, demands more vast and comprehensive regulations, and greater pecuniary resources than could be obtained at this moment. With regard to the other plans which were proposed, partly for the reorganization of the present Institute, and partly for the establishment of a new one, difficulties have arisen which render the establishment of these plans void; and thus the case now occurs which your Excellency may remember I always anticipated, much to my regret, at the very beginning of our correspondence in December, 1840,—there is no opportunity on my side for a practical, influential, musical efficiency in Berlin. Herr Eichhorn declared that this would be altered in the course of time; that everything was being done in order to bring about a different state of things, and he requested me to wait with patience till the building was completed which it was proposed to erect. I think, on the contrary, that it would not be responding properly on my part to the confidence the King has placed in me, if I were not at once to employ my energies in fulfilling what your Excellency at that time told me, in the name of the King, were his designs; if, instead of at least making the
- 57. attempt to animateand ennoble my art in this country (as your Excellency was pleased to say), I were to continue to work for myself personally; if I were to wait instead of to act. The very depth of my gratitude for such flattering confidence constrains me to say all this candidly to his Majesty,— to state that circumstances, over which I have no control, now render the fulfilment of his commands impossible. My wish is that his Majesty would permit me in the meantime to reside and to work, and to await his commands in some other place, where I could for the moment be useful and efficient. As soon as the building is finished, of which Herr Eichhorn spoke, or so soon as the King required any service from me, I should consider it a great happiness to hasten back and to exert my best energies for such a Sovereign, whose mandates are in themselves the highest rewards for an artist. I would fain have written this to the King sooner, but when I reflected that my communication would only meet his Majesty’s eye among a vast number of others, I thought I could express my views and feelings of most sincere gratitude, more plainly and better, verbally, even if only by a few words; and that your Excellency may be so obliging as to promote my wish is my present request, and the object of this letter.—I am, your Excellency’s most devoted Felix M. B.
- 58. To His Majestythe King of Prussia.[59] Berlin, October, 28th, 1842 Your Majesty, In the memorable words your Majesty was pleased to address to me, you mentioned that it was intended to add a certain number of able singers to the existing Royal Church choirs, to form a nucleus for these choirs, as well as for any amateurs of singing who might subsequently wish to join them, serving as a rallying-point and example, and in this manner gradually to elevate and to ennoble church music, and to ensure its greater development. Also, in order to support the singing of the congregation by instruments, which produce the most solemn and noble effects,—as your Majesty may remember, during the celebration of the Jubilee in the Nicolai Church,—it is proposed that a small number of instrumentalists (probably selected from the members of the Royal Orchestra) should be engaged, who are also intended to form the basis for subsequent grand performances of oratorios, etc. The direction of a musical choir of this instructive nature, a genuine Royal Orchestra, your Majesty expressed your intention to entrust to me, but, till its formation, to grant me entire freedom of choice with regard to my place of residence. The execution of this plan will fulfil to the utmost all my wishes as to public musical efficiency; I can never cease to be grateful to your Majesty for it, and I do not doubt that the organization of such an institution could be effected here without any serious difficulties. But I would request your Majesty not to devolve this organization on me personally, but merely to permit me to co-operate with my opinion and advice, which I shall always be gladly prepared to give. Until however, to use your Majesty’s own expression, the instrument is ready on which I am hereafter to play, I wish to make use of the freedom of action so graciously accorded me, and shortly to return to Leipzig, for the direction of the Town Hall concerts. The orders which your Majesty was pleased to give me, I shall there with the utmost zeal and to the best of my abilities carry into execution; at the same time I entreat your Majesty, as I am engaged in no
- 59. public sphere ofaction here till the organization of the Institute, and am till then to enjoy entire liberty, to be allowed to give up one-half of the salary, previously granted to me, so long as I take advantage of this entire freedom from work. In repeating my heartfelt thanks for all the favours which your Majesty has so liberally bestowed on me,—I am, till death, your Majesty’s devoted servant, Felix Mendelssohn Bartholdy.
- 60. To Carl Klingemann,London. Leipzig, November 23rd, 1842. We are now again settled in Leipzig, and fairly established here for this winter and till late in the spring. The old localities where we passed so many happy days so pleasantly are now re-arranged with all possible comfort, and we can live here in great comfort. I could no longer endure the state of suspense in Berlin; there was in fact nothing certain there, but that I was to receive a certain sum of money, and that alone should not suffice for the vocation of a musician; at least I felt more oppressed by it from day to day, and I requested either to be told plainly I should do nothing (with which I should have been quite contented, for then I could have worked with an easy mind at whatever I chose), or be told plainly what I was to do. As I was again assured that the results would certainly ensure my having employment, I wrote to Herr von Massow begging him to procure me an audience of the King, that I might thank him verbally, and endeavour to obtain my dismissal on such and such grounds, requesting him to communicate the contents of this letter to his Majesty; this he did, and appointed a day for the audience, at the same time saying that the affair was now at an end; the King very much displeased with me, and that it was his intention to take leave of me in very few words. He had made me some proposals in the name of the King to which I could not altogether agree, and with which I do not now detain you, as they led to nothing, and could lead to nothing. So I was quite prepared to take my leave of Berlin in very bad odour, however painful this might be to me. I was at length obliged also to speak to my mother on the subject, and to break to her that in the course of eight days I must return to Leipzig; I could not have believed that this would have affected her so terribly as it actually did. You know how calm my mother usually is, and how seldom she allows any one to have a glimpse of the feelings of her heart, and therefore it was doubly and trebly painful to me to cause her such a pang of sorrow, and yet I could not act otherwise; so next day I went to the King with Massow—the most zealous friend I have in Berlin—and who first took a final leave of me in his own house. The King must have been in an especial good humour, for instead of finding him angry with me, I never saw him so amiable and so really
- 61. confidential. To myfarewell speech he replied: he could not indeed compel me to remain, but he did not hesitate to say, that it would cause him heartfelt regret if I left him; that by doing so, all the plans which he had formed from my presence in Berlin would be frustrated, and that I should leave a void which he could never fill up. As I did not admit this, he said if I would name any one capable of carrying such and such plans into execution as well as he believed I could do, then he would entrust them to the person I selected, but he felt sure I should be unable to name one whom he could approve of. The following are the plans which he detailed at full length; first of all, to form a kind of real capelle, that is, a select choir of about thirty very first-rate singers, and a small orchestra (to consist of the élite of the theatrical orchestra); their duties to consist in Church music on Sundays and at festivals, and besides this, in performing oratorios and so forth; that I was to direct these, and to compose music for them, etc. etc. “Certainly,” said I, “if there were any chance of such a thing here, if this were only accomplished;” it was the very point at issue on which I had so much insisted. On which he replied again, that he knew perfectly well I must have an instrument to make music on, and that it should be his care to procure such an instrument of singers and players; but when he had procured it, he must know that I was prepared to play on it; till then I might do as I liked, return to Leipzig, or go to Italy,—in short, be entirely unfettered; but he must have the certainty that he might depend on me when he required me, and this could only be acquired by my remaining in his service. Such was at least the essential substance of the whole long conversation; we then separated. He said I was not to give him my decision immediately, because all difficulties could not be for the moment entirely obviated; I was to take time to consider, and to send my answer to Massow, who was present during the whole of this conversation of an hour and a quarter. He was quite flushed with excitement when we left the room, repeating over and over again, “Surely you can never now think of going away!” and to tell you the truth, I thought more of my dear mother than of all the rest. In short, two days afterwards I wrote to the King, and said that after his words to me I could no longer think of leaving his service, but that, on the contrary, my best abilities should be at his command so long as I lived. He had mentioned so and so (and I repeated the substance of our conversation), that I would take advantage of the liberty he had granted me, and remain in Leipzig until I was appointed to some definite sphere of work; on which
- 62. account, I beggedto relinquish one-half of my salary, so long as I was not really engaged in active work. This proposal he accepted, and I am now here again with my wife and child. I have been obliged definitively to decline the offers of the King of Saxony; but in order to do so in the most respectful manner, I went to Dresden a few days after my return here, thanked the King once more verbally, and entreated him not the less to bestow the twenty thousand thalers (which an old Leipziger bequeathed in his will to the King for the establishment of an Academy of Art) to found a school for music in Leipzig, to which he graciously acceded. The official announcement came the day before yesterday. This music school is to be organized next winter, at least in its chief features; when it is established, I may well say that I have been the means of procuring a durable benefit for music here. If they begin anything solid in Berlin, I can settle there with a clear conscience; if they allow the matter to stand over, it is probable that I may go on with my half-salary and my situation here for more than a year, and my duties be confined, as now, to executing particular commands of the King,—for instance, I am to supply him with music for the “Midsummer Night’s Dream,” the “Storm,” and “Œdipus Coloneus.” Such then is the desired conclusion of this long, long transaction. Forgive all these details, but I wished to inform you minutely of every particular. A request occurs to me which I long ago intended to have made to you. In Switzerland I saw my former guide, Michael, whom, on my previous mountain-expeditions, I always found to be an excellent, honest, obliging fellow, and on this occasion I met with him again, married to a charming pretty woman; he has children, and is no longer a guide, but established as landlord of the ‘Krone.’ During our first visit to Meiringen this summer, we lived at the Hôtel de Reichenbach, but the second time we were at the ‘Krone,’ and quite delighted with the cleanliness, and neatness, and the civil behaviour of all the people in the house. It is a most genuine Swiss village inn, taken in its best sense. Now Michael’s greatest wish is to be named among the inns at Meiringen, in the new edition of Murray’s ‘Switzerland,’ and I promised to endeavour to effect this for him.[60] Is it in your power to get this done? The first inn there is the ‘Wilde Mann,’ the second the ‘Reichenbach,’ and the third undoubtedly the ‘Krone;’ and if Murray recommends it as such, I am convinced it will do him credit. He might also mention that it is most beautifully situated, with a full view of the
- 63. Engelhorn, and theglacier of the Rosenlaui. Michael said that the editor of the Handbook had been there, and very much fêté by the other landlords; his means did not admit of this, still he would give him a good round sum of money if he would only mention him. I was indignant, and said, “Without money, or not at all.” But I thought of many musical newspapers and composers, so I did not lecture him much on the subject, from the fear that he might one day hear something of the same sort from one of my colleagues, and take his revenge. There is now a general complaint, that the large town hotels have superseded the smaller comfortable genuine Swiss inns; this is one of the latter sort. Murray must really recommend it. Pray do what you can about this, and tell me if you succeed. Forgive my troubling you, the secretary to an embassy, with such things, but if you knew Michael you would like him, I know. I would fain draw a great deal now, and gladly devote myself to all manner of allotria, including composition; but I see lying before me an enormous thick packet of proofs of my A minor symphony, and the ‘Antigone,’ which must absorb all my leisure time; and then the frightful heap of letters! My dearest friend, may these lines find you in good health, and in a happy frame of mind; may you think of me, as I shall of you, so long as life lasts; and may you also soon be able to tell me yourself that it is so, and again rejoice your true friends by your presence, for Cecile writes this letter from first to last along with me, and knows all I have said, and is, like myself, for ever and ever your friend. F. M. B.
- 64. To his Mother. Leipzig,November 28th, 1842. Dearest Mother, As pen and paper must again serve instead of our usual evening hour for tea, I begin by making a suggestion, which is, whether you would like me to write to you regularly every Saturday (perhaps only a few words, but of this hereafter); and that one of the family, as often as you cannot or will not write, should undertake to send me a punctual reply. In addition to the joy of knowing beforehand the day when I am to hear of you, it is in some degree indispensable to ensure my writing to you, for time must be found for a weekly letter; while, were this not the case, I should be ashamed to send you only a few lines, should it happen that I could not accomplish more. You can have no idea of the mass of affairs—musical, practical, and social—that have accumulated on the table in my study since my return here. The weekly concerts; the extra ones; the money the King has at length bestowed at my request on the Leipzigers, and for the judicious expenditure of which I only yesterday had to furnish the prospectus; the revisal of “Antigone” and of the A minor symphony, its score and parts; and a pile of letters. These are the principal points, which, however, branch off into a number of secondary ones. Besides, Raupach has already sent me the first chorus of “Athalia.” The “Midsummer Night’s Dream” and “Œdipus” daily work more busily in my head; I am really anxious at last to make the “Walpurgis Nacht” into a symphony cantata, for which it was originally intended, but did not become so from want of courage on my part, and I must also complete my violoncello sonata. Old Schröder’s concert took place three days ago, in which I played, and directed the overture to “Ruy Blas;” the old déclamatrice delighted us all exceedingly by the great power and spirit of her voice, and every gesture. In particular passages I thought she laid rather too much stress on the expression of the words, and gave too much preference to details over the voice; but as a whole her genius was highly remarkable. In her youth, had she the reputation of laying more stress on effect than was admissible? and what were her best parts in those days? Her daughter (looking younger, and
- 65. wilder, and moreof a madcap than ever) sang also, and sings this evening in Döhler’s concert; she will also probably sing in our subscription concert next Thursday; the days which she passes in any town, are not of the most quiet description for her acquaintances. We had besides, Tichatschek, Wagner, Döhler, Mühlenfels,—so there was a continual hurry and excitement last week. Make them read aloud to you at the tea-table the passage from the last of Lessing’s ‘Antiquarian Letters,’ “Wenn ich Kunstrichter wäre,” etc. etc.,— and tell me whether any of you dispute the point, or whether you all agree with me, that it is the most exhaustive address which can be made to a critic, indeed to every critic. At this moment, when so many artists, old and young, good and bad, come here, this passage daily recurs to me.—Your Felix.
- 66. To Paul MendelssohnBartholdy. Leipzig, December 5th, 1842. My dear Brother, As we agreed (and indeed very properly) that I was to take no step with regard to my affairs in Berlin without informing you immediately of every detail, I write you these lines to-day, although I am over head and ears in business. I received yesterday from the King the following communication: — “By the enclosed written document you will perceive the tenor of the communication I have this day made on the subject of an Institute for the Improvement of Church Singing; it is addressed to the Special Commissioners, W. G. R. von Massow and W. G. R. General Intendant of Court Music, Graf von Redern. I have also, in compliance with your own wish, informed the Minister of State, Eichhorn, and the Finance Minister, Von Bodelschwingh, that, until you enter on your functions, you decline receiving more than fifteen hundred thalers, instead of three thousand. I nominate you General Music Director, and entrust to you the superintendence and direction of church and sacred music as your appointed sphere of action.—Charlottenburg, November 22nd, 1842.” The enclosure consists of a Cabinet order, which is drawn up in a most clear and judicious style, entirely in the spirit of our interview, and thoroughly in accordance with my wishes, manifestly with the co-operation of Herr von Massow, and with the true and honest purpose of carrying out the affair. That no material obstacles exist, is again evident from this cabinet order, but whether I may consider the accomplishment of the project as certain, I cannot say with any security till I actually see it. The affair of the Conservatorium was still further advanced, and seemed even more decided. On the other hand, I adhere to my former views, and do what I can to promote the project, and to display my goodwill towards it. Herr von Massow writes to me (only yesterday) that I had better soon come again to Berlin, to converse with him and Graf von Redern, and that only one or two days would be required; I shall, however, answer him that I mean to go there on the 17th, and have arranged to remain till the 23rd. A
- 67. longer stay isunfortunately impossible; still you and I can have some political gossip together, and be inseparable during my stay. The King having on this occasion conferred on me a new title,[61] almost embarrasses me; I am unwilling to be of the number of those in the present day, who possess a greater number of decorations than they have written good compositions, and yet it seems rather like it; at all events, I really have no idea what return I can possibly make for all this, still, as I have not in any way sought it, I may be excused. To refuse such a thing is out of the question, and there is no one who does not rejoice in being over-estimated, because on some other occasion the balance is sure to be made even by depreciation.—Ever your Felix.
- 68. To His Mother. Leipzig,December 11th, 1842. Dearest Mother, On the 21st or 22nd, we give a concert here for the King, who has sworn death and destruction to all the hares in the country round. In this concert we mean to sing for his benefit (how touching!) the partridge and hare hunt out of the “Seasons.” My “Walpurgis Nacht” is to appear once more in the second part, in a somewhat different garb indeed from the former one, which was somewhat too richly endowed with trombones, and rather poor in the vocal parts; but to effect this, I have been obliged to re-write the whole score from A to Z, and to add two new arias, not to mention the rest of the clipping and cutting. If I don’t like it now, I solemnly vow to give it up for the rest of my life. I think of bringing with me to Berlin a movement from the “Midsummer Night’s Dream,” and one from “Œdipus.” The music school here, please God! will make a beginning next February; Hauptmann, David, Schumann and his wife, Becker, Pohlenz, and I, are to be the teachers at first. It commences with ten sinecures; the rest who may wish to have instruction, must pay seventy-five thalers a year. Now you know all that I know, the rest can only be taught by experience and trial. I wished for you recently at a subscription concert. I think I never played the Beethoven G major concerto so well,—my old cheval de bataille; the first cadence especially, and a new return to the solo, pleased me exceedingly, and apparently the audience still more. What you write to me about the répertoire of your Berlin concerts, does not inspire me with any wish to hear more about them. The arrangement of the “Aufforderung zum Tanz,” and the compositions of English ambassadors,—these are valuable things! If experiments are to be thus made and listened to, it would be advisable to be rather more liberal towards the works of our Fatherland. You will again say that I am cynical; but many of my ideas are so intimately connected with my life and my views on art, that you must be indulgent with regard to them. The monument to old Sebastian Bach is now very handsome.[62] Bendemann was here the day before yesterday, to inspect it once more. All
- 69. the inner scaffoldinghad been removed, so the pillars and smaller columns, and scrolls, and above all the bas-reliefs, and the grand, antiquated old features sparkled clearly in the sun, and caused me great delight. The whole structure, with its numerous elegant decorations, is really typical of the old fellow. It is now covered up again, and will remain so till March, when it is to be inaugurated on his birthday, by one of his motetts. Cedars are to be planted round the monument, and a Gothic seat placed in front of it. We are anxious, however, not to make too much fuss on the subject, and to avoid the present pompous style of phraseology, and the worship of art and artists, which is so much the fashion. Here, the outward aspect of things is now as much too flourishing, as it formerly was too miserable for artists, which would be very pleasant for us, but it does harm to the cause. Art is becoming spoiled and sluggish, so we should rather be grateful to our present enemies than be angry with them. I also consider it too much good fortune that the King of Prussia has nominated me General Music Director. This is another new title and new honour, whereas I really do not know how to do enough to deserve the old ones. This is a hallowed day for us all, with its delightful and memorable recollections;[63] think of me too on this anniversary, as I do of you and of him, so long as life endures.—Your Felix.
- 70. To Pastor JuliusSchubring, Dessau. Leipzig, December 16th, 1842. My dear Schubring, I now send you, according to your permission, the text of “Elijah,” so far as it goes. I do beg of you to give me your best assistance, and return it soon with plenty of notes on the margin (I mean Scriptural passages, etc.). I also enclose your former letters on the subject, as you wished, and have torn them out of the book in which they were. They must, however, be replaced, so do not forget to send them back to me. In the very first of these letters (at the bottom of the first page), you properly allude to the chief difficulty of the text, and the very point in which it is still the most deficient—in universally valid and impressive thoughts and words; for of course it is not my intention to compose what you call “a Biblical Walpurgis Night.” I have endeavoured to obviate this deficiency by the passages written in Roman letters, but there is still something wanting, even to complete these, and to obtain suitable comprehensive words for the subject. This, then, is the first point to which I wish to direct your attention, and where your assistance is very necessary. Secondly, in the “dramatic” arrangement. I cannot endure the half operatic style of most of the oratorio words, (where recourse is had to common figures, as, for example, an Israelite, a maiden, Hannah, Micaiah, and others, and where, instead of saying “this and that occurred,” they are made to say, “Alas! I see this and that occurring.”) I consider this very weak, and will not follow such a precedent. However, the everlasting “he spake” etc., is also not right. Both of these are avoided in the text; still this is, and ever will be, one of its weaker aspects. Reflect, also, whether it is justifiable that no positively dramatic figure except that of Elijah appears. I think it is. He ought, however, at the close, at his ascension to heaven, to have something to say (or to sing). Can you find appropriate words for this purpose? The second part, moreover, especially towards the end, is still in a very unfinished condition. I have not as yet got a final chorus; what do you advise it to be? Pray study the whole carefully, and write on the margin a great many beautiful arias, reflections,
- 71. pithy sentences, choruses,and all sorts of things, and let me have them as soon as possible. I also send the ‘Méthode des Méthodes.’ While turning over its leaves, I could not help thinking that you will here and there find much that will be useful. If that be the case, I beg you will keep it as long as you and your young pianoforte player may require it. I don’t use it at all. If it does not please you, I can send you instead, a sight of Zimmermann’s ‘Pianoforte School,’ which is composed pretty much on the same principle, and has only different examples, etc. Speaking is a very different thing from writing. The few minutes I lately passed with you and yours, were more enlivening and cheering than ever so many letters.—Ever your Felix M. B.
- 72. To Paul MendelssohnBartholdy. Leipzig, December 22nd, 1842.[64] My dear Brother, I wrote to you the day after our arrival here that we were all well, and living in our sorrow as we best could, dwelling on the happiness we once possessed. My letter was addressed to Fanny, but written to you all; though it seems you had not heard of it, and even this trifle shows, what will day by day be more deeply and painfully felt by us,—that the point of union is now gone, where even as children we could always meet; and though we were no longer so in years, we felt that we were still so in feeling. When I wrote to my Mother, I knew that I wrote to you all, and you knew it too; we are children no longer, but we have enjoyed what it really is to be so. Now, this is gone for ever! At such a time, with regard to outward things, we are as if in a dark room, groping to find the way, hour after hour. Tell me if we cannot arrange that I should write to one of you by turns once every week, and get an answer from you, so that we may at least hear of each other every three weeks, independent of more frequent letters; or say whether any better arrangement occurs to you. I thank you a thousand times for your kind question about the house. I had thought of asking you for it, and now you offer it to me. But before we finally settle this, I should like you to bring the subject cautiously on the tapis, in the presence of our sisters and brother-in-law. If you perceive that any unpleasant feeling is awakened in their minds by such a proposal, when for the first time, in Berlin, I am not to live under the same roof with them, and if they give any indication of such a feeling, even by a single word or remark, (you will quickly observe this, and I rely entirely on you,) then we must give it up. In any other event, I shall thankfully accept your kindness. My next visit to Berlin will be a severe trial to me; indeed, all I say and do is a trial to me,—anything, in short, that is not mere patient endurance. I have, however, begun to work again, and that is the only thing which occupies me a little. Happily, I have some half-mechanical work to do,—transcribing, instrumentation, and similar things. This can be accomplished by a kind of almost animal instinct, which we can follow, and which does us more good than if we had it not. But yesterday I was obliged to direct. That was terrible. They told me
- 73. that the firsttime would be terrible, but sooner or later it must be done. I thought so too, but I would fain have waited for a few weeks. The first thing was a song of Rochlitz’s; but when in the rehearsal the alto sang, piano, “Wie der Hirsch schreit,”[65] I was so overcome, that I was obliged afterwards to go out of the room, to give free vent to my tears. To-day, Heaven be praised, I am not required to see or speak to any one, and my cough is better. Thus time glides on; but what we have once possessed is not less precious, and what we have now lost not less painful with time. Farewell, dearest Brother. Continue to love me.—Your Felix.
- 74. To Professor Köstlin,Tübingen. Leipzig, January 12th, 1843. Dear Herr Köstlin, or rather, dear Herr Godfather, You have caused me much joy by your kind letter of yesterday, and by the happy intelligence it contained, and above all, by your wish that I should be godfather! Indeed, you may well believe that I gladly accede to the request, and after reading your letter, it was some moments before I could realize, that I could not possibly be present at the baptism. In earlier days, no reasoning would have been of any avail; I would have taken post horses and arrived in your house for the occasion. This I cannot now do, but if there be such a thing as to be present in spirit, then I shall indeed be so. The remembrance of me by such well-beloved friends, and this proof of your regard, which causes a still more close and enduring tie between us, cannot fail to cause true joy and exhilaration of heart; and believe me, I feel this joy, and thank you and your wife for it. That I am to be godfather is then settled; but there are a thousand things I still wish to know, and if, when the christening is over, you do not write me all the details which you omit in this letter, you must expect a good scolding. You forget that I have myself three children, so I am doubly interested in such things. You do not even mention the name the boy is to have, and whether he is fair or dark, or has black or blue eyes. My wife is as desirous as I am to know all this, and we hope that after the christening you will write to us every particular. You were rather displeased with me for being so bad a correspondent. I earnestly entreat of you never to be displeased with me on that account; I cannot remedy this; it is a fault which, in spite of the best resolutions on my part, I constantly fall into, and which I shall never be cured of so long as I live. There is so much that stands in my way; first, a really instinctive dislike to pen and paper, except where music is concerned; then the various scattered branches of a perfect maze of professional and other avocations, which I am obliged to undertake partly for myself and partly for others, so that I really sometimes can only carry on life like a person in a crowd pushing his way, and shoving along with both his elbows, using his feet too, as well as his fists and teeth, etc. This is, in
- 75. fact, my moodmany a week; I extort the time for writing music, otherwise I could not go on from day to day, but I cannot find leisure to write letters. We have had recently a bitter heavy loss to bewail,—that of my dear Mother. I intended to have written in a gay mood all through this letter, and not by a single word to allude to anything, that by its melancholy nature might disturb your happiness, but I feel that I must write this to you, otherwise all that I say would appear mere hypocrisy. You must therefore take part in my sorrow, for I could not conceal from you the event that during the last few weeks, has so bowed us down from grief, and which it will be long before we can recover from. Yet such a letter as yours is welcome at all times, and in all sorrow, and just as I know how you will feel towards me on hearing this, so you know how cordially I sympathize with your joy; this may well be called sincere attachment! Give your wife a thousand greetings and congratulations from me. Tell me if she has composed new songs or anything else; what I should like best would be to receive one from her in a letter; they always delight me so much, when I hear and play them.—Ever your devoted Felix Mendelssohn Bartholdy.
- 76. To Fanny Hensel,Berlin. Leipzig, January 13th, 1843. ... We yesterday tried over a new symphony by a Dane of the name of Gade, and we are to perform it in the course of the ensuing month; it has given me more pleasure than any work I have seen for a long time. He has great and superior talents, and I wish you could hear this most original, most earnest, and sweet-sounding Danish symphony. I am writing him a few lines to-day, though I know nothing more of him than that he lives in Copenhagen, and is twenty-six years of age, but I must thank him for the delight he has caused me; for there can scarcely be a greater than to hear fine music; admiration increasing at every bar, and a feeling of congeniality; would that it came less seldom!
- 77. To A. W.Gade, Professor of Music, Copenhagen. Leipzig, January 13th, 1842. Sir, We yesterday rehearsed for the first time your symphony in C minor, and though personally a stranger, yet I cannot resist the wish to address you, in order to say what excessive pleasure you have caused me by your admirable work, and how truly grateful I am for the great enjoyment you have conferred on me. It is long since any work has made a more lively and favourable impression on me, and as my surprise increased at every bar, and yet every moment I felt more at home, I to-day conceive it to be absolutely necessary to thank you for all this pleasure, and to say how highly I esteem your splendid talents, and how eager this symphony (which is the only thing I know of yours) makes me to become acquainted with your earlier and future compositions; but as I hear that you are still so young, it is the thoughts of those to come in which I particularly rejoice, and your present fine work, causes me to anticipate these with the brightest hopes. I once more thank you for it and the enjoyment I yesterday had. We are to have some more rehearsals of the symphony, and shall probably perform it in the course of three or four weeks. The parts were so full of mistakes, that we were obliged to revise them all, and to have many of them transcribed afresh; next time it will not be played like a new piece, but as one familiar and dear to the whole orchestra. This was indeed the case yesterday, and there was only one voice on the subject among us musicians, but it must be played so that every one may hear it properly. Herr Raymond Härtel told me, there was an idea of your coming here yourself in the course of the winter. I hope this may be the case, as I could better and more plainly express my high estimation and my gratitude to you verbally, than by mere empty written words. But whether we become acquainted or not, I beg you will always look on me as one who will never cease to regard your works with love and sympathy, and who will ever feel the greatest and most cordial delight in meeting with such an artist as yourself, and such a work of art as your C minor symphony.—Your devoted Felix Mendelssohn Bartholdy.
- 78. To Carl Klingemann,London. Leipzig, January 13th, 1843. I cannot as yet at all reconcile myself to distraction of thought and every- day life, as it is called, or to life with men who in fact care very little about you, and to whom what we can never forget or recover from, is only a mere piece of news. I now feel however more vividly than ever what a heavenly calling Art is; and for this also I have to thank my parents; just when all else which ought to interest the mind appears so repugnant, and empty, and insipid, the smallest real service to Art lays hold of your inmost thoughts, leading you so far away from town, and country, and from earth itself, that it is indeed a blessing sent by God. A few days previous to the 11th, I had undertaken to transcribe my “Walpurgis Nacht,” which I had long intended to do, and caused the voice parts of the whole of the voluminous score, to be written out and copied afresh. Then I was summoned to Berlin, and after an interval of some weeks, I have now begun to write the instrumental parts in my little study, which has a pretty view of fields, and meadows, and a village. I sometimes could not leave the table for hours, I was so fascinated by such pleasant intercourse with the old familiar oboes and tenor violins, which live so much longer than we do, and are such faithful friends. I was too sorrowful, and the wound too recent, to attempt new compositions; but this mere mechanical pursuit and employment, was my consolation the whole time that I was alone, when I had not my wife and children with their beloved faces, who make me forget even music, and cause me daily to think how grateful I ought to be to God, for all the benefits he bestows on me. You have not quite understood my previous letter. You say “I could not act otherwise in my official position.” It was not that, it was my Mother I alluded to. All the plans and projects have since then been dragging on slowly; I have my half-salary, and begun the music for the “Midsummer Night’s Dream,” “Œdipus” and others for the King. My private opinion is still, that he is resolved to allow things to rest as they are; in the meantime, I have established the Conservatorium here, the official announcement of which you will read in the newspapers, and it gives me a great deal to do.
- 79. To Madame EmmaPreusser. Leipzig, February 4th, 1843. Dear Lady, I send “Siebenkäs,” according to your desire. May it cause you half the pleasure it caused me when I first read it, and very frequently since. I believe that the period when we first learn to love, and to know such a glorious work, is among the happiest hours of our lives. As you have read very little of Jean Paul, were I in your place, I would not concern myself much about the prologues, but at first entirely discard the “Blumenstücke,” and begin at once at page 26, and follow the story of “Siebenkäs” to its close. When you have read this, and perhaps also the “Flegel Jahre,” and some more of his wonderful works, then no doubt you will like and prize all he has written,—even the more laboured, the less happy, or the obsolete,— and then you will no longer wish to miss the “Blumenstücke,” the prologues, and the “Traum im Traum,” etc. etc. As soon as you wish for anything new, you will always find me at the service of you and yours.—Your devoted Felix Mendelssohn Bartholdy.
- 80. To A. W.Gade, Professor of Music, Copenhagen. Leipzig, March 3rd, 1843. Sir, Your C minor symphony was performed for the first time yesterday at our eighteenth subscription concert here, to the lively and unalloyed delight of the whole public, who broke out into the loudest applause at the close of each of the four movements. There was great excitement among the audience after the scherzo, and the shouting and clapping of hands seemed interminable; after the adagio the very same; after the last, and after the first,—in short, after all! To see the musicians so unanimous, the public so enchanted, and the performance so successful, was to me a source of delight as great as if I had written the work myself, or indeed I may say greater,— for in my own compositions, the faults and the less successful portions always seem to me most prominent, whereas in your work, I felt nothing but pure delight in all its admirable beauties. By the performance of yesterday evening you have gained the whole of the Leipzig public, who truly love music, as permanent friends; none here will ever henceforth speak of you or of your works but with the most heartfelt esteem, and receive with open arms all your future compositions, which will be assiduously studied, and joyfully hailed, by all friends to music in this town. “Whoever wrote the last half of this scherzo is an admirable genius, and we have a right to expect the most grand and glorious works from him.” Such was the universal opinion yesterday evening in our orchestra and in the whole hall, and we are not fickle here. Thus you have acquired a large number of friends for life by your work; fulfil then our wishes and hopes by writing many, many works in the same style, and of the same beauty, and thus imparting new life to our beloved art; and to effect this, Heaven has bestowed on you all that He can bestow. Besides the rehearsal which I formerly wrote to you about, we recently had two others, and with the exception of some trifling unimportant mistakes, the symphony was played with a degree of spirit and enthusiasm which at once showed how highly enchanted the musicians were with it. I hear that it is to be published by Kistner, so permit me to ask, whether the
- 81. heading of thefirst introduction, 6/4 time, afterwards repeated, may not give rise to misapprehension? If I am not mistaken it is marked moderato sostenuto. Instead of this sostenuto, ought it not rather to be printed con moto, or con molto di moto? That heading would, it seems to me, lead to the right tempo, if it were 6/8 time instead of 6/4; but in 6/4 time, it is so very customary to count the separate crotchets slowly and deliberately, that I think the movement would be taken too slow, which I found to be the case at the first rehearsal, until I no longer paid any attention to the notes or the heading, but adhered to the sense alone. As many musicians cling so closely to such headings, I was resolved at all events to mention to you my doubts on this subject. Allow me to thank you once more for your obliging letter, and the friendly intention which you inform me of in it;[66] but I thank you still more for the pleasure which you have caused me by the work itself; and pray believe that no one will follow your future course with warmer sympathy, or anticipate your future works with more anxiety and hope than your Felix Mendelssohn Bartholdy.
- 82. To I. Moscheles,London. Leipzig, April 30th, 1843. ... Our Music Academy here has made a famous beginning; fresh notices of students arrive almost daily, and the number of teachers, as well as of lessons, have been necessarily very much increased. Two serious maladies, however, are apparent, which I mean vigorously to resist with might and main so long as I am here: the Direction is disposed to increase and generalize,—that is, to build houses, to hire localities of several stories,—whereas, I maintain that for the first ten years, the two rooms we have, in which simultaneous instruction can be given, are sufficient. Then all the scholars wish to compose and to theorize, while it is my belief that practical work, thorough steady practising, and strict time, a solid knowledge of all solid works, etc., etc., are the chief things which can and must be taught. From these, all other knowledge follows as a thing of course, and anything further is not the affair of learning, but the gift of God. I need not however, I am sure, say that notwithstanding this, I am far from wishing to render Art a mere handicraft.
- 83. To M. Simrock,Bonn. Leipzig, June 12th, 1843. Sir, Herr Herrmann, some time since, inquired of you once, in my name, about the printed score of the “Zauberflöte;” but I now apply to yourself to know whether any copy of it still exists in the original German, or if any ever did exist? And if neither be the case, I should like to know whether you are disposed to allow the original correct text to be substituted in your plates of this opera, and some proofs to be taken? It appears to me almost a positive duty, that such a work should descend to posterity in its unvitiated form; we indeed all know perfectly well, for instance, the aria beginning, with the words “Dies Bildniss ist bezaubernd schön,” but if in the course of a few years the younger musicians always see it printed thus, “So reizend hold, so zaub’risch schön,” they will acquire a false idea of Mozart’s thoughts; and I go so far as to assert, that even the most undeniably bad passages in such a text deserve to be retained, as Mozart composed music for them, and they have thus become household words all through Germany. If improvements are to be proposed, it is all very well, but in that event they ought to stand side by side with the original words; in no case must they be entirely banished, otherwise fidelity towards the great deceased master is not properly observed. I beg you will say a few words on this point when you write to Herr Herrmann; and if you resolve to alter your plates, then I shall be the first, but certainly not the last, of your customers to thank you for it.—Your obedient Felix Mendelssohn Bartholdy.
- 84. To G. Otten,Hamburg. Leipzig, July 7th, 1843. Sir, My best thanks for your obliging letter, which contains much that is really far too kind and flattering about myself and my music. Gladly, in compliance with your friendly invitation, would I at some future time come to express my thanks to you personally, and to play to you as you wish me to do. Since we met in Dessau I have learnt a good deal more, and have made progress. But you must not compare my playing with my music; I feel quite embarrassed by such an idea, and I am certainly not the man to prevent people worshipping the golden calf, as it is called in the fashion of the day. Moreover, I believe that this mode will soon pass away, even without opposition. To be sure, a new one is sure to start up; on this account therefore it seems to me best to pursue one’s own path steadily, and especially to guard against an evil custom of the day, which is not included in those you name, but which however does infinite harm,—squandering and frittering away talents for the sake of outward show. This is a reproach which I might make to most of our present artists, and to myself also more than I could wish; I have no great inclination therefore to extend my travels, but rather to restrict them far more, in order to strive with greater earnestness for my own improvement instead of the good opinion of others. I conclude by thanking you for your friendly letter, and pray remember kindly your obedient Felix Mendelssohn Bartholdy.
- 85. To Paul MendelssohnBartholdy. Leipzig, July 21st, 1843. Dear Brother, I had almost hoped to be able to answer your letter in person, for I was very nearly taking a journey to Berlin again. Herr von Massow has sent me a communication connected with that tedious everlasting affair, which irritated me so much that it almost made me ill, and I do not feel right yet. In my first feeling of anger, I wished to go to Berlin to speak to you and break off the whole affair; but I prefer writing, and so I am now writing to you. Instead of receiving the assent to the proposals on which we had agreed in the interview of the 10th,[67] Herr von Massow sends me a commission to arrange for orchestra and chorus, without delay, the chorale, “Herr Gott, Dich loben wir,” the longest chorale and the most tiresome work which I ever attempted; and the day after I had finished it and sent it off, I receive an official document which I must sign before the assent of the King can be solicited; when I had signed it, the others present at that conference would also subscribe their names. In this deed all the stipulations are correctly stated, but six or eight additional clauses are written on the margin, not one syllable of which had ever been named during the conference, invalidating the whole intention of the above stipulations, and placing myself and the Institute in the most entire subservience to Herr von Küstner,—and in short, showing in the clearest light all the difficulties to which I formerly alluded, and the existence of which Herr von Massow denied. Among other things, it is said, the appointment of the orchestra for all church music is to be devolved on the theatrical music direction; before every concert there must be an application made to the General Intendancy, whether the day, which according to our agreement was to be settled once for all at the beginning of the winter, is to continue the same or be altered, etc.; all things of which not one syllable had been alluded to in the conference. As I told you, I fretted myself till I was quite ill about it. Remembering your words, I thought it the most judicious plan to write direct to the King, and break off the affair. After two days’ consideration, I did not think I was justified in doing so; I therefore wrote to Herr von Massow, why and wherefore I could not give
- 86. my signature, requestinghim to inform me whether the King intended to carry out our former agreement. If he did not feel disposed to do so, or if he, Herr von Massow, considered it necessary to insert new clauses in the agreement, I should then consider the affair impracticable, and must act accordingly. In the other view of the case, he knew that I was prepared to come; I was also to say how far I had got with “Œdipus.” I answered that in accordance with Tieck’s wish, I had arranged the “Midsummer Night’s Dream” with music, to be performed in the new palace; that I had also, by special commission from the King, written choruses,[68] and that I had not resumed the choruses of “Œdipus” since the previous autumn, because another Greek piece had been appointed to be performed. I said all this in a friendly manner, but I do assure you that the affair cost me four most angry, disturbed, and irksome days. If I could only have spoken to you for a single hour! I should have been glad to know whether you approved of my course, that is of my letter, or whether you would have preferred a short letter resigning the appointment. It is really too provoking that in all and everything the same spirit prevails; in this case too, all might be smoothed over and set to rights by a few words, and every moment I expect to hear them spoken, and then there would be a possibility of something good and new; but they are not spoken, and they are replaced by a thousand annoyances, and my head at last is so bewildered that I think I become almost as perverted and unnatural, as the whole affair is at last likely to turn out. Forgive me for causing you to have your share of annoyance, but now I have told you all—and enough. I have not been able to work during these days. To make up for this, I have done the “Jungfrau” for you in Indian ink; the mountain I think is excellent, but I have again utterly destroyed the pines in the foreground. I mean now, too, to resume your sonata.—Your Felix.
- 87. To Paul MendelssohnBartholdy. Leipzig, July 26th, 1843. Dearest Brother, I have just received your kind letter, and indeed at the very moment when I was about to write to you and beg you to give me quarters. Next Tuesday, the 1st of August, I am obliged to return to Berlin to rehearse and perform the “Tausendjährige Reich,” and to hear from the King his views with regard to the composition of the Psalms. He yesterday summoned me for this purpose, and of course I must go, and of course I must live with you; but is it also of course that my visit is convenient to you? This time I shall remain at least eight days; on the sixth is the celebration of the above- mentioned “Reich.” Give me a line in answer. I have a reply to my letter from Von Massow, who writes me the King’s invitation; he says we are sure to agree, and that some matters of form are the only things in question; that I shall spare myself the annoyance and vexation which such a tiresome correspondence must entail, and that as I am coming at all events for the “Tausendjährige Reich,” I can also reply personally to the zehntausendjährige affair. Herr von Massow, in fact, says pretty plainly, “Asking and bidding make the bargain;” that he wished to see whether I would sign; and this not being the case, the others would no doubt give way, etc. etc. All this is very confusing, and I do not at all like it. To be sure, it is true that his head must also be in a maze, and he appears to take all imaginable trouble about the affair. I mean to bring you the whole of the everlasting papers for your inspection; we can read them together when we meet. I hope, on this occasion, not merely to have a Court dinner with the King, but a satisfactory discussion on business; probably the easiest mode of bringing about a result. I wish, if possible, to defer this till after the celebration of the tausendjährig festival; the chorale, that I wrote for it, is, I believe, just what the King wishes, at all events it furnishes an opportunity for a complete understanding. My anger, which was indeed greater on this occasion than for a long time past, I shook off in a defile on the way to Naumburg, close to Rippach, where you drive down to Meissenfels; and a couple of good talks and walks
- 88. with Mühlenfels, fairlybanished every trace of it. Kösen was a pretty sight; we met Mlle. F—— and Herr C—— under the hazel bushes and lovely lime-trees, and from every shrub, instead of glow-worms glittered the order of the red eagle, of different classes; but it was really beautiful. And now I am writing music once more instead of painting fir-trees; therefore I cannot positively promise to finish the “Jungfrau” before eight days. I have washed out the forest recently, for the second time. It is a year the day after to- morrow since we set off to Switzerland.—Your Felix.
- 89. To Paul MendelssohnBartholdy. Leipzig, August 26th, 1843. Dear Brother, I yesterday received a letter from Herr von Massow containing the intelligence that the King had fully sanctioned the affair of the Wirklich Geheimrath; I wished to write this to you instantly.[69] To-day I got a second letter, with the information that the King desires to have three representations in the New Palace in the second half of September, namely, 1, “Antigone;” 2, “The Midsummer Night’s Dream;” 3, “Athalia” (“Medea” is to be given between Nos. 1 and 2, and all the four within fourteen days), and I am invited to Berlin for the purpose. Now I would rather not write, for I have a frightful quantity of things to do before then, as not one of the scores is yet fit for the transcriber, and the overture to “Athalia” still wanting, as well as the instrumentation of the whole, etc. etc. I have written nevertheless that I would come, and the music should be finished.—Ever your Felix.
- 90. Welcome to ourwebsite – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com