Data Analysis and Visualization

Página 1

Part 1

Types of Studies

Experimental studies evaluate causal effects

They interfere to keep certain factors under control in the target population and create randomized groups

Observational studies evaluate associations

They observe what lives in a target population in the real world

Types of variables

The analysis is usually started with data exploration for one variable at a time and between two or more variables for identifying possible relationships

Types of variables

Continuous or numerical variables

Takes numerical values and are also known as quantitative variables and can be

Interval variables: have a numerical value that can be measured along a continuum e.g. temperature (in °C)

Ratio variables: are interval values with a measurement value of 0 (zero) e.g. distance, weight

Categorical variables

Are qualitative variables stored in factors in R and can be

Nominal variables: levels are labels or descriptions which cannot be ordered e.g. eye color (brown/blue/...) OR gender (male/female)

Ordinal or ranked variables: levels are labels which can be ordered or ranked in a logical order e.g. height as tall/medium/short

Dichotomous data: have two possible outcome values e.g. Yes/No

Página 2

Exploring variables

For categorical variables:

Table frequency, relative

frequency and percentage

Bar graph or barplot

For numerical variables:

Mean and median

Variance and standard deviation

Histogram and distribution

Quantiles and boxplot

Summary statistics

R has a wide range of functions available for obtaining sumary statistics

This means pulling up min, max, men edian, sd, var, quantile

sumary() provides range of descriptive statistcs

The variance (var) is a measure of how the values are dispersed around the

mean and calculated by taking the sum of the squared differences from the

mean divided by the number of values minus 1 (N-1)

The standard deviation (sd) is the square root of the variance

and a measure how the values are spread out\

Quantiles are cutpoints dividing the range of a probability distribution

The 4-quantiles are called quartiles (Q)

The difference between upper (75%) and lower (25%) quartiles is called interquartile range (IQR) quantile(iris$Petal.Length) # 0% 25% 50% 75% 100%

Página 3

Graphical exploration

For categorical variables

Bar graph or barplot

Pie chart

For numerical variables

Stem-and-Leaf Plot

Histogram and distribution

Boxplot and violin plot

Quantile-quantile plot (Q-Q plot)

Barplot

A bar graph or barplot is used to visualize the possible values or levels of the

categorical variable can be taken in bars together with the number of times each

value has been observed (frequency)

Stem-and-leaf plot

A stem-and-leaf plot classifies quantitative data in a textual graph according

to their most significant numeric digits

Histogram

A histogram draws counts of how many observations fall within the specified

divisions called “bins” of the x-axis

Página 4

Distribution

In a graph, the data points can produce different shapes

The distribution shape visualizes how the data is spread together with useful statistical info, e.g., mean and range

Different factors, such as the number of peaks, symmetry, and skewness, define the distribution shape

Distribution Peaks

The peaks of the distribution are called modes and tell you in which areas the data count is higher

1) Unimodal distribution -> one peak or mode

2) Bimodal distribution -> two peaks or modes

3) Multimodal distribution -> three or more peaks or modes

Distribution symmetry and skewness

An asymmetric graph has two sides that are mirror images of each other.

Distributions that are not symmetrical differ in skewness, i.e., long tails in the positive (right skew) or negative (left skew) direction on the x-axis.

Kurtosis describes the tail shape of a distribution and tells how “peaked” a graph is

Negative kurtosis = platykurtic, e.g., uniform distribution

Positive kurtosis = leptokurtic, e.g., students’ t distribution

The normal distribution has zero kurtoses with a standard tail shape called mesokurtic.

Positive value -> too little data in tails, negative value -> too much data in tails

Página 5

Probability distributions

List of distribution & functions available in R: help(Distributions)

For every distribution -> 4 functions prepended with a letter (d, p, q ,r)

indicating their functionality

Example for the normal (or Gaussian) distribution

The dnorm function returns the value of the probability density function for the normal distribution (given parameters x, μ and σ)

Kernel density plot

A density plot is an effective way to visualize the distribution of a variable.

The density() function computes kernel density estimates

A kernel is a special type of probability density function (PDF). Kernel density estimation is a non-parametric method of estimating the PDF.

Default smoothing kernel to be used is gaussian

Quantiles and boxplot

A boxplot, also called box-and -whisker plot, shows the spread and center of data and is a way to represent the five number summary statistics in a plot:

1. Variable minimum

2. Lower quartile (Q1 or 25%)

3. Median

4. Upper quartile (Q3 or 75%)

5. Variable maximum

Boxplots can be used for individual variables but also for variables by group

Página 6

Violin plot

A violin plot is a combination of a boxplot and a kernel density plot

Quantile-quantile plot

A better way to tell whether your data is distributed is typically

using a so-called quantile-quantile (Q-Q) plot

Allows visual inspection of the similarity between the underlying

distributions of two data sets. The points will lie close to the y=x identity line if the two distributions are similar.

Sample sizes of the data sets do not need to be equal  R calculates extra values

by interpolation for the smallest sample to create two samples of the same size

The qqnorm() function creates a quantile-quantile plot evaluating the fit

of sample data to the normal distribution (sample vs theoretical quantiles)

qqline() adds a line to a theoretical, by default normal, Q-Q plot

The qqplot() function produces a Q-Q plot of two datasets

Fitting distributions

The fitdistr(sampledata, densityfunction) from the MASS package

uses maximum-likelihood fitting of univariate distributions

Página 7

Exercises

Exercise 3.1

Load the mice weight data from the previous lecture

Look at the structure and five-number summary statistics (min, max, mean, Q1, Q3)

What is the standard deviation of the body weight

Make a barplot of the mice diet frequencies

Make a histogram of the body weight

Make a boxplot of the mice body weight by diet group

Make a Q-Q plot of the mice body weight “show” vs “hf”

Exercise 3.2

Simulate a dataset of 10000 random values having a right skewed distribution using rnbinom() with mean = 10 and sd = .5

Make a hist() and set the x-axis limits (xlim) to the min and max of the dataset

Make another histogram using density on y-axis (prob = TRUE) instead of frequency

Compute the density estimates (default gaussian kernel) and use this to plot the density using lines()

Determine the skewness and kurtosis

Exercise 3.3

Load the hospital patients dataset from the previous Exercises 2.

Look at the structure. What are the variable types?

Are the classes of the variables matching the types? If not change (as.nnn).

Make a barplot of gender and a barplot of hospital survival (frequencies)

What are the quartiles (4-quantiles) of the ages?

Can you also get the 20%, 40%, 60% and 80% quantiles? Use the help: “probs =“

Plot a histogram of the patients ages. Next make a density plot of the ages.

Now plot the histogram again with a gray density line. What is the distribution mode?

Does this change if you increase the bins (e.g. breaks=10)

Make a boxplot of the patients weights. Any outliers?

How could you know of which patient the weight is an outlier?

Now program a loop by going over the features Length, Weight and HospitalDays and for each of these features make a histogram and boxplot.

Página 8

Part 2

Graphical parameters and layout

Graphic options, such as fonts, lines, plotting symbols, axes, titles, and colors, allow users to customize many features of graphs.

Graphical parameters can be specified in two ways:

1. Using the par() function -> settings used until changed in the session

2. Provide the graphical parameter directly in the plotting function -> only for the specific plot

In R, you can easily combine multiple plots into one graph (figure) having a specific graphical layout:

1. In the par() function, use mfrow=c(nrows,ncols) or mfcol=c(nrows,ncols) to create a matrix and fill the plot with rows or by columns

2. Using the layout(matrix object) function where the location of the figures is specified in a matrix object

Text and symbol size

Text and symbol size in graphs can be controlled with cex = number indicating how text and symbols are scaled relative to

1=default

e.g. cex = 1.5 is 50% larger, cex = 0.5 is 50% smaller

The magnification of axis, labels, titles and subtitles can be set relative to cex using cex.axis, cex.lab, cex.main and cex.sub

To set the title of the plot use main = "Main title"

Página 9

Plotting lines

Line options can be changed with the line type lty= option and with the line width lwd= option (2 for double width)

Plotting symbols

The symbols to use for the plotting points can be specified with the pch= option

Additionally, for symbols 21 to 25, border color (col=) and fill color(bg=) can be specified

Axes

In many plotting functions axis and text options can be set

Use main="" and sub="" to include a title and subtitle

Use xlab and ylab to set the x-axis and y-axis labels

Use xlim=c() and ylim=c() to set the min and max values for the axes

Even more...

For even more finer control additional functions can be used called separately, just after the plotting function (not within)

title() in combination with text size, font, color and rotation

text() for text within the graph including labeling of points

mtext() for text in the margins

axis() to create custom axes

abline() to add reference lines to a plot, also grid() can be used

legend() to add a legend to a graph

Página 10

Font

Font size and style can be easily set with

font is an integer specifying the font to use:

1=plain, 2=bold, 3=italic, 4=bold italic, 5=symbolation

font.axis, font.lab, font.main font.sub

ps determines the font point size (text size=ps.cex)

family can be used to set the font family for drawing text

standard values are serif, sans, mono, or symbol

mapping is device dependent

Text and labels

Text can be added within a graph with text() or in one of the four margins with mtext():

adj specifies the x (and y) adjustment of the labels

pos is a position specific value of 1 to 4 indicating positions below, left, above or right of the coordinates respectively

side sets on which side of the plot the text should appear (1 to 4 like pos)

line means on which MARgin line, starting at 0 counting outwards text(x, y = NULL,

Reference lines

Add a reference line using abline(h=yvalues, v=xvalues)

Other graphical parameters (lty, lwd, col) can also be specified

Legend

For many figures, it is essential to add a compelling legend

Use legend(location, title, legend, ...)

The most important parameters are:

location: an x,y coordinate or keywords bottom, top, left, right, top left, top right, bottom left, bottom right, center

legend: character vector with the labels

PCH: vector of point symbols

bty for box type, bg for background color

Combining plots

Multiple plots can be combined into one figure using par() or layout()

par(mfrow=c(nrows, ncols)) creates a figure of rows x ncols filled by rows

par(mfcol=c(nrows, ncols)) creates a figure of nrows x ncols filled by columns

par(mar=c(bottom,left,top, right)) sets the margins of the plot in the number of lines of text (0.2 inches), default = c(5, 4, 4, 2) + 0.1

Use par(mai=c()) to set the margins in inches

Página 11

Layout

With the layout(matrix) function, the location of figures can be specified in a matrix object, e.g., matrix(c(1,1,2,3)) plots one figure in row 1 and two figures in row 2

To control the size of each figure, it is possible to use vectors of values for width and height, e.g., widths=c(3,1) -> column 1 = 3/4, column 2 = 1/4, and heights=c(1,2) --> row 1 = 1/3, row 2 = 2/3

Advanced layout control

To create a combined figure with fine control over the figure arrangementuse the fig=c(x1, x2, y1, y2) parameter setting the size with x and y coordinates

The full graph ->from the bottom left corner (0,0) to the top right corner (1,1)

fig= starts a new plot -> to add to an existing plot use new=TRUE

Colors

A good color scheme can make a big difference when presenting data.

R uses the hexadecimal system to represent colors with two characters: red, green, and blue (#rrggbb).

Two additional characters can be used to add transparency (#rrggbbaa).

Using colors in plots

In the previous color examples, the col parameter was used

In plots, you can additionally use the parameters col. Main, col. Sub, col. lab

Now, let’s make some plots again with colors!

Página 12

Scatter plot

A scatter plot pairs up values of two quantitative variables in a data set and display them as geometric points inside a Cartesian diagram.

Simple linear regression is a statistical method that allows us to summarizeand study relationships between two continuous (quantitative) variables.

On the scatter plot a regression line, the best-fitting straight line through the points, can be drawn to see the trend/relationship between the variables.

With the lm() function it is possible to generate a linear regression model of the two variables which can be drawn as a trend line with abline()

The lowess() function computes the LOWESS smoother (locally weighted scatterplot smoothing) and the lowess line can be added with lines()

Scatterplot matrix with pairs()

pairs() produces a matrix of scatterplots which shows all pairwise combinations of continuous variables in scatterplots

Parallel coordinates

Parallel coordinates can be used to visualize relationships between multiple variables

Each variable (column) corresponds to a vertical axis, and each data element (row) is displayed as a series of connected points (values of each variable) along the vertical axes

Bubble chart

A bubble chart compares three variables: the first on the x-axis, the second on the y-axis, and the third by the area size of the data points

It’s like a scatterplot with a third “bubble” dimension

Página 13

Stacked and grouped barplot

A stacked or grouped barplot can be used to plot two categorical variables

Instead of a table frequency of one categorical variables (e.g. iris$Species) use a table frequency of two categorical variables

Página 14

Exercises

Exercise 4.1

Load the urine data from a previous lecture.

Make two boxplots next to each other.

The first boxplot shows r (presence of calcium oxalate crystals) on x-axis as group (0 or 1) vs calc (calcium concentration) on y-axis without any other parameters set.

Now make a second similar boxplot but now specifiying title, outlier symbol x, titles for x- and y-axis and set the limit of the y-axis from 0 to 15.

Change the 0/1 group names on the x-axis to no/yes

Is there a difference in calcium concentration when crystals are present?

Exercise 4.2

Load the Arabidopsis dataset from the lme4 package

Look at the structure of this dataset: how many observations and variables?

How many factors and numeric variables?

Make one figure showing barplots for the different categorical variables, with title “Barplot variable name”, each bar in a different color (at least 9 colors i.e. max. levels) and axes set with labels and limit (ylim)

Make a second figure with histograms of the four numerical variable and add the density lines.

Which numerical variables can be set as factor because they have only a few possible numerical values?

Set them as.factor and make a new figure with barplots.

Can you add the frequencies above the bars?

Make a new figure with boxplots of the remaining numerical values per region with title, labels and colors.

Also make a scatterplot of two numerical values with title, colors and legend.

Each region in different color and symbol.

Página 15

Part 3

Dimensionality Reduction

When a dataset has many variables or features (high dimensionality), dimensionality reduction projects the data into a space with fewer dimensions to limit the (computational) complexity.

Dimensionality reduction is a technique for reducing the number of variables under consideration into a set of principal variables.

Frequently used methods include MDS, PCA, t-SNE, and UMAP.

Reducing the dimensionality can be performed linearly or non-linearly.

MultiDimensional Scaling (MDS)

Multidimensional scaling (MDS) is a visual representation of distances or similarities between sets of objects

More similar objects (having shorter distances) are closer together than less similar objects (having longer distances).

MDS can also serve as a dimension-reduction technique for high-dimensional data (Buja et al., 2007)

dist() computes and returns a distance matrix using a specified distance measure, which is used to compute the distances between the rows of a data matrix

The distance measure can be euclidean (default), maximum, manhattan,Canberra, binary, or minkowski

Euclidean is the average straight-line distance between two points in Euclidean space.

Página 16

Principal Component Analysis (PCA)

PCA is a linear transformation of the data which identifies the axes where the data has the most variance

PCA reduces the dimensions of a dataset into principal components (PCs)

PCs are ordered by decreasing the amount of variance

The default function from the base package is prcomp()

Many more methods are available, e.g., pcaMethods in Bioconductor

Center, scale, and sometimes log-transform to filter off trivial effects that could dominate the PCA

Página 17

t-SNE

t-SNE = t-distributed Stochastic Neighbor Embedding is a non-linear dimensionality reduction algorithm that can find patterns in data by identifying observed clusters based on similarity of data points with many features (but is not a clustering algorithm)

The perplexity parameter takes into consideration the number of close neighbors of each point and has a complex effect on the results

Analyze multiple plots with different perplexities

Typical values are between 5 and 50

Página 18

UMAP

Uniform Manifold Approximation and Projection is a novel manifold learning technique for dimension reduction.

UMAP algorithm is competitive with t-SNE for visualization quality

→ arguably preserves more of global structure, superior run time performance

→ no computational restrictions on the embedding dimension

→viable as general purpose dimension reduction technique for machine learning

Página 19

Exercises

Exercise 5.1

Load Taskesen2016-GTExdata.csv and Taskesen2016-SampleLabels.csv in R

The GTExdata contains 1641 rows (samples) and 16142 columns

Make a GTExdata subset with samples of only three clusters identified in the research article

,e.g., cluster 5 (Muscle), cluster 6 (Artery| Blood Vessel), and cluster 7 (Heart)

Use SampleLabels to get the rows for these samples in GTExdata

Set symbols and colors by Tissue.type

Make an MDS plot of the GTExdata subset

Make a tSNE plot of the GTExdata subset