Sorting correlation coefficients by their magnitudes in a SAS macro

Theoretical Background

Many statisticians and data scientists use the correlation coefficient to study the relationship between 2 variables.  For 2 random variables, X and Y, the correlation coefficient between them is defined as their covariance scaled by the product of their standard deviations.  Algebraically, this can be expressed as

\rho_{X, Y} = \frac{Cov(X, Y)}{\sigma_X \sigma_Y} = \frac{E[(X - \mu_X)(Y - \mu_Y)]}{\sigma_X \sigma_Y}.

In real life, you can never know what the true correlation coefficient is, but you can estimate it from data.  The most common estimator for \rho is the Pearson correlation coefficient, which is defined as the sample covariance between X and Y divided by the product of their sample standard deviations.  Since there is a common factor of

\frac{1}{n - 1}

in the numerator and the denominator, they cancel out each other, so the formula simplifies to

r_P = \frac{\sum_{i = 1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i = 1}^{n}(x_i - \bar{x})^2 \sum_{i = 1}^{n}(y_i - \bar{y})^2}} .

 

In predictive modelling, you may want to find the covariates that are most correlated with the response variable before building a regression model.  You can do this by

  1. computing the correlation coefficients
  2. obtaining their absolute values
  3. sorting them by their absolute values.

Read more of this post

Advertisements

University of Toronto Statistical Sciences Union Career Panel

I am delighted to be invited to speak at the University of Toronto Statistical Sciences Union’s first ever Career Panel.  If you plan to attend this event, I encourage you to read my advice columns on career development in advance.  In particular, I strongly encourage you to read the blog post “How to Find a Job in Statistics – Advice for Students and Recent Graduates“.  I will not cover all of the topics in these columns, but you are welcomed to ask questions about them during the question-and-answer period.

Here are the event’s details.

Time: 1 pm to 6 pm

  • My session will be held from 5pm to 6 pm.

Date: Saturday, March 25, 2017

Location: Sidney Smith Hall, 100 St. George Street, Toronto, Ontario.

  • Sidney Smith Hall is located on the St. George (Downtown) campus of the University of Toronto.
  • Update: The seminars will be held in Rooms 2117 and 2118.  I will speak in Room 2117 at 5 pm.

 

If you will attend this event, please feel free to come and say “Hello”!

Analytical Chemistry Lesson of the Day – Accuracy in Method Validation and Quality Assurance

In pharmaceutical chemistry, one of the requirements for method validation is accuracy, the ability of an analytical method to obtain a value of a measurement that is close to the true value. There are several ways of assessing an analytical method for accuracy.

  1. Compare the value from your analytical method with an established or reference method.
  2. Use your analytical method to obtain a measurement from a sample with a known quantity (i.e. a reference material), and compare the measured value with the true value.
  3. If you don’t have a reference material for the second way, you can make your own by spiking a blank matrix with a measured quantity of the analyte.
  4. If your matrix may interfere with the analytical signal, then you cannot spike a blank matrix as described in the third way.  Instead, spike your sample with an known quantity of the standard.  I elaborate on this in a separate tutorial on standard addition, a common technique in analytical chemistry for determining the quantity of a substance when matrix interference exists.  Standard addition is an example of the second way of assessing accuracy as I mentioned above.  You can view the original post of this tutorial on the official JMP blog.