Machine Learning Lesson of the Day – Parametric vs. Non-Parametric Models

A machine learning algorithm can be classified as either parametric or non-parametric.

A parametric algorithm has a fixed number of parameters.  A parametric algorithm is computationally faster, but makes stronger assumptions about the data; the algorithm may work well if the assumptions turn out to be correct, but it may perform badly if the assumptions are wrong.  A common example of a parametric algorithm is linear regression.

In contrast, a non-parametric algorithm uses a flexible number of parameters, and the number of parameters often grows as it learns from more data.  A non-parametric algorithm is computationally slower, but makes fewer assumptions about the data.  A common example of a non-parametric algorithm is K-nearest neighbour.

To summarize, the trade-offs between parametric and non-parametric algorithms are in computational cost and accuracy.

6 Responses to Machine Learning Lesson of the Day – Parametric vs. Non-Parametric Models

  1. Reblogged this on victoria anugrah and commented:
    It clears my mind about non-parametric methods.🙂

  2. Krupali says:

    Pre-exam day and your easy explanation! Kudos.

  3. Alpha Pi says:

    I would like to point out that “linear regression” is not an algorithm, it’s a widely known technique in statistics and not machine learning. We could would be talking about “machine learning” in linear regression if we were to perform parameter estimation with something like a genetic algorithm instead of the common OLS method. But since this is not specified, it’s incorrect to regard linear regression as machine learning. Generally, there are no true parametric algorithms in machine learning, and that’s what differentiates it to statistics. And before someone mentions Artificial Neural Networks, it should be clear that they are NOT parametric in the statistical sense because their parameters do not explain anything about the model and their model is a black box.

Your thoughtful comments are much appreciated!

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: