Generative classifiers

A Generative Model ‌explicitly models the actual distribution of each class. A Generative Model ‌learns the **joint probability distribution p(x,y). **It predicts the conditional probability with the help of Bayes Theorem.

  • Assume some functional form for P(Y), P(X|Y)
  • Estimate parameters of P(X|Y), P(Y) directly from training data
  • Use Bayes rule to calculate P(Y |X)

Examples:

  • ‌Naïve Bayes
  • Bayesian networks
  • Markov random fields
  • ‌Hidden Markov Models (HMM)

Discriminative classifiers

A Discriminative model ‌models the decision boundary between the classes. A Discriminative model ‌learns the conditional probability distribution p(y|x).

  • Assume some functional form for P(Y|X)
  • Estimate parameters of P(Y|X) directly from training data

Examples:

  • ‌Logistic regression
  • Scalar Vector Machine
  • ‌Traditional neural networks
  • ‌Nearest neighbour
  • Conditional Random Fields (CRF)s

Some test questions

  1. ‌ **What are the problems these models can solve? **Classification tasks.
  2. **Which model learns joint probability? **The generative model.
  3. ‌**Which model learns conditional probability? **the discriminative model.
  4. **What happens when we give correlated features in discriminative models? **They will produce similar output as they are correlated?
  5. **What happens when we give correlated features in generative models? **We can’t tell as we do not learn the decision boundaries?
  6. **Which models works very well even on less training data? **The discriminative model will work very well as only one set of parameter has to be learned?
  7. Is it possible to generate data from with the help of these models? Don’t know.
  8. Which model will take less time to get trained? The discriminative model will take less time as only one set of parameter has to be learned?
  9. **Which model will take less time to predict output? **The discriminative model will take less time as prediction can be performed directly by the learned conditional probability. The generative model additionaly needs to apply Bayes’ Theorem.
  10. **Which model fails to work well if we give a lot of features? **The generative model?
  11. **Which model prone to overfitting very easily? **The generative model?
  12. ‌**Which model prone to underfitting easily? **The discriminative model?
  13. ‌**What happens when training data is biased over one class in Generative Model? **The joint probability will be biased as class probabilites will be biased. Thus, the conditional probability will be biased, too.
  14. **What happens when training data is biased over one class in Discriminative Models? **It happens the same like in the generative case. We will receive biased conditional probability.
  15. **Which model is more sensitive to outliers? **The discriminative model is more sensitive to outliers as we model the boundaries. Each outlier will shift the class boundaries.
  16. ‌**Are you able to fill out the missing values in a dataset with the help of these models? **Yes — if the dataset offers features.

Reference