Fundamentals of Machine Learning

  Intro to Machine Learning

  • Main concepts of machine learning: learning from precedents (supervised), objects, features, answers, model algorithms, learning method, empirical risk, overfitting
  • Overfitting prevention: HoldOut, LeaveOneOut, CrossValidation
  • Chronology of significant events in machine learning
   Slides

  Matrix differentiation

  • Recap of function differentiation
  • Differential
  • Matrix differential properties
  • Hessian
   Slides

Homework 1: Gateway to Pandas

Understanding Pandas data structures: DataFrame and Series. Data summarizing, filtering, sorting. Simple implementation of your own TED Talks recommendation model to deepen your knowledge and proficiency with Pandas.

  Linear Models, Stochastic Gradient Descent

  • Linear models of regression and classification
  • The Stochastic Gradient (SG, SAG) method is suitable for any models and loss functions
  • Approximation of the threshold loss function
  • Regularization solves the multicollinearity problem and also reduces overfitting
  • Likelihood maximization and minimization of the empirical risk are different views on the same optimization problem
   Slides

  Linear models practice

  • Analytical and numerical approaches to solve linear regression
  • from sklearn.linear_model import LinearRegression
  • How to choose learning rate
   ipynb

  Logical Rules and Decision Trees

  • Logical regularity definition
  • Local rules searching and modifications, Pareto front
  • Decision Trees: definition, construction and using
   Slides

  Metric methods practice

  • K-nearest neighbors classifier (KNN)
  • Parzen Window Method
  • Potential Function Method
  • Nadaraya-Watson Estimator
   ipynb

Homework 2: Linear models theory

Ten theoretical tasks about logistic regression, decision trees and other models.

  Ensembles, gradient boosting and random forest

  • Simple and weighted voting, mixture of experts
  • Boosting, bagging, RSM
  • XGBoost, CatBoost, LightGBM
  • Random forest
   Slides

  Ensembles practice

  • Bagging, Boosting
  • GradBoost, XGoost, CatBoost, LightGBM
  • Random forest
   ipynb

Homework 3: Mini projects: Horror Trees or Bayes Guards SMS

  Intro to neural networks and backpropagation

  • Rise of neural networks
  • Expressive power of neural network
  • Backpropagation algorithm
   Slides

  Backpropagation practice

  • MNIST dataset
  • Computational graph
  • Micrograd by Andrej Karpathy
   ipynb

Homework 4: SGD and Simple Neural Network in NumPy

  Recurrent neural networks basics

  • Disadvantages of Feed-Forward Neural Networks
  • Architectures of Recurrent Networks
  • Vanilla RNN, LSTM, GRU
   Slides

  Building char-rnn from scratch

  • Data and random initialization
  • Cross entropy loss
  • Train and sample
   ipynb

  Intro to language modelling: bigrams

  • Makemore by Andrej Karpathy
  • Bigrams language modeling
   ipynb

  Intro to language modelling: Multi Layer Perceptron

  • MLP language model paper
  • Cross entropy loss
  • Some results
   ipynb

  Activations, Gradients, BatchNorm

  • Logits visualization and dead neurons
  • BatchNorm
  • Gradients and weights plots
   ipynb

  Building a WaveNet

  • PyTorchify previous code
  • Dilated causal convolutional layers
  • Brief preview of convolutions
   ipynb

  Convolutional Neural Networks

  • Brief history of computer vision
  • The progress of convolutional neural nets
  • Details of AlexNet model
   Slides

Convolutional NN practice

  • trainer(), train_epoch() and train_on_batch
  • LeNet architecture
  • Layers visualization, tensorboard and lime
   ipynb

Homework 5: Image classification task on Kaggle

  Attention mechanism and transformer architecture

  • Attention for machine translation and annotating images
  • Architecture of a Transformer model
  • BERT and masked language modeling
   Slides

  Attention practice

  • Gathering parallel texts
  • Attention in RNN
  • Transformer model
   ipynb

Homework 6: Backprop in Numpy and Implementing GPT-2

  Building GPT from scratch

  • Attention is all you need
  • Math trick in self-attention
  • Layer normalization and dropout
   ipynb

  Tokenization

  • We must tokenize information for computer
  • Byte-pair encoding algorithm
  • Sentencepiece BPEs (Llama and Mistral series)
   ipynb

  Bayesian methods intro

  • Conditional probability and Bayes' theorem
  • Comparison of Frequentist and Bayesian approaches
  • Markov Chain Monte Carlo (MCMC) and Gibbs sampling
   Slides

  Bayesian methods practice

  • Conjugate distributions
  • Maximum Likelihood Estimation
  • Metropolis-Hastings algorithm
   ipynb

If you want to use the materials (e.g., figures) in your paper/report and to cite this course, you can do this using the following BibTex:

@misc{avalur2024mlCourse,
title={Fundamentals of ML},
url={https://avalur.github.io/ml_fundament.html},
author={Alexander Avdiushenko},
year={2024},
month={Sep}
}