Main Crux of Machine Learning

We need to predict a variable $\mathbf{y}$ in which a model needs to be defined which shows the relationship between output $\mathbf{y}$ and input $\mathbf{x}$. Once the parameters $\{\theta\}$ of the model is refined iteratively the output $\mathbf{y}$ is inferred.

Components of machine learning

1. $\mathbf{y}$ → tensor/vector (depending on the model) of output (inference/prediction)

2. $\mathbf{x}$ → tensor/vector (depending on the model) of input

3. “model” → relationship between $\mathbf{x}$ and $\mathbf{y}$

   $$ \mathbf{y} = f(\mathbf{x},\{\theta\}) $$

4. “parameters” → how the $\mathbf{x}$ is scaled using set of $\{\theta\}$

Here $\mathbf{x}$ and $\mathbf{y}$ values can range from single valued variable to multidimensional tensors depending on the problem at work.

1. The term ‘learning’ or ‘training’ refers to the optimization of $\{\theta\}$ on a pre-derived or gathered ‘dataset’ $\{x_i , y_i\}$, in our case we would assume the total number of datapoints is $N$
2. To know whether one has found good $\{\theta\}$, a ‘Loss Function’ needs to be defined that would dictate how close you are to the true relationship of $\mathbf{x}$ and $\mathbf{y}$. The general form can look as

$$ L(\{\theta\}) = \sum_{i=1}^N [f(x_i,\{\theta\}) - y_i]^2
$$

1. The objective of the machine learning is to optimize the loss function or cost function.

So before tackling a machine learning algorithm we need to divide it in 2 parts

• gathering the dataset
• finding a model
• finding a good algorithm for optimization of loss function - optimization algorithm
  • gradient descent
  • BFGS
  • stochastic gradient descent
  • ADAM
  • etc.

The discussion for dataset gathering is beyond this lecture, so we would confine our discussion to model and optimization algorithm

Beginner Model

Linear Regression

I hope you all have done least squares fit in your school or colleges. So one might try to think what the difference between regression and machine learning. In least squares fit we mainly consider closed form of solution. See the following appendices for details.

<aside> 🗒️

What is a closed form solution?

→ A closed form solution is an exact solution to an equation using a finite number of mathematical operations and functions.

</aside>

Appendices :

Closed Form

Linear Regression

So, in this case our model or the relationship between $\mathbf{x}$ and $\mathbf{y}$ is