Simple Linear Regression from Scratch!

Comparison of Various Gradient Descent Techniques [Image by Author]!

Using the already created libraries like sklearn or tensorflow or pytorch or any other is very easy, & just after learning using these libraries, giving yourself a title of Machine Learning Engineer or Deep Learning Engineer is a complete disaster/blunder. In reality, what makes you capable of giving this title to yourself is the amount of knowledge in that field, & here, knowledge doesn’t mean the ability to use the already created libraries, here, knowledge means the internal working of the algorithms, ability to create them from scratch. Once a person is able to identify the actual working of the algorithms, then that person can solve any custom problems that can appear to be unsolvable at the first sight, or there is no already built solution for that problem.

Pre-requisites for the best understanding of this blog:

  1. Basic Understanding of Machine Learning & its types (no in-depth intuition is required).
  2. Basic understanding of Gradient Descent as a whole, you should be aware of the weight & bias update process (no in-depth intuition is required). If you want to understand Gradient Descent as a whole in-depth, refer to my blog mentioned below:

Linear Regression

Linear Regression fitting the points [Image by Author]!

Cost/Loss Function used for all of the approaches

Mean Squared Implementation!

Linear Regression using Stochastic Gradient Descent

  1. The maximum number of iterations is reached.
  2. Error calculated is less than the set threshold (A threshold is set which means that if we are getting the error in the model prediction below/less than this number i.e., threshold, then we are satisfied. We do this because every time the error between the prediction & actual values can’t be 0).
Code for the Stochastic Gradient Descent Implementation from Scratch!

Linear Regression using Batch Gradient Descent

  1. The maximum number of iterations is reached.
  2. Error calculated is less than the set threshold (A threshold is set which means that if we are getting the error in the model prediction below/less than this number i.e., threshold, then we are satisfied. We do this because every time the error between the prediction & actual values can’t be 0).
Code for the Batch Gradient Descent Implementation from Scratch!

Linear Regression using Mini Batch Gradient Descent

  1. The maximum number of iterations is reached.
  2. Error calculated is less than the set threshold (A threshold is set which means that if we are getting the error in the model prediction below/less than this number i.e., threshold, then we are satisfied. We do this because every time the error between the prediction & actual values can’t be 0).
Code for the Mini Batch Gradient Descent Implementation from Scratch!

Comparison of all the Gradient Descent Approaches with an example

# Importing Standard Scaler
from sklearn.preprocessing import StandardScaler
# Generating the sample data
x = np.arange(50, 150)
# Transforming the data using Standard Scaler
x = StandardScaler().fit_transform(x)
# Generating the data for linear equation as y = 3x + 9
y = 3 * x + 9
# Converting features & target values into 2-D arrays, so that they can be passed into the above functions to calculate the weights & biases according to a particular Gradient Descent Algorithm.
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)
Sample data generated above is plotted [Image by Author]
Comparison of different Gradient Descent Approaches [Image by Author]!
Plotting line using the weight & bias calculated through the Batch Gradient Descent [Image by Author]!
Comparison of Various Gradient Descent Techniques [Image by Author]!

Each vertical line represents end of iterations required to calculate weights & biases for a particular Gradient Descent approach. Each algorithm loss/error showing line & line showing iteration ending is having the same colour.

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store