In this blog post, we’ll write code to demonstrate how gradient descent works.
Let’s first write the whole code from scratch without using Keras so that you can link the underlying math to code. This is imperative for a deeper understanding of the concepts. Once we’re through that, we’ll solve the same problem using Keras and see what improvements are made by using the library.
Let’s begin by setting up our environment for implementing gradient descent. Fire up a Google Colaboratory notebook. We’ll break the code down so we can point out all the important aspects. We’ll be using the numpy library to write vectorized code.
Notice the second line in the listing below. You’ll often see this in machine learning codes.
import numpy as np
np.random.seed(1337) # Generate dummy data
This line sets the random seed that is used to generate all random numbers in the following code. If any piece of code starts with the same seed, the “random” numbers generated throughout the code are still going to be random but the setting of the seed ensures that every time you run the code, you’ll get the same random numbers. This ensures the reproducibility of your results.
When we write np.random.seed(1337), we’re not generating random numbers—we’re creating a predetermined sequence that appears random but follows a specific pattern. This sequence influences everything from how we initialize our model weights to how we shuffle our training data, making it possible to recreate exact experimental conditions whenever needed. The number itself (whether it’s 1337, 42, or any other integer) isn’t special, but using the same seed guarantees that our random number generator will produce identical sequences every time, which is crucial for debugging, testing, and ensuring our results are reliable and reproducible.
You can access the complete code listing for this post here.
The way we use random seeds changes depending on our goals: during development and research, we typically want consistent results to compare different approaches and debug issues, so we set specific seeds to create reproducible environments. This way, we can share our code with others who can run it and make sure they get the same results as we did. However, in production systems where we want our models to be robust and adaptable, we often skip setting seeds to embrace true randomness. This duality reflects a broader principle in machine learning: We need controlled environments to develop and validate our models, but we also need to ensure that they can handle the genuine randomness of real-world data. The power of random seeds lies in this flexibility—they give us the ability to switch between perfectly reproducible experiments and truly random operations.
With that out of the way, let’s proceed to generate some dummy data using numpy’s built-in functions. The first line in the listing below creates 100 random samples, and the second line defines the ground truth for the model parameters θ0 and θ1.
X = np.random.randn(100, 1) # 100 samples, 1 feature
true_params = np.array([2.5, 3.7]) # True param values
y = true_params[0] + true_params[1] * X
+ np.random.normal(0, 0.1, (100, 1))
print("Shape of X:", X.shape, " Shape of y:", y.shape)
We then go ahead and generate the output values y based on the inputs and also add a sprinkle of random noise to make our data more realistic. After all, in the real world, patterns are rarely perfect. The last line shows the shape of the inputs and outputs. This is one of the most important aspects of real-world engineering of machine learning models. At the moment, the shape of X and y is both (100, 1). This means 100 rows with 1 column each. It’s a good idea to get in the habit of looking at shapes of different matrices in your code. As we venture into more sophisticated models with multiple features and complex architectures, keeping track of these matrix dimensions becomes increasingly important. Matrix shape mismatches are often the hidden culprits behind many debugging sessions, so developing an intuition for them early on is invaluable.
We then move on to complete our setup before we start with the actual training process. The first line in the next listing adds a column to our input matrix, converting it to the shape (100, 2). The first column is populated with all 1s using the np.ones function, and the np.hstack function does the heavy lifting of producing the final matrix in the correct form.
X_b = np.hstack((np.ones((X.shape[0], 1)), X)) # set x0=1
print("Shape of X_b:", X_b.shape) # look at the shape again
# Initialize parameters
theta = np.random.randn(2, 1)
learning_rate = 0.01
n_iterations = 1000
batch_size = 32 # Mini-batch size
We then go ahead and initialize our model parameters with some random values. We use randn(2, 1) because we have two model parameters and need just one combination. Finally, we set the hyperparameters for learning_rate and batch_size along with the definition of the number of iterations we need for the gradient descent step. Now, we’re ready to go ahead and perform the actual gradient descent steps and improve our initial random guess.
We’ll first shuffle the data as shown in the first three lines of this listing. This helps us do
the randomization once and then sample batches in order instead of randomly sampling
them during each step.
indices = np.random.permutation(len(X)) # Shuffle the data
X_b = X_b[indices]
y = y[indices]
for i in range(n_iterations):
# Get current batch indices
start_idx = (i * batch_size) % len(X)
end_idx = min(start_idx + batch_size, len(X))
# Get batch data
X_batch = X_b[start_idx:end_idx]
y_batch = y[start_idx:end_idx]
y_pred = X_batch @ theta # Forward pass on batch
# Compute gradients using batch
gradients = (2/len(X_batch)) *
X_batch.T @ (y_pred - y_batch)
theta -= learning_rate * gradients # actual step
print("Final parameters:", theta.flatten())
print("True parameters:", true_params)
Then, we start the loop that performs the gradient descent step iteratively. In the step, we calculate the start and end index of the batch so that the number of data points in our batch is equal to the batch_size hyperparameter. The line start_idx = (i * batch_size) % len(X) calculates where each batch should begin in our dataset through a clever use of the modulo operator. As we iterate through our data, we multiply the current iteration by our batch size to determine where we theoretically should start, but because we might exceed the length of our dataset, we use the modulo operator to wrap back around to the beginning when needed. For example, if we have 100 samples and a batch size of 32, our first batch would start at index 0, the second at 32, the third at 64, and the fourth at 96. When we try to calculate the start of the fifth batch, we’d get 128, but because that’s larger than our dataset size of 100, the modulo operator gives us the remainder after division (28), effectively wrapping us back to continue processing from the 28th sample. This creates a continuous cycle through our data, allowing us to process multiple epochs while ensuring we use every sample in our dataset.
Once we have the batch ready in X_batch and y_batch, we simply perform what is called the forward pass, that is, carrying our matrix-vector multiplication of the features in X_batch with the model parameters theta. Numpy gives us a built-in syntax for performing this very common operation—the @ symbol. Applying this operation gives us the predicted values for features. We then calculate the loss using a derivative formula.
After computing the gradients, we can simply apply the gradient descent step. As you can see, the vector theta is being updated based on the learning rate and the gradients. An important point to note here is that numpy automatically takes care of simultaneous updates. Using vectorized code gives us this additional benefit along with the speed up.
Finally, after the specified iterations have been completed, we output the computed values as well as ground truth model parameters. Run the code and play around with different values so that you understand it well. Afterwards, we can go ahead and see how using Keras would make our life easy when solving the same problem using this extremely well-written library.
Once you’ve fully understood the from-scratch code, we can go ahead and convert the code to use Keras instead. We’ll begin by importing the required packages and then setting up our data. You can access the code listing at this page under the 03-02-SGD-keras notebook.
This listing is very similar to what we had before, except we have the Keras imports that we didn’t have earlier. In addition, compare the data setup we did with what we have here. When using Keras, we don’t have to set up the bias term because Keras will take care of it for us automatically. Next, we’ll set up the Keras “model.”
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
# Generate dummy data
np.random.seed(42)
X = np.random.randn(100, 1) # 100 samples, 1 feature
true_params = np.array([2.5, 3.7]) # True parameters [θ₀, θ₁]
y = true_params[0] + true_params[1] * X + np.random.normal(0, 0.1, (100, 1))
# Shuffle the data
indices = np.random.permutation(len(X))
X = X[indices]
y = y[indices]
The specification of the Keras model that will carry out the learning for us is given in the next listing. A Keras model is a structure that takes numerical inputs and produces numerical outputs through a series of mathematical operations. At its most basic level, a model consists of layers, where each layer performs specific mathematical transformations on the data that passes through it. These transformations involve matrix operations such as multiplication and addition, along with optional nonlinear mathematical functions. This is exactly what we need for our gradient descent. The input layer is Shape(1, ). That means it will take input where each data point will have just one feature. The trailing comma isn’t a typing mistake—it turns the number into a tuple as required by Keras (we’ll cover this further later). The Dense(1, use_bias=True) line means that we’ll output a single value from this model (i.e., our prediction) and that Keras should automatically take care of the bias term for us. Again, we’ll be discussing this in much more detail when we get to the use of Keras for much larger models. For now, this model is specified using these two lines and then compiled.
The model.compile() function sets up two essential components for training: the optimizer (controls how the model updates its internal parameters) and the loss function (measures how wrong the model’s predictions are). It’s like configuring the training rules before the actual training begins.
# Create and compile the model
model = keras.Sequential([
layers.Input(shape=(1,)), # Input layer for 1 feature
layers.Dense(1, use_bias=True) # Linear layer with bias
])
# Configure the training
model.compile(
optimizer=keras.optimizers.SGD(learning_rate=0.01),
loss=‘mse’ # Mean squared error
)
There are many more optimizers available in Keras. All of them have gradient descent at their core but each one adds different optimizations. It’s good to experiment with them and see what difference they make. You can find more about the available optimizers here.
Finally, we can start the actual training process, as shown in this listing.
# Train the model
history = model.fit(
X, y,
batch_size=32,
epochs=1000,
)
# Get final parameters
weights = model.get_weights()
print("Final parameters:", [weights[0][0][0], weights[1][0]])
print("True parameters:", true_params)
This final bit goes ahead and actually performs the training based on the model hyperparameters specified earlier. You’ll notice that we don’t actually have a loop in here. The reason is that Keras will automatically take care of parallelization of our code and make any optimizations that are possible given our underlying environment. Once the whole training is done, which might take a bit of time, Keras will output the weights using the last two lines. The exact semantics of the shape of the weight variables will be discussed once we’ve looked at more complicated models. For now, just know that you’ll get the value out first and then the value instead of the other way around.
With that, you’re now equipped with a thorough understanding of one of the most important pillars of modern machine learning—gradient descent.
Editor’s note: This post has been adapted from a section of the book Keras 3: The Comprehensive Guide to Deep Learning with the Keras API and Python by Mohammad Nauman. Dr. Nauman is a seasoned machine learning expert with more than 20 years of teaching experience and a track record of educating 40,000+ students globally through his paid and free online courses on platforms like Udemy and YouTube.
This post was originally published 7/2026.