Neural Networks with TensorFlow/Keras¶
Here I walk you through building simple feedforward neural networks with Keras for two classic tasks:
- Classification — predicting a 0/1 label
- Regression — predicting a continuous numeric value
The goal isn’t to build a state-of-the-art model — it’s to understand, step by step, how a neural network is constructed, trained, and evaluated using a tiny, hand-crafted dataset where we know the underlying rule. This makes it much easier to see whether the network is actually learning something sensible.
How to use this notebook in class: run each cell in order. Try changing numbers (epochs, neurons, learning rate) and re-running to build intuition.
Part 1 — Classification Model¶
The problem we’re solving¶
We want to predict a binary label (0 or 1) from 4 binary input features.
Rule used to generate the data: if two or more of the four inputs are 1, the label is 1; otherwise the label is 0.
The network never sees this rule — it only sees examples (X_train, y_train) and has to discover the pattern from data. This is the essence of supervised learning.
The network architecture¶
Input Layer
x1 x2 x3 x4 (4 features)
│
┌──────────────┐
│ Hidden Layer │ 2 neurons, ReLU activation
│ 2 Neurons │
└──────────────┘
│
┌──────────────┐
│ Hidden Layer │ 3 neurons, ReLU activation
│ 3 Neurons │
└──────────────┘
│
Output Layer
1 Neuron
(Sigmoid → 0/1)
- ReLU (Rectified Linear Unit) is used in the hidden layers — it’s the standard default activation because it’s simple and helps networks learn non-linear patterns efficiently.
- Sigmoid is used on the output layer because it squashes the output into the range (0, 1), which we interpret as a probability of the label being 1. This is the standard choice for binary classification.
Step 1 — Create the training data¶
Each row of X_train is one training example with 4 binary features. y_train holds the corresponding correct label, generated by the “two or more 1s → label 1” rule described above.
This is a deliberately tiny dataset (only 10 examples) so it’s easy to read and reason about — real-world datasets would typically have thousands or millions of examples.
import importlib.metadata
# List the distribution package names
packages = ["tensorflow",]
for package in packages:
try:
version = importlib.metadata.version(package)
print(f"{package} version: {version}")
except importlib.metadata.PackageNotFoundError:
print(f"{package} is not installed in this environment.")
tensorflow version: 2.15.0
import numpy as np
# The dataset is created with the following simple rule:
# If two or more input values are 1, then the label is 1. Otherwise, the label is 0.
X_train = np.array([
[0,0,0,0],
[0,0,0,1],
[0,0,1,0],
[0,1,0,0],
[1,0,0,0],
[1,1,0,0],
[1,1,1,0],
[1,1,1,1],
[0,1,1,1],
[1,0,1,1]
], dtype=float)
# Binary labels
y_train = np.array([
0,
0,
0,
0,
0,
1,
1,
1,
1,
1
], dtype=float)
Step 2 — Build and compile the model¶
We use Keras’s Sequential API, which lets us stack layers one after another — perfect for simple feedforward networks like this one.
Densemeans fully connected: every neuron in this layer is connected to every neuron in the previous layer.input_shape=(4,)tells the first layer to expect 4 input features.model.compile(...)configures how the model will learn:optimizer='adam'— the algorithm that updates the network’s weights during training (Adam is a popular, well-behaved default).loss='binary_crossentropy'— the function the optimizer tries to minimize; it’s the standard loss for binary (0/1) classification problems.metrics=['accuracy']— an easy-to-interpret number we’ll track alongside the loss, but it does not directly drive training.
model.summary()prints the architecture: layer names, output shapes, and the number of trainable parameters (weights + biases) in each layer.
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
# Two equivalent ways to define the same architecture are shown below.
# The list-based version (commented out) is more compact; the .add() version
# below is more explicit and easier to follow step-by-step in class.
# model = Sequential([
# Dense(2, activation='relu', input_shape=(4,), name="Hidden_Layer_1"),
# Dense(3, activation='relu', name="Hidden_Layer_2"),
# Dense(1, activation='sigmoid', name="Output_Layer")
# ])
# 1. Initialize a blank model
model = Sequential()
model.add(Dense(units=2, activation='relu', input_shape=(4,), name="Hidden_1"))
model.add(Dense(units=3, activation='relu', name="Hidden_2"))
model.add(Dense(units=1, activation='sigmoid', name="Output"))
# Compile the model: tell Keras how it should learn
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy']
)
# Display model architecture: layers, shapes, and parameter counts
model.summary()
Model: "sequential_4"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Hidden_1 (Dense) (None, 2) 10
Hidden_2 (Dense) (None, 3) 9
Output (Dense) (None, 1) 4
=================================================================
Total params: 23 (92.00 Byte)
Trainable params: 23 (92.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Step 3 — Visualize the architecture¶
plot_model renders the network graphically (saved to junk.png), which is often a more intuitive way to “see” the layer-by-layer structure than reading the text summary above.
Note: this requires the
pydotandgraphvizpackages to be installed; if it errors out, the textmodel.summary()above already tells you everything you need.
from tensorflow.keras.utils import plot_model
plot_model(model, to_file='junk.png', show_shapes=True, show_layer_names=True)
Step 4 — Train the model¶
model.fit() is where the actual learning happens. The model makes a prediction, compares it to the true label using the loss function, and adjusts its internal weights slightly to reduce the error — repeating this process for every example, over and over.
epochs=700— one epoch means the model has seen the entire training set once. 700 epochs means it sees the (tiny) dataset 700 times. Small datasets typically need many epochs to learn well.verbose=0suppresses the per-epoch progress output to keep the notebook tidy; the training history is still recorded in thehistoryobject so we can plot it afterward.
Discussion point: what happens if you reduce epochs to, say, 50? Try it and see whether the model still learns the rule.
# Train the model
history = model.fit(
X_train,
y_train,
epochs=700,
verbose=0
)
print("Training Complete!")
Training Complete!
Step 5 — Test on unseen data and evaluate¶
Now we check whether the model generalizes, i.e. whether it correctly predicts labels for examples it has never seen during training (X_test). This is the real test of learning — a model that just memorizes the training set isn’t actually useful.
For each test example we print:
- the raw predicted probability (output of the sigmoid, between 0 and 1)
- the predicted class, obtained by thresholding the probability at 0.5
- the actual class, for comparison
Finally, model.evaluate() reports the loss and accuracy on the test set as a single summary number.
# Test Data
X_test = np.array([
[1,1,0,1],
[0,0,1,1],
[1,0,0,0],
[1,1,1,1]
], dtype=float)
# Actual answers
y_test = np.array([1,0,0,1])
# Prediction
predictions = model.predict(X_test)
print("\nResults\n")
for i in range(len(X_test)):
predicted_probability = predictions[i][0]
predicted_class = 1 if predicted_probability >= 0.5 else 0
print(f"Input : {X_test[i]}")
print(f"Actual Class : {y_test[i]}")
print(f"Predicted Prob. : {predicted_probability:.4f}")
print(f"Predicted Class : {predicted_class}")
print("-"*40)
# Evaluate overall performance on the test set
loss, accuracy = model.evaluate(X_test, y_test, verbose=0)
print(f"\nTest Accuracy : {accuracy:.2f}")
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x000002AA32100040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. 1/1 [==============================] - 0s 71ms/step Results Input : [1. 1. 0. 1.] Actual Class : 1 Predicted Prob. : 0.6271 Predicted Class : 1 ---------------------------------------- Input : [0. 0. 1. 1.] Actual Class : 0 Predicted Prob. : 0.8405 Predicted Class : 1 ---------------------------------------- Input : [1. 0. 0. 0.] Actual Class : 0 Predicted Prob. : 0.3345 Predicted Class : 0 ---------------------------------------- Input : [1. 1. 1. 1.] Actual Class : 1 Predicted Prob. : 0.9462 Predicted Class : 1 ---------------------------------------- WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_test_function.<locals>.test_function at 0x000002AA32102980> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Test Accuracy : 0.75
Step 6 — Visualize training progress¶
Plotting the loss and accuracy over epochs helps us see how training went, not just the final result:
- Loss curve should generally trend downward — if it’s flat from the start, the model may not be learning (e.g. learning rate too low); if it’s very noisy or increasing, something may be wrong (e.g. learning rate too high).
- Accuracy curve should generally trend upward toward 1.0 as the model gets better at correctly classifying training examples.
Discussion point: because we only have 10 training examples, these curves can look a bit “jumpy” compared to curves from larger, real-world datasets.
import matplotlib.pyplot as plt
# Plot Loss
plt.figure(figsize=(6,4))
plt.plot(history.history['loss'], linewidth=2)
plt.title("Training Loss")
plt.xlabel("Epoch")
plt.ylabel("Binary Crossentropy Loss")
plt.grid(True)
plt.show()
# Plot Accuracy
plt.figure(figsize=(6,4))
plt.plot(history.history['accuracy'], linewidth=2)
plt.title("Training Accuracy")
plt.xlabel("Epoch")
plt.ylabel("Accuracy")
plt.grid(True)
plt.show()
Part 2 — Regression Model¶
The problem we’re solving¶
Instead of predicting a category (0/1), we now want to predict a continuous numeric value. This is called regression.
We’ll generate target values using a known linear formula so we can sanity-check whether the network is learning something reasonable:
$$y = 2x_1 + 3x_2 + x_3 + 4x_4$$
The network architecture is almost identical to the classification model — same number of layers and neurons — but with two key differences we’ll point out below: the output activation and the loss function.
Step 1 — Create the training data¶
X_train holds the 4 input features per example, and y_train holds the target numeric value computed (approximately) from the formula above. Unlike classification, there’s no thresholding here — these are real, continuous numbers.
import numpy as np
#Training Dataset
X_train = np.array([
[1,2,3,1],
[2,1,1,2],
[3,2,2,3],
[4,2,3,4],
[5,3,2,5],
[6,3,4,6],
[7,4,3,7],
[8,4,5,8],
[9,5,4,9],
[10,5,5,10]
], dtype=float)
y_train = np.array([
15,
16,
26,
33,
39,
51,
58,
65,
73,
80
], dtype=float)
Step 2 — Build and compile the model¶
Compare this to the classification model — notice the two important changes:
- Output activation is
'linear'instead of'sigmoid'. Linear (i.e. “no squashing”) activation lets the output take any real number, which is what we need for regression — a sigmoid would incorrectly squash our predictions into the 0–1 range. - Loss is
'mean_squared_error'instead of'binary_crossentropy'. MSE measures the average squared difference between predicted and actual values — the natural choice for regression. We also track'mae'(Mean Absolute Error) as a more interpretable metric: “on average, how far off are our predictions, in the original units?” - We also explicitly set a learning rate of
0.01for the Adam optimizer (instead of using its default), which controls how big a step the optimizer takes when updating weights each time. Smaller learning rates train more slowly but more carefully; larger ones train faster but risk overshooting.
# Build Model
# model = Sequential([
# Dense(2, activation='relu', input_shape=(4,), name="Hidden_Layer_1"),
# Dense(3, activation='relu', name="Hidden_Layer_2"),
# Dense(1, activation='linear', name="Output_Layer")
# ])
model = Sequential()
model.add(Dense(units=2, activation='relu', input_shape=(4,), name="Hidden_Layer_1"))
model.add(Dense(units=3, activation='relu', name="Hidden_Layer_2"))
model.add(Dense(units=1, activation='linear', name="Output_Layer"))
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
loss='mean_squared_error',
metrics=['mae']
)
model.summary()
Model: "sequential_5"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Hidden_Layer_1 (Dense) (None, 2) 10
Hidden_Layer_2 (Dense) (None, 3) 9
Output_Layer (Dense) (None, 1) 4
=================================================================
Total params: 23 (92.00 Byte)
Trainable params: 23 (92.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Step 3 — Visualize the architecture¶
Same idea as before: a visual diagram of the layers, saved to junk.png.
from tensorflow.keras.utils import plot_model
plot_model(model, to_file='junk.png', show_shapes=True, show_layer_names=True)
Step 4 — Train the model¶
Same model.fit() pattern as in the classification example. Here we use epochs=1000 since regression on this small, noisy-ish dataset benefits from more training iterations to converge well.
# Train
history = model.fit(
X_train,
y_train,
epochs=1000,
verbose=0
)
print("Training Complete!")
Training Complete!
Step 5 — Plot training history, then test and evaluate¶
First we plot the MSE loss and MAE curves over training — both should trend downward as the model improves.
Then, just like in Part 1, we test the trained model on unseen data (X_test) and compare its predictions to the true values (y_test), finishing with overall Test MSE and Test MAE scores.
Discussion point: MAE is often easier to explain to a non-technical audience than MSE, because it’s in the same units as the target variable (e.g. “off by 2.3 on average”), whereas MSE is in squared units.
# Plot History
plt.figure(figsize=(10,4))
plt.subplot(1,2,1)
plt.plot(history.history['loss'])
plt.title("Training Loss")
plt.xlabel("Epoch")
plt.ylabel("MSE")
plt.subplot(1,2,2)
plt.plot(history.history['mae'])
plt.title("Mean Absolute Error")
plt.xlabel("Epoch")
plt.ylabel("MAE")
plt.tight_layout()
plt.show()
# Test Data
X_test = np.array([
[1.5, 2, 2, 1],
[2.5, 2, 3, 2],
[4.5, 3, 3, 4],
[6.5, 4, 4, 6],
[8.5, 5, 5, 8]
], dtype=float)
# Actual prices
y_test = np.array([
15,
22,
37,
53,
69
], dtype=float)
# Prediction
predictions = model.predict(X_test, verbose=0)
print("\nPrediction Results")
print("-"*60)
for features, actual, predicted in zip(X_test, y_test, predictions):
print(f"Features : {features}")
print(f"Actual Value : {actual:.2f}")
print(f"Predicted Value : {predicted[0]:.2f}")
print("-"*60)
loss, mae = model.evaluate(X_test, y_test, verbose=0)
print(f"\nTest MSE : {loss:.2f}")
print(f"Test MAE : {mae:.2f}")
WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_predict_function.<locals>.predict_function at 0x000002AA345AAB60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Prediction Results ------------------------------------------------------------ Features : [1.5 2. 2. 1. ] Actual Value : 15.00 Predicted Value : 14.20 ------------------------------------------------------------ Features : [2.5 2. 3. 2. ] Actual Value : 22.00 Predicted Value : 22.18 ------------------------------------------------------------ Features : [4.5 3. 3. 4. ] Actual Value : 37.00 Predicted Value : 36.85 ------------------------------------------------------------ Features : [6.5 4. 4. 6. ] Actual Value : 53.00 Predicted Value : 52.42 ------------------------------------------------------------ Features : [8.5 5. 5. 8. ] Actual Value : 69.00 Predicted Value : 67.99 ------------------------------------------------------------ WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_test_function.<locals>.test_function at 0x000002AA34614E00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. Test MSE : 0.41 Test MAE : 0.54
Recap & Discussion Questions¶
| Classification | Regression | |
|---|---|---|
| Goal | Predict a category (0 or 1) | Predict a continuous number |
| Output activation | sigmoid |
linear |
| Loss function | binary_crossentropy |
mean_squared_error |
| Key metric | Accuracy | MAE |
Questions to explore¶
- What happens to training if you increase or decrease the number of neurons in the hidden layers?
- What happens if you remove a hidden layer entirely — can the network still learn the rule?
- Why would using
sigmoidon the regression model’s output layer be a mistake? - Try lowering the number of training epochs for each model — at what point does performance start to degrade?
- Both datasets here are tiny (10 training examples). How might results differ with a much larger, noisier real-world dataset?
Great content! Keep up the good work!