Exploring Neural Networks: Sign Language Classification with Python

Hi there! Welcome to my blog where I share my coding projects and insights. Today, I want to walk you through a fascinating project I worked on involving neural networks and sign language data. This project was both challenging and insightful, and I hope you’ll find it as exciting as I did.

---

🚀 Project Overview

In this project, I developed a neural network model to classify sign language gestures using the MLPClassifier from scikit-learn. The goal was to analyze how different training and test splits affect the model’s performance and to dive deep into metrics like class-level accuracy and confusion matrices.

---

🛠 Technologies Used

• Python

• Pandas for data manipulation

• Scikit-learn for building the neural network

• Numpy for numerical operations

• Matplotlib/Seaborn (optional for visualization)

---

🔍 The Dataset

I worked with two CSV files:

1. sign_mnist_13bal_train.csv – Training data

2. sign_mnist_13bal_test.csv – Test data

Each dataset contained pixel values representing hand gestures and a target column labeled class to indicate the gesture type.

---

🧠 The Neural Network

I implemented a simple neural network with:

• Input Layer: Matching the number of features (784 pixels after flattening the images).

• One Hidden Layer: With 8 neurons for simplicity.

• Output Layer: Representing the different sign classes.

Here’s the architecture breakdown:

• Activation Function: ReLU

• Solver: Adam optimizer

• Loss Function: Cross-Entropy Loss

• Random State: 42 for reproducibility

---

🧑‍💻 Code Walkthrough

Data Preprocessing

First, I loaded the data and normalized the pixel values to a range of 0 to 1 for better model performance. I also split the features (X) and target labels (y).

import pandas as pd
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import train_test_split
from collections import defaultdict
import numpy as np

# Load and preprocess training data
train_data = pd.read_csv('sign_mnist_13bal_train.csv')
X_train = train_data.drop('class', axis=1) / 255.0
y_train = train_data['class']

# Load and preprocess test data
test_data = pd.read_csv('sign_mnist_13bal_test.csv')
X_test = test_data.drop('class', axis=1) / 255.0
y_test = test_data['class']

Training the Model

I trained the model across different test sizes (10% to 40%) to evaluate how varying the training size impacts performance.

neural_net_model = MLPClassifier(hidden_layer_sizes=(8), random_state=42, tol=0.005)

for test_size in range(10, 41, 10):
    X_train_split, X_test_split, y_train_split, y_test_split = train_test_split(
        X_train, y_train, test_size=test_size/100, random_state=42)
    neural_net_model.fit(X_train_split, y_train_split)

Evaluating Performance

For each split, I calculated class-level accuracy, overall accuracy, and generated confusion matrices.

    y_pred_train = neural_net_model.predict(X_train_split)
    y_pred_test = neural_net_model.predict(X_test_split)

    # Overall Accuracy
    overall_accuracy = (y_pred_test == y_test_split).mean() * 100
    print(f"Overall Test Accuracy: {overall_accuracy:.1f}%")

    # Confusion Matrix
    cm = confusion_matrix(y_test_split, y_pred_test)
    print("Confusion Matrix:")
    print(cm)

---

📊 Results and Insights

Here are the key observations from the model’s performance:

1. Accuracy Challenges: The accuracy was relatively low across all splits, with overall test accuracy ranging from 10.3% to 17.3%. This highlights room for improvement, possibly by fine-tuning the model or increasing the hidden layer size.

2. Class-level Accuracy: Some classes performed better than others. For instance:

   • Class 5 achieved 50% accuracy with a 10% test size.

   • Class 8 reached 33% accuracy consistently.

3. Confusion Matrix: The confusion matrix revealed that many gestures were misidentified, especially for classes with fewer examples.

4. Training Size Effect: Larger training sets (smaller test sizes) improved training accuracy but didn’t significantly boost test accuracy. This suggests potential overfitting or insufficient training data.

---

🔧 Challenges Encountered

• Balancing the dataset could have improved class-level accuracy.

• Adding more hidden layers and experimenting with other activation functions might have enhanced performance.

---

🌟 Next Steps

To improve the model:

• Experiment with deeper networks or other architectures like CNNs (Convolutional Neural Networks).

• Use data augmentation to artificially expand the dataset.

• Visualize the learning process with loss curves and confusion matrices.

---

🖼 Output Sample

Here’s a snapshot of the results:

Example Confusion Matrix

[[0 0 0 0 0 1 0 0]
 [0 0 0 0 0 1 0 1]
 [0 0 0 0 0 1 0 0]
 [0 0 0 0 0 0 1 0]
 ...
]

Test Accuracy

Overall Test Accuracy: 17.3%

---

🌏 GitHub Repository

Feel free to check out the code for this project on GitHub. Clone it, or contribute to further improvements!

---

✍️ Final Thoughts

This project was an incredible learning experience. Working with neural networks taught me the importance of proper data preprocessing and model evaluation. Although the results weren’t perfect, they laid a solid foundation for future improvements.

Let me know your thoughts and suggestions in the comments. Don’t forget to like and share if you found this insightful!

Happy coding! 🚀