CNN-Based Behavioral Cloning for Autonomous Driving (NVIDIA PilotNet & Vision Transformer)

An end-to-end deep learning pipeline that predicts continuous steering angles directly from front-facing camera images, trained and tested on the Udacity Self-Driving Car Simulator. The project achieves full autonomous laps with smooth, adaptive real-time driving control.

Key contributions and features:

Multi-camera data pipeline (center, left, right) with steering angle correction, trained on ~97,000 samples

Steering distribution balancing via histogram-based undersampling to fix bias toward near-zero steering angles

Full preprocessing pipeline (cropping, YUV conversion, blur, resizing) inspired by NVIDIA's PilotNet paper

Custom data augmentation (flip, pan, affine scaling, brightness adjustment) using Albumentations for GPU-accelerated training

Two model architectures implemented and compared: NVIDIA PilotNet CNN and a Vision Transformer (ViT) with a custom regression head, trained using a two-stage fine-tuning strategy (frozen backbone warm-up, then full fine-tuning)

A real-time inference and deployment system (Adaptive Response Control) with curve-severity classification, dynamic steering boost, adaptive smoothing, and multi-profile throttle control

A live monitoring dashboard using Socket.IO streaming, showing camera feed, 3D steering wheel visualization, and real-time Plotly analytics (steering timeline, speed/acceleration plots, heatmaps)

A separate FastAPI endpoint for on-demand steering prediction from uploaded images

This was a team project (7 contributors); I worked on it as part of a Deep Learning course/project.