Senior Project · LiDAR-based Autonomous Driving (DDPG)

Problem Statement

Chulalongkorn University operates CU Toyota Ha:mo as part of an on-campus self-mobility service. The challenge was to develop an autonomous driving system that could respond to customer use cases, but previous research using Deep Q-Network (DQN) had two critical limitations:

Limitations of prior work (DQN-based)

1. Discontinuous steering: DQN outputs discrete actions, causing jerky control unsuitable for smooth driving

2. Poor generalisation: Tested only on the training path; expected to fail on unseen routes

3. No ACC functionality: Only handled emergency stops, not adaptive following

My Solution Strategy

Replace DQN with Deep Deterministic Policy Gradient (DDPG), an actor-critic algorithm designed for continuous action spaces. DDPG outputs continuous values (e.g., steering angle ∈ [-1, +1]) instead of discrete bins, enabling smoother control. Extend the system to handle two parallel tasks: (1) Smooth Steering and (2) Adaptive Cruise Control.

System Architecture

The codebase is divided into 5 modular Python files, each with a single responsibility. This separation enabled iterative development: I could modify reward functions or state representations without touching the DDPG agent code itself.

File	Role	Responsibility
`ddpg_agent.py`	Core AI	DDPG algorithm: Actor, Critic, Replay Buffer, OU Noise, Soft Update
`carla_steer_env.py`	Environment	Steering Task · 7-dim state · LiDAR + Collision sensor
`carla_acc_env.py`	Environment	ACC Task · 4-dim state · Ego + lead vehicle spawn
`train.py`	Training	DDPG training loop · checkpoint saving · reward plotting · resume support
`test.py`	Evaluation	10-run evaluation · measures track error / time gap · plots results

DDPG Architecture · Actor-Critic with Experience Replay

Core Components

1. ReplayBuffer

Experience Replay

Stores transitions (s, a, r, s', done) in a deque buffer of max size 100,000. During training, batches of 64 transitions are sampled uniformly at random: this breaks temporal correlation between consecutive samples, which is essential for stable gradient updates in deep RL.

          ddpg_agent.py · ReplayBuffer class
          Python
        
class ReplayBuffer:
    def __init__(self, max_size=100000):
        self.buffer = deque(maxlen=max_size)

    def add(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def sample(self, batch_size):
        # sample uniformly at random — breaks temporal correlation
        batch = random.sample(self.buffer, batch_size)
        return map(np.array, zip(*batch))

2. Ornstein-Uhlenbeck Noise

Exploration Strategy

Standard Gaussian noise is uncorrelated across time steps, which produces jittery exploration behaviour unsuitable for vehicle control. OU noise generates temporally-correlated random values, better suited to physical systems with momentum like cars.

mu (μ)

0.0

Mean value the noise reverts toward

theta (θ)

0.15

Speed of mean reversion

sigma (σ)

0.2

Magnitude of random fluctuation

3. Actor Network · π(s) → a

Policy Network

The Actor takes the current state and outputs a continuous action. The final tanh activation bounds output to [-1, +1], then scaled by action_bound. BatchNorm after the first two hidden layers stabilises training across the varying state distributions.

Input

state_dim

7 (steer) · 4 (ACC)

→

Dense

256

ReLU + BN

→

Dense

256

ReLU + BN

→

Dense

128

ReLU

→

Output

action_dim

tanh · bound

4. Critic Network · Q(s, a) → Q-value

Value Function

The Critic estimates the Q-value for a (state, action) pair. Unlike the Actor, state and action enter via separate processing branches and are concatenated mid-network: this design choice allows the network to learn richer state-action interactions than direct early-fusion would.

Architecture

State branch: Input(state_dim) → Dense(256) → BN → Dense(128) Action branch: Input(action_dim) → Dense(128) ↓ concatenate Dense(256) → Dense(128) → Dense(1) # Q-value

5. DDPG Hyperparameters

Tuned via Iterative Testing

Actor LR

1e-4

Actor learning rate

Critic LR

1e-3

Critic learning rate (10× higher)

Gamma (γ)

0.99

Discount factor for future reward

Tau (τ)

0.005

Soft update rate for target networks

Batch Size

Samples per gradient update

Buffer Size

100K

Max transitions stored

Loss Functions

Critic Loss = MSE( y − Q(s, a) ) where y = r + γ · Q_target(s', π_target(s')) Actor Loss = −mean( Q(s, π(s)) ) # maximises Q-value Soft Update: θ_target ← τ·θ + (1 − τ)·θ_target

Smooth Steering Task

Path Following · Town03 Map · Coords (56.2, -4.2)

0.613 m

Avg Path Error

Objective: Train an agent that steers a vehicle along a reference path while minimising lateral deviation and steering jerk. Throttle is fixed; the agent only controls steering.

State Space Design (7-dim)

I designed a 7-dimensional state vector after iterating across several earlier configurations. Model 4 (the final design) was the best performer: the inclusion of last_action was the breakthrough that produced smooth, jerk-free steering.

#	State Variable	Range	Purpose
1	lidar_front	[0, 1]	Normalised front LiDAR distance
2	lidar_front_left	[0, 1]	LiDAR at angle −15° to −60°
3	lidar_front_right	[0, 1]	LiDAR at angle +15° to +60°
4	angle_diff (φ)	[−1, 1]	Heading diff between car and path
5	dist_from_track	[0, 1]	Lateral distance from centre line (max 3m)
6	position_factor	{−1, +1}	Left (−1) or right (+1) of track centre
7	last_action	[−1, 1]	Previous steering: enables smooth control

Reward Function Design

The reward function is where most of the engineering effort went. The COS(φ) term rewards alignment with the path; the -0.1 × dist_from_track term penalises lateral drift; and the jerk penalty kills sudden steering changes that would feel uncomfortable for passengers.

Steering Reward Function

Reward = COS(φ) − 0.1 × dist_from_track if |action − last_action| < 0.4 = −1 if |action − last_action| ≥ 0.4 (jerk penalty) Collision penalty = −10.0 Out-of-track = −5.0 (lateral dist > 3 m)

Rationale of each term

· COS(φ): Maximum (=1) when car heading aligns with path direction; smoothly decays to −1 when 180° opposite

· −0.1 × dist_from_track: Linear penalty on lateral drift; coefficient 0.1 chosen so it doesn't dominate the alignment signal

· −1 jerk penalty: Triggered when steering changes by ≥0.4 between consecutive steps: teaches the agent to plan smooth manoeuvres

· −10 / −5 hard penalties: Terminate the episode quickly when the agent fails catastrophically (collision or off-track)

Sensor Configuration

Sensor	Configuration	Usage
LiDAR (ray_cast)	`range=20m · 1 channel · 56K pts/sec`	Surroundings detection · split into 3 angular zones
Collision Sensor	`built-in CARLA`	Flags episode termination on impact

Test Results

Key finding · Generalisation

The trained model performed almost as well on an unseen path (0.693m error) as on the training path (0.613m). This was the central success criterion, proving the agent learned general steering principles rather than memorising one specific route. Previous DQN-based research had failed this test.

Adaptive Cruise Control

2-Second Following Distance · Town03 · Coords (-145.5, 100)

1.914 s

Avg Time Gap

Objective: Maintain a safe 2-second time gap from the lead vehicle, the industry-standard safe-following distance. The agent controls throttle/brake (continuous, single axis) while the lead car cruises at a random fixed speed.

State Space Design (4-dim)

For ACC, I designed a 4-dimensional state vector: distance to lead car (current and rate-of-change), time-gap in seconds, and ego velocity. The dist_change dimension was critical for the agent to learn predictive braking.

#	State Variable	Range	Purpose
1	dist_obstacle	[0, 1]	Normalised forward distance (max 50m)
2	dist_change	[−1, 1]	Rate of distance change · closing/opening
3	dist_in_seconds	[0, 1]	Time gap = distance / velocity
4	car_velocity	[0, 1]	Ego speed normalised (max 30 m/s)

Reward Function Design

The ACC reward uses an asymmetric penalty: too far behind incurs full penalty (n=1), but too close incurs only half penalty (n=0.5). This was a deliberate design choice: collision is far more dangerous than excessive following distance, so the loss landscape needed to reflect that asymmetry.

ACC Reward Function

T = dist_in_seconds # time gap to lead car Reward = 1 − n × |2 − T| n = 1.0 if T ≥ 2 s (full penalty when too far behind) n = 0.5 if T < 2 s (HALF penalty when too close) Collision penalty = −10.0 Stopped penalty = −2.0 (velocity < 0.1 m/s)

Design intuition

Maximum reward (=1.0) is achieved exactly at T = 2 seconds. The asymmetric n value (1.0 vs 0.5) makes the agent more tolerant of being slightly too close than slightly too far, but the −10 collision penalty ensures it never gets too close. This shapes a useful trade-off rather than letting the agent default to following far behind.

Vehicle Setup

Vehicle	Spawn Location	Initial Speed
Ego (Toyota Prius)	`(-145.5, 100.0, 0.5)`	Random 10–15 m/s (~36–54 km/h)
Lead (Tesla Model 3)	`(-145.5, 80.0, 0.5)`	Random 6–10 m/s (fixed within episode)

Test Results

Challenge mode validation

In Challenge Mode, the lead vehicle decelerates mid-episode from ~10 m/s down to 3–6 m/s. The trained agent successfully reduced throttle and applied brake in response, proving it had learned reactive control rather than just steady-state following.

Training Methodology

Both tasks used 5,000 episodes with a warm-up phase of random actions (1,000 steps for steering, 500 for ACC) before training begins. This warm-up populates the Replay Buffer with diverse experiences before any gradient updates, critical for stable early training.

Parameter	Steering	ACC	Purpose
Total Episodes	`5,000`	`5,000`	Sufficient for convergence on both tasks
Warmup Steps	`1,000`	`500`	Random actions to seed Replay Buffer
Save Interval	`500 ep`	`500 ep`	Periodic checkpoints for resume support
Moving Average	`50 ep`	`50 ep`	Window for selecting "best model"

        train.py · Command Line Interface
        Shell
      
# Train Steering Task
python train.py --task steer --episodes 5000

# Train ACC Task
python train.py --task acc --episodes 5000

# Resume from checkpoint (e.g. crashed at episode 2,300)
python train.py --task steer --resume models/steer_xxx/best_model

# Specify CARLA host/port (for remote simulator)
python train.py --task steer --host localhost --port 2000

Testing & Evaluation Protocol

Every trained model was evaluated over 10 independent runs in two modes: Normal (training conditions) and Challenge (different path for steering / decelerating lead car for ACC). The evaluation script outputs path-trace plots and action-response plots for visual verification.

The testing discipline that became my BA habit

This Senior Project planted the seed of what I now call "structured testing methodology": define acceptance criteria up front, test across multiple scenarios (including edge cases), measure quantitatively, and iterate until convergence. The exact same discipline appears in my UAT work on the CRM Loyalty project (5 build cycles, 60 defects tracked), different domain, identical method.

Final Results Summary

Task	Mode	Result	Target / Metric
Steering	Normal	`0.613 m`	Minimise avg path error
Steering	Challenge (unseen path)	`0.693 m`	Generalisation test
ACC	Normal	`1.914 s`	Target: 2.000s · Err: 0.086s (4.3%)
ACC	Challenge (decel lead)	`Adaptive`	Reactive braking verified

What this project taught me

The technical content was Deep RL, but the habits the project planted in me are exactly the habits that make a good Business Analyst today:

Four transferable disciplines

1. Modular system design. 5 files, single responsibility per file. The same separation principle I apply when structuring BRDs, FSDs, ERDs, and Test Cases: each artifact has one purpose and one audience.

2. Iterative refinement with quantified feedback. Model 1 → 2 → 3 → 4, each version measured against the same metric (path error). Same as UAT Cycle 1 → 2 → 3 → 4 → 5 in my CRM project.

3. Reward / acceptance criteria design. Asking "what behaviour do I want to incentivise?", whether shaping an RL reward or writing UAT pass/fail criteria, the design discipline is identical.

4. Generalisation testing. Always test on data different from what you trained on. In BA work: always test edge cases, not just the happy path.

Technologies used

Python 3 TensorFlow 2.x NumPy Matplotlib CARLA Simulator 0.9.9 CARLA Python API DDPG Algorithm Actor-Critic Architecture Experience Replay OU Noise Soft Target Update LiDAR Sensor Processing Reward Function Engineering Hyperparameter Tuning

LiDAR-based AutonomousDriving via Deep RL

Problem Statement

My Solution Strategy

System Architecture

Core Components

State Space Design (7-dim)

Reward Function Design

Sensor Configuration

Test Results

State Space Design (4-dim)

Reward Function Design

Vehicle Setup

Test Results

Training Methodology

Testing & Evaluation Protocol

Final Results Summary

What this project taught me

Technologies used

LiDAR-based Autonomous
Driving via Deep RL