Drone Navigation Detection and Path Tracing

[Panchadip-128]

Pull Request for DL-Simplified 💡

Issue Title : Drone Navigation Detection and Path Tracing with Reinforcement Learning

• Info about the related issue (Aim of the project) : This project visualizes a drone's pathfinding journey in a grid environment, using both classical A* search and Reinforcement Learning (RL) techniques to achieve optimal navigation. The drone aims to reach a target location while avoiding obstacles and optimizing path cost. This file provides a comprehensive overview of the project’s structure, setup instructions, and available visualizations.
• Name: Panchadip Bhattacharjee
• GitHub ID: Panchadip-128
• Email ID: [email protected]
• Idenitfy yourself: (Mention in which program you are contributing in. Eg. For a JWOC 2022 participant it's, JWOC Participant) GSSoC Ext 2024 Contributor

Closes: #929

Describe the add-ons or changes you've made 📃

Features
Pathfinding with A Algorithm*: Finds an optimal, shortest path from the starting position to the target using the A* heuristic. Reinforcement Learning Navigation: A reinforcement learning model trains to achieve the navigation goal while avoiding obstacles, rewarding efficient paths. Dynamic Obstacles: Specify obstacle positions to simulate real-world barriers and allow pathfinding adaptations. Comprehensive Visualizations: Includes static, dynamic, and 3D visualizations of the environment, path costs, and drone’s decision-making process. Real-time Animation: Watch the drone’s actions in a step-by-step movement toward the target.

Project Structure
pathfinding block: Contains the A* algorithm and helper functions for calculating paths. reinforcement_learning block: Implements the reinforcement learning environment using OpenAI Gym, where the drone learns an optimal policy for navigation. visualizations block: Defines visualization functions, including static, dynamic, and heatmap visualizations.

Setup Instructions
Clone the repository:

git clone https://github.com/Panchadip-128/Drone-Navigation_Detection_using_RL.git cd Drone-Navigation_Detection_using_RL

Install required dependencies:
pip install -r requirements.txt
Run the script: Drone-Navigation_Detection_using_RL.ipynb

Usage:
Specify Start, Target, and Obstacle Positions: Set coordinates for the drone’s starting position, the target, and obstacles. Choose Navigation Algorithm: Run either the A* pathfinding method or the reinforcement learning model to observe different navigation approaches.

Select Visualization Type: View different visualizations of the environment, path, costs, and drone movements.

Visualizations
The project includes several visualizations to illustrate pathfinding and navigation strategies in the environment.

Basic Environment Setup Sets up a grid environment, marking the drone’s starting position, the target, and obstacles.

def visualize_environment(drone_pos, target_pos, obstacles, grid_size=(10, 10))
env graph

Static Path Visualization Displays a static view of the calculated A* path from start to target.

def visualize_path(drone_pos, target_pos, obstacles, path)
a star graph

Heatmap of Pathfinding Costs Shows a heatmap for traversal costs to each grid cell, providing insight into pathfinding challenges.

def visualize_cost_heatmap(start, goal, obstacles, grid_width, grid_height)
pathfinding_heat-map

Dynamic Movement Visualization Animates the drone’s movement toward the target, step-by-step, showing real-time path adjustments.

Navigation Graph

3D Surface Plot of Pathfinding Costs Visualizes the cost distribution across the grid in 3D, highlighting areas with high or low pathfinding costs.

3D Path Finding Cost Suraface schematic

Navigation Graph:

Drone Navigation Graph

Reinforcement Learning (RL) Model Overview In addition to the A* algorithm, this project includes a reinforcement learning approach to allow the drone to learn optimal navigation strategies through interaction with the environment. The RL agent is implemented using OpenAI Gym and trained with the Proximal Policy Optimization (PPO) algorithm from stable-baselines3.

RL Environment The RL environment for the drone is defined in DroneEnv, an OpenAI Gym environment that:

Defines the drone’s possible actions: Up, Down, Left, Right, and diagonal moves. Contains a custom reward function: Positive Reward: Awarded for reaching the target. Penalty: Applied when the drone hits an obstacle or moves away from the target. Exploration vs. Exploitation: Introduces a small exploration rate (epsilon) to encourage the drone to explore initially before converging on optimal paths.

Training the RL Model
from stable_baselines3 import PPO

env = DroneEnv()
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10000) # Training the model with adjustable timesteps
Evaluation
After training, the RL model navigates the drone autonomously, continuously adjusting its path based on learned policies. This approach enhances the drone’s flexibility, enabling it to adapt even with changing obstacles or targets.

Type of change ☑️

What sort of change have you made:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, local variables)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested? ⚙️

To ensure the functionality and robustness of the drone navigation code, various testing methodologies have been employed. The following sections detail the testing approaches, scenarios, and outcomes.

1. Unit Testing
   Component Testing: Each function within the DroneEnv class was tested individually to verify its correctness. This includes:
   reset(): Verified that the drone's state resets to the initial position.
   step(action): Tested all possible actions to ensure the drone's position updates correctly and that it handles boundaries and obstacles properly.
   render(): Confirmed that the output displays the correct positions of the drone, target, and obstacles.
2. Integration Testing
   Environment Interaction: The integration of various components was tested by running the complete environment. This involved:
   Executing sequences of actions to observe the overall behavior of the drone in navigating toward the target while avoiding obstacles.
   Ensuring that rewards and termination conditions are triggered appropriately when the drone collides with obstacles or reaches the target.
3. Simulation Testing
   Multiple Episodes: The environment was run over multiple episodes (e.g., 100) to assess the drone's learning curve:
   Observations were made regarding the number of steps taken to reach the target and the cumulative rewards received across episodes.
   The render() function provided real-time visualization of the drone's navigation, allowing for immediate feedback on its performance.
4. Performance Testing
   Learning Efficiency: The efficiency of the reinforcement learning model was evaluated by monitoring key performance metrics:
   Tracking the average number of steps taken by the drone to reach the target over multiple training episodes.
   Analyzing the cumulative rewards to assess improvements in navigation over time.
   Conducting parameter tuning to find optimal configurations that enhance the drone's performance.
5. Edge Case Testing
   Boundary Conditions: Tests were conducted to ensure the drone does not move outside the grid boundaries:
   The drone's reactions were observed when attempting to execute actions that would lead it out of bounds.
   Special scenarios involving proximity to obstacles were tested to verify that collision detection works correctly.
   Describe how it has been tested
   Describe how have you verified the changes made

Checklist: ☑️

• My code follows the guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly wherever it was hard to understand.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added things that prove my fix is effective or that my feature works.
• Any dependent changes have been merged and published in downstream modules.

[github-actions]

Our team will soon review your PR. Thanks @Panchadip-128 :)

[Panchadip-128]

@abhisheks008 Please review once

[abhisheks008]

Remove the .gitignore file. Apart from this it looks good to me.
@Panchadip-128

[abhisheks008]

The model files is empty. What's the purpose of it?

[Panchadip-128]

@abhisheks008 For me its showing files inside model directory

[Panchadip-128]

Oh sorry , I have removed it
Please check now @abhisheks008

[Panchadip-128]

@abhisheks008

[abhisheks008]

Looks good to me. Approved!
@Panchadip-128