Production Release

.pre-commit-config.yaml

@@ -95,7 +95,7 @@ repos:
   # Lint / autoformat: Python code
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: "v0.8.4"
+    rev: "v0.9.1"
     hooks:
       # Run the linter
       - id: ruff

README.md

@@ -65,6 +65,11 @@ Our vision is to create a platform that is not only accessible and user-friendly
 4. **Seamless Pipeline**: Ensure smooth processing and dissemination of imagery data.
 5. **User-Friendly Application**: Designed for ease of use, allowing even non-professional pilots to contribute.
 
+### Supported Drones
+
+To see a list of supported and unsupported drones, please visit
+the [FAQ](https://hotosm.github.io/drone-tm/about/faq/#q-what-drones-are-supported) page.
+
 ## Getting Started
 
 To get started with Drone TM:
@@ -84,17 +89,18 @@ To get started with Drone TM:
 |✅| 🖥️ upload of drone imagery collected during flight |
 |✅| 📱 follow terrain during flight plan generation in hilly/mountainous regions |
 |✅| 🖥️ merging of drone imagery into a final combined image for the project |
+|✅| 📱 flight plans working on DJI Mini 4 Pro, Air 3, and Mavic 3 |
+|✅| 📱 upload flight plan to drone via mobile app (no laptop required) |
+|✅| 🖥️ precise georeferencing of final imagery using Ground Control Points |
 |⚙️| 🖥️ automated Digital Elevation Model inclusion (no manual upload required) |
 |⚙️| 🖥️ automated uploading of final imagery to OpenAerialMap (+ credit to user that uploads) |
 |⚙️| 📱 allow adjustments to the flight plan orientation based on field conditions |
-|⚙️| 🖥️ precise georeferencing of final imagery using Ground Control Points |
+|⚙️| 🖥️ separate workflows for processing individual images vs batch processing in ODM |
 | | 📱 capture of imagery at multiple (configurable) angles from the drone camera |
-| | 📱 support for more drone models (DJI first, other manufacturers next) |
-| | 📱 removing laptop requirement for flight plan upload to drone (via mobile instead) |
+| | 📱 continue to add support for additional drone models |
 | | 🖥️ user access management for each part of the UI |
 | | 📱 & 🖥️ real-time notifications for drone flight progress & task status |
 | | 📱 improved offline capabilities of Drone-TM, reducing reliance on stable internet in the field |
-|⚙️| 🖥️ separate workflows for processing individual images vs batch processing in ODM |
 | | 🖥️ scaling of ODM imagery processing to hundreds of images in parallel |
 | | 🖥️ better usage of 3D model data collected by drones |
 | | 📱 HOT community mapping drone: cheap, mapping optimized, materials sourced locally |

docs/about/faq.md

@@ -1,7 +1,5 @@
 # ❓ Frequently Asked Questions ❓
 
-# Frequently Asked Questions (FAQ)
-
 ## Q. What problem do we solve with DroneTM?
 
 **Enhancing Emergency Response**
@@ -39,9 +37,8 @@ You can contribute to DroneTM in multiple ways:
 
 Currently, only processed data is available for download. The final outputs include:
 
-- **2D Orthophoto**
-- **Digital Terrain Model (DTM)**
-- **Digital Surface Model (DSM)**
+- **2D Orthophoto**: a birdseye view of your area (like a high resolution satellite image).
+- **3D Point Cloud**: this can be used to generate Digital Terrain / Surface Models.
 
 ---
 
@@ -65,7 +62,7 @@ no-fly zones. There is a feature to draw these zones on the project map, and the
 
 ---
 
-## Q. Can I use any drones to contribute to DroneTM or are there any specifications of drones?
+## Q. What Drones Are Supported?
 
 Currently, DroneTM is tested on **DJI Mini 4 Pro**, and flight plans are optimized for its camera specifications.
 However, the system is compatible with any DJI drones that support waypoint features, such as:
@@ -76,3 +73,25 @@ However, the system is compatible with any DJI drones that support waypoint feat
 
 Users can also download the **GeoJSON** of the task area, load it into their
 drones, and create custom flight plans as per the specifications provided by the project creator.
+
+!!! note
+
+      We have two angles to increase the number of supported drones:
+
+      1. Support for interfacing with flight plan software such as DroneDeploy,
+      Litchi, DroneLink, meaning any drone supported there is supported by
+      DroneTM.
+
+      2. Support for Open-Source flight plan software such as ArduPilot and iNAV,
+      widening our support signficantly to cheap custom-made drones, and in future
+      the HOT mapping drone.
+
+## Q. What Drones Are Not Supported?
+
+We don't have an exhaustive list of all unsupported drones, but in general, if
+the drone is not listed as supported, then DroneTM will probably not work with
+it.
+
+Our goal is to support affordable community mapping drones that ideally cost
+less than 1000 USD, so support for expensive commercial drones will not be
+a priority for now.

docs/index.md

@@ -1,3 +1,38 @@
 # Drone Tasking Manager
 
-Welcome to the docs!
+**Drone TM** is an integrated digital public good solution designed to harness
+the power of the crowd to generate high-resolution aerial maps of any location.
+
+This innovative platform provides drone pilots, particularly in developing
+countries, with job opportunities while contributing to the creation of
+high-resolution datasets crucial for disaster response and community resilience.
+
+## Problem Statement
+
+In low-income and disaster-prone areas, the accessibility to near real-time satellite datasets is severely restricted. High-resolution satellite imagery, when available, is often prohibitively expensive and outdated. Full-scale aircraft mapping is not a viable option due to its high costs and operational complexity. Traditional mapping solutions, relying on professional consultants with expensive equipment, often result in delays and lack of locally relevant data. Existing drone operation tools are mostly proprietary and not designed for large-scale collaborative efforts, limiting their effectiveness for community-driven projects.
+
+## Solution
+
+**Drone TM** offers a solution to these challenges by empowering communities to utilize drones for immediate and locally relevant mapping needs. Our platform:
+
+- Provides a user-friendly, inclusive application enabling anyone with a drone, including inexpensive consumer or DIY drones, to contribute to a global repository of free and open aerial imagery.
+- Facilitates community-driven drone operations, ensuring immediate response and responsible mapping that considers local needs.
+- Coordinates aerial survey activities among multiple pilots through an open-source tasking platform, incorporating tools and processes to ensure coordinated flight plans for effective imagery acquisition.
+- Offers a seamless pipeline for processing and dissemination of the collected imagery.
+
+## Vision
+
+Our vision is to create a platform that is not only accessible and user-friendly but also inclusive, enabling widespread participation in creating high-resolution aerial maps. By leveraging the power of community-operated drones, we aim to build a resilient and responsive solution that addresses the needs of low-income and disaster-prone areas.
+
+## Features
+
+1. **Crowdsourced Mapping**: Empower drone pilots to contribute to a global imagery repository.
+2. **Community-Driven Operations**: Enable communities to use drones for immediate and locally relevant mapping.
+3. **Open-Source Platform**: Coordinate aerial surveys with an open-source tasking platform.
+4. **Seamless Pipeline**: Ensure smooth processing and dissemination of imagery data.
+5. **User-Friendly Application**: Designed for ease of use, allowing even non-professional pilots to contribute.
+
+### Supported Drones
+
+To see a list of supported and unsupported drones, please visit
+the [FAQ](https://hotosm.github.io/drone-tm/about/faq/#q-what-drones-are-supported) page.

src/backend/app/db/db_models.py

@@ -26,6 +26,7 @@
     UserRole,
     State,
     RegulatorApprovalStatus,
+    ImageProcessingStatus,
 )
 from sqlalchemy.orm import (
     object_session,
@@ -155,6 +156,11 @@ class DbProject(Base):
     regulator_approval_status = cast(
         RegulatorApprovalStatus, Column(Enum(RegulatorApprovalStatus), nullable=True)
     )
+    image_processing_status = cast(
+        ImageProcessingStatus,
+        Column(Enum(ImageProcessingStatus), default=ImageProcessingStatus.NOT_STARTED),
+    )  # status of image processing
+
     regulator_comment = cast(str, Column(String, nullable=True))
     commenting_regulator_id = cast(
         str,

src/backend/app/gcp/gcp_routes.py

@@ -30,7 +30,7 @@ async def find_images(
     fov_degree = 82.1  # For DJI Mini 4 Pro
     result = await project_schemas.DbProject.one(db, project_id)
     return await gcp_crud.find_images_in_a_task_for_point(
-        project_id, task_id, point, fov_degree, result.altitude
+        project_id, task_id, point, fov_degree, result.altitude_from_ground
     )
 
 
@@ -48,5 +48,5 @@ async def find_images_for_a_project(
     task_id_list = await list_task_id_for_project(db, project_id)
 
     return await gcp_crud.find_images_in_a_project_for_point(
-        project_id, task_id_list, point, fov_degree, result.altitude
+        project_id, task_id_list, point, fov_degree, result.altitude_from_ground
     )

src/backend/app/migrations/versions/f78cde896334_image_processing_status.py

@@ -0,0 +1,69 @@
+"""image_processing_status
+
+Revision ID: f78cde896334
+Revises: b18103ac4ab7
+Create Date: 2025-01-15 05:11:08.788485
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "f78cde896334"
+down_revision: Union[str, None] = "b18103ac4ab7"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Create the enum type in the database
+    image_processing_status_enum = sa.Enum(
+        "NOT_STARTED",
+        "PROCESSING",
+        "SUCCESS",
+        "FAILED",
+        name="imageprocessingstatus",
+    )
+    image_processing_status_enum.create(
+        op.get_bind()
+    )  # Bind the enum type to the database
+
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column(
+        "projects",
+        sa.Column(
+            "image_processing_status",
+            sa.Enum(
+                "NOT_STARTED",
+                "PROCESSING",
+                "SUCCESS",
+                "FAILED",
+                name="imageprocessingstatus",
+            ),
+            nullable=False,
+            server_default=sa.text("'NOT_STARTED'"),
+        ),
+    )
+
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("projects", "image_processing_status")
+
+    # Drop the enum type from the database
+    image_processing_status_enum = sa.Enum(
+        "NOT_STARTED",
+        "PROCESSING",
+        "SUCCESS",
+        "FAILED",
+        name="imageprocessingstatus",
+    )
+    image_processing_status_enum.drop(op.get_bind())  # Drop the enum type
+
+    # ### end Alembic commands ###

src/backend/app/models/enums.py

@@ -74,6 +74,15 @@ class RegulatorApprovalStatus(IntEnum, Enum):
     REJECTED = 2
 
 
+class ImageProcessingStatus(IntEnum, Enum):
+    """Enum to describe all possible statys of a Image Processing for a Project"""
+
+    NOT_STARTED = 0
+    PROCESSING = 1
+    SUCCESS = 2
+    FAILED = 3
+
+
 class ProjectVisibility(IntEnum, Enum):
     """Enum describing task splitting type."""
 
@@ -200,11 +209,11 @@ class EventType(str, Enum):
 
 
 class FlightMode(str, Enum):
-    """The flight mode of the drone.
-
-    The flight mode can be:
-    - ``waylines``
-    - ``waypoints``
+    """
+    The flight mode of the drone.
+        The flight mode can be:
+        - ``waylines``
+        - ``waypoints``
     """
 
     waylines = "waylines"

src/backend/app/projects/image_processing.py

@@ -5,7 +5,7 @@
 import shutil
 from pathlib import Path
 from app.tasks import task_logic
-from app.models.enums import State
+from app.models.enums import State, ImageProcessingStatus
 from app.utils import timestamp
 from app.db import database
 from app.projects import project_logic
@@ -183,7 +183,7 @@ async def _process_images(
 
             # Start a new processing task
             task = self.process_new_task(
-                images_list,
+                list(set(images_list)),
                 name=name
                 or (
                     f"DTM-Task-{self.task_id}"
@@ -352,6 +352,7 @@ async def process_assets_from_odm(
     message=None,
     dtm_task_id=None,
     dtm_user_id=None,
+    odm_status_code: Optional[int] = None,
 ):
     """
     Downloads results from ODM, reprojects the orthophoto, and uploads assets to S3.
@@ -443,6 +444,21 @@ async def process_assets_from_odm(
                         conn, dtm_project_id, dtm_task_id, "assets_url", s3_path_url
                     )
 
+        if not dtm_task_id:
+            # Update the image processing status
+            pool = await database.get_db_connection_pool()
+            async with pool as pool_instance:
+                async with pool_instance.connection() as conn:
+                    await project_logic.update_processing_status(
+                        conn,
+                        dtm_project_id,
+                        (
+                            ImageProcessingStatus.SUCCESS
+                            if odm_status_code == 40
+                            else ImageProcessingStatus.FAILED
+                        ),
+                    )
+
     except Exception as e:
         log.error(f"Error during processing for project {dtm_project_id}: {e}")
 

src/backend/app/projects/project_logic.py

@@ -38,13 +38,14 @@
     create_placemarks,
     terrain_following_waylines,
 )
-from app.models.enums import FlightMode
+from app.models.enums import FlightMode, ImageProcessingStatus
 
 
 async def get_centroids(db: Connection):
     try:
         async with db.cursor(row_factory=dict_row) as cur:
-            await cur.execute("""
+            await cur.execute(
+                """
                 SELECT
                     p.id,
                     p.slug,
@@ -61,7 +62,8 @@ async def get_centroids(db: Connection):
                     task_events te ON t.id = te.task_id
                 GROUP BY
                     p.id, p.slug, p.name, p.centroid;
-            """)
+            """
+            )
             centroids = await cur.fetchall()
 
             if not centroids:
@@ -335,6 +337,25 @@ async def process_drone_images(
     )
 
 
+async def update_processing_status(
+    db: Connection, project_id: uuid.UUID, status: ImageProcessingStatus
+):
+    print("status = ", status.name)
+    """
+    Update the processing status to the specified status in the database.
+    """
+    await db.execute(
+        """
+        UPDATE projects
+        SET image_processing_status = %(status)s
+        WHERE id = %(project_id)s;
+        """,
+        {"status": status.name, "project_id": project_id},
+    )
+    await db.commit()
+    return
+
+
 async def process_all_drone_images(
     project_id: uuid.UUID, tasks: list, user_id: str, db: Connection
 ):
@@ -363,6 +384,10 @@ async def process_all_drone_images(
         webhook=webhook_url,
     )
 
+    # Update the processing status to 'IMAGE_PROCESSING_STARTED' in the database.
+    await update_processing_status(db, project_id, ImageProcessingStatus.PROCESSING)
+    return
+
 
 def get_project_info_from_s3(project_id: uuid.UUID, task_id: uuid.UUID):
     """