Uav Vlm Exploration

I built UAV VLM Exploration, a Python project combining VLM-guided UAV navigation and autonomous exploration with PX4 SITL, ROS2 Humble, and Gazebo Harmonic. It implements See Point Fly (SPF), Frontier-Based Exploration (FBE), and vision-language-model-guided autonomous navigation.

My UAV VLM Exploration repository is on @abdulwasaeee's GitHub.

I started the repository in 2026-05, and it was last updated in 2026-06.

The project uses PX4 SITL, ROS2 Humble, and Gazebo Harmonic.

I identify See Point Fly (SPF) as CoRL 2025.

pkill -f "px4|gz|MicroXRCE|ros2|singularity|xterm" 2>/dev/null
sleep 8
cd ~/irobot/px4_sim
bash run_server.sh e1547056

After starting the simulation, I wait approximately 90 seconds for all xterms to initialize.

The documented startup sequence is to clean up existing processes, start the simulation, launch the core VLM navigation stack, and then start the autonomous exploration modules. Each documented terminal runs inside singularity exec --nv ~/irobot/uav_stack.sif and sources the ROS2 Humble, PX4 ROS2 workspace, and planner workspace setup files.

source /opt/ros/humble/setup.bash
source ~/irobot/px4_ros2_ws/install/setup.bash
source ~/irobot/planner_ws/install/setup.bash

# Terminal A — VLM interface
export OPENAI_API_KEY=$OPENAI_API_KEY
export OLLAMA_API_KEY=$OLLAMA_API_KEY
ros2 launch uav_vlm vlm.launch.py

# Terminal B — SPF orchestrator
ros2 launch uav_global_planner spf_orchestrator.launch.py

# Terminal C — RViz
export DISPLAY=$DISPLAY
rviz2

# Terminal D — Frontier Explorer
ros2 launch uav_exploration explorer.launch.py

# Terminal E — VLM Frontier Selector
export OPENAI_API_KEY=$OPENAI_API_KEY
python3 ~/irobot/planner_ws/uav_exploration/src/vlm_frontier_selector.py

# Terminal F — Metrics Logger
python3 ~/irobot/planner_ws/uav_exploration/src/exploration_metrics.py --ros-args -p run_name:=vlm_run

After Terminal A launches, my interface greets the user: “Hi! I am your inspection drone. How can I help you today?”

• Example instructions I can give the drone include fly to the yellow cylinder, find the red box, what do you see, go forward, and move left 2.

The navigation pipeline starts with user_instruction_node publishing instructions on /user/instruction. vlm_spatial_grounding captures RGB and depth images, queries GPT-4o, obtains a 2D pixel and depth, projects the target into a 3D world pose, and publishes /spf/target_pose. spf_orchestrator calls the navigation action server and publishes /uav/global_planner_status.

In my planner, SPF direct mode skips A* and publishes a direct single-pose path on /uav/global_path.

The waypoint manager uses rotation-first behavior before translation. The motion-planning node publishes /uav/cmd_vel; during rotation it holds velocity at zero, while during translation it uses obstacle avoidance and motion primitives. The setpoint publisher sends /fmu/in/trajectory_setpoint to the MicroXRCE-DDS Agent, then PX4 SITL, and finally Gazebo Harmonic.

• I added SPFPhase states IDLE, ROTATING, TRANSLATING, and COMPLETE to waypoint_manager_node.cpp; it publishes /uav/mission_phase, has a yaw-alignment threshold parameter, and aligns yaw before translation.
• My mp_node.cpp subscribes to /uav/mission_phase; in ROTATING it publishes zero velocity, and in TRANSLATING it performs normal obstacle avoidance.
• My setpoint_publisher_node.cpp adds FlightState::ROTATING and forces velocity to zero during yaw alignment.

I increased cmd_timeout_s from 0.5 to 1.0.

My motion-planning node triggers a stop when obstacle distance is < 0.47 m.

I reported a 14.8% improvement in exploration efficiency for FBE + VLM, with a slight increase in runtime attributed to VLM inference overhead.

• My documented ESTOP workaround is to restart the simulation and move the UAV away from the obstacle.
• When the drone is stuck after mission completion, my documented workaround is to send a new instruction.
• For VLM hallucinations, my documented workaround is to retry from another viewpoint.
• For cmd_vel timeouts, my documented workaround is to continuously publish waypoints.

My README cites Yamauchi, B. (1997), “A Frontier-Based Approach for Autonomous Exploration.”

• I (Ahmed, e1547056) handled VLM integration, SPF implementation, and exploration metrics.
• Shantam handled system architecture and system design.

The README leaves several details open: it does not give the contents of its SPF Paper reference section, benchmark environment or repetitions, or the calculation behind exploration efficiency. It does not explain the VLM frontier-selection algorithm or specify the yaw-alignment threshold. It also says a cmd_vel timeout switches the setpoint publisher back to HOVER after 1 second, without explaining how that behavior relates to the increase of cmd_timeout_s from 0.5 to 1.0.

singularity exec --nv ~/irobot/uav_stack.sif bash -c "

After mission completion, my waypoint manager continues publishing the last waypoint.

Gazebo lighting affects object recognition in my UAV project.

After a cmd_vel timeout, my setpoint publisher switches back to HOVER after 1 second.