Autonomous drone

The problem

The mission is a chain of decisions, not a route. The drone takes off from a known home position and flies a series of global waypoints. In front of each gate it stops, photographs a signboard, and sends that image to a language model, which answers left or right. That answer decides which waypoint it flies next. After the third gate it moves over four candidate storefronts, looks down, works out which one is the target, and descends onto it.

There is no ground station making decisions and no external tracking system telling the aircraft where it is. All of that has to come off two cameras and an autopilot.

What I built

The team had no formal roles. My share was the parts everything else stands on: the training data for the landing detector, the camera calibration, the altitude channel, and a great deal of troubleshooting.

• A Unity simulation that generated roughly 10,000 labelled images of the landing targets, and the training and fine-tuning of the detector on top of it.
• Kalibr calibration: the transform matrices from both cameras to base_link, and the time offset between the image stream and the IMU.
• The downward-facing depth camera and the altitude channel built on it.
• Flashing the flight controller and setting up the radio.
• The parts that only show themselves in flight — the lift-off, and the depth camera's vision window as the airframe pitches.

Key technical decisions

The training set is synthetic, because collecting it was never going to happen. A detector that tells four storefronts apart from above needs thousands of labelled examples, and photographing them by hand was not possible in the time available. So I built the scene in Unity and generated the data instead: about 10,000 labelled frames, each containing anywhere from zero to sixteen boxes, then fine-tuned on roughly 500 real images so the model met real motion blur and sensor noise before the day.

Domain randomisation is what makes synthetic data generalise — randomising background, texture, scale, placement and lighting so the network learns the shape rather than the scene. The lighting half of it refused to work until I moved the Unity project onto the High-Definition Render Pipeline, which is what finally made light behave realistically enough to be worth randomising.

The drone kept seeing the floor as a wall. Flying forward means pitching forward, and a forward-pitched depth camera looks partly at the ground ahead of it. The obstacle mapper had no way to tell that from a real obstacle, so the local occupancy grid filled up with floor and the planner refused to go anywhere. The fix was to stop trusting the whole frame: the camera's usable vision window is narrowed dynamically from the drone's own IMU pitch, so only the part of the view that can legitimately hold an obstacle is ever mapped.

Altitude gets a sensor of its own. Height inferred from a forward camera, or from the geometry of small markers, is the weakest axis in the whole stack. So the downward depth camera does nothing but measure ground-relative range, fused in as a dedicated Z channel while the visual odometry carries X and Y.

It has a floor: the sensor cannot measure closer than about 30 cm. Below that the estimator falls back to the visual odometry's own Z and rides through, so crossing the threshold does not put a step in the height estimate — and a step there goes straight into the controller.

Odometry took four attempts. RGB-D SLAM fused with the IMU drifted badly in Z. A tracking camera with built-in odometry looked better until small jitters turned into large position spikes. RGB-D with the depth-camera Z still drifted. What finally held was semi-direct visual odometry, measured at roughly 0.3 m of drift across the whole arena on a hand-recorded run. There was no way to rank those four from a desk. Each one had to be flown.

What it taught me

The most valuable work I did here was not code. It was sitting down with about forty flight logs from a single day, lining them up chronologically, and working out what had actually gone wrong — because the obvious answer was wrong.

Everything we suspected turned out to be healthy. Feature tracking held steady at 260–330 matches per frame from early afternoon into the evening, so it was not a camera, a lighting or a texture problem. Measured height to the floor never exceeded about three metres, so the aircraft was where we thought it was.

The failure was in the estimator. From a point in the evening onward the autopilot's filter began dropping into its constant-position fallback — the mode it enters when it decides it has no trustworthy horizontal reference, stops integrating motion and assumes the vehicle is holding still. Once there it dead-reckons on the IMU and drifts without bound. The logs show the estimate wandering tens of metres inside a room twelve metres across, while the measured height stayed under three.

The part that stuck with me is the red herring. A parameter had been changed shortly before things got worse, and it looked exactly like the culprit. It was not. The fallback was already firing on more than half the flights before the change, and reverting it did nothing — because a parameter edit does not reset the filter's internal state, and neither does re-arming. Once an estimator has spent flights dead-reckoning, its bias and covariance estimates are corrupt, and that corruption carries from flight to flight. Only a full power cycle clears it.

Being close in time is not the same as being the cause, and the thing you touched last is not automatically the thing that broke. The controller was doing its job correctly the entire time. It was being handed a broken estimate. I have not trusted a convenient coincidence the same way since.