Ember

We were inspired by this summer's wildfires. Most wildfire tech starts working once there's already smoke. We wanted to work on what happens before that, which is the dry fuel that builds up around forests and homes. A lot of that is just fallen leaves.

Clearing them isn't hard. It's just repetitive, and the leaves keep falling. That seemed like a good job to hand off to a robot.

The leaf detection model, the autopilot and the 3D simulator we tested it in, and the live rover and leaf tracking on our dashboard.

The leaf detection runs on the iPhone mounted on the rover. A computer vision model finds leaves in the camera feed, and the phone's LiDAR tells us how far away they are and which things are obstacles instead of leaves. The dashboard shows where the rover is and every leaf it has found as it happens, so you can watch what the rover is thinking from a laptop and drive it from the browser.

My initial plan was to train our own leaf detection model. But there was no good public datasets for dead leaves on the ground. The closest one I found was made for spotting disease on crop leaves. The model I trained on it scored 0.54 mAP50 on green leaves but only 0.04 on dead ones, because none of its photos were of leaves on the ground. I also trained a model to spot obstacles like water, rocks, and trees, and it barely picked up water or rocks. Training a model also takes hours, and for hackathon, our biggest problem was time.

So I switched to Google's OWL-ViT. Instead of training it, you describe what you want in plain text, like "a dry brown leaf" or "a pile of autumn leaves", and it finds it in the image. No dataset and no training needed. It isn't perfect on its own, since it also picks up brown things that aren't leaves. So I paired it with LiDAR. Anything more than 25 cm off the ground or wider than 1.5 m gets thrown out, which removed most of the bark, fences, and furniture. LiDAR also covered obstacles, so we dropped the obstacle model completely.

Drawing a box around a leaf on screen was the easy part. The detections flicker between frames, anything brown looks like a leaf, and once the rover gets close, the leaf drops out of view under the camera.

So I saved every leaf as a spot on the ground instead of a box on the screen, using the phone's motion tracking. If two sightings are within 25 cm of each other, they count as the same leaf. A leaf only becomes a target after it's been seen three times. Until then, it gets forgotten if it isn't seen again within four seconds. Since the rover is driving to a spot on the ground, it can keep going even after the leaf is out of view.

The rover can't see under itself, so when the scoop missed, it had no idea. I made it notice a miss later instead. If it drives away and sees the same leaf still sitting there, that leaf goes back on the list. After three tries it gives up so it doesn't get stuck on one leaf.

Our hardware wasn't ready until the morning of submissions, so I couldn't wait for the rover to test the autopilot. I built a 3D simulator in the browser instead. It puts the rover on a fake lawn with leaves, obstacles, and noisy detections, and runs the same autopilot code the phone runs. That let me watch the whole job play out: find the leaves, drive to each one, scoop, bring them back to the pile, and retry the ones it missed.

The simulator can't tell you how the real camera behaves, so I also took the phone outside. I held it where it would sit on the rover and walked around, acting as the rover myself. That let me test the leaf detection on real leaves and check that the autopilot was steering me the right way, all before the rover could drive.