The Problem: Tagging Aerial Photos

Both during and in the aftermath of a serious weather incident, drones and manned aircraft gather thousands of aerial photos of the affected areas. These images potentially reveal which buildings and other infrastructure have been affected — but only after each one has been tagged as to precisely what location it’s showing. Unfortunately, the images generally lack this metadata.

Tagging the photos by hand tremendously slows the National Guard’s response. After an incident, its team usually needs about 12 hours to complete the task. Unfortunately, this process has until now remained a manual one. It’s a challenging task to automate since the photos are taken from varying altitudes and at oblique angles.

The Solution: Matching Photos With Machine Learning

The breakthrough? Matching real photos with artificial ones. Bellwether has synthesized a database of simulated reference photos to use as exemplars. When a real photo matches one from the database, it’s tagged — the system then knows precisely where and what it’s a photo of. To synthesize the reference images, X tapped Google’s wealth of unique geospatial resources, the underlying basis for products such as Google Earth and Maps.

The Extensibility Of Predictive AI

This approach is extensible. “Extending beyond our deployment with the National Guard, our goal is to make this kind of service fundamentally easier for a wider group of disaster responders,” says Russell. “It can be applied across rescue and rebuild responses to various weather-related phenomena, including heat waves and tornadoes, for example.”

The Universality Of Predictive AI

Whether shooting the moon or shooting for more typical enterprise goals, ML’s core capacity to generate confidence levels solves operational challenges universally, across industries. Which customers will probably buy? Marketing targets them. Which transactions are probably fraudulent? Banks block them. Which addresses will probably receive a delivery tomorrow? UPS plans for them.

The Quantification Of Uncertainty

So, how confident is confident enough? It depends. Each project must determine the best choice of decision threshold based on practical necessity. For example, the National Guard needs photos that have matched with very high confidence. In contrast, marketing and fraud detection can afford to target many cases that don’t pan out — an unavoidable part of the numbers games that those kinds of operations inevitably play.

Conclusion

Machine learning plays a central role in moonshots like this — just as it does in more common enterprise systems. After all, matching photos is exactly the kind of inexact process that ML handles well. No match is a sure thing, since the aerial photos don’t match up exactly. They each originate from a unique distance, zoom, and angle, they’re potentially occluded by weather conditions, and the landscape they capture has often been affected, sometimes disastrously.

FAQs

Q: How does the Bellwether system work?

A: The Bellwether system uses machine learning to match real photos with artificial ones, allowing it to automatically tag aerial images and provide crucial information to first responders.

Q: How does the system handle uncertainty in the matching process?

A: The system uses confidence levels to assign a level of certainty to each match, allowing it to weed out uncertain matches and provide accurate information to responders.

Q: How does the system plan to extend its capabilities in the future?

A: Bellwether plans to extend its capabilities to other disaster response scenarios, such as predicting where the most lives could be saved and predicting environmental incidents before they happen.