Bellingcat tests LLMs at geolocation

TLDR

• Bellingcat tested models on 52 unpublished, metadata-scrubbed photos; several placed at least 10 within 1 km.¹
• Top models placed over half the photos within 25 km, but none was close to perfect at the exact spot.¹
• Bellingcat says Anthropic's GeoGuessr-based comparison has workarounds, and a researcher it quotes flags possible training-data contamination.¹

Bellingcat has published its latest round of AI geolocation benchmarks, testing closed and open-weight models on 52 photos that had never been online, and says the best models narrow a location quickly but none reliably finds the exact spot.¹

Why it matters: this is a methodology update for investigators deciding whether to add LLMs to a geolocation workflow. Top models could narrow a context-free photo to a general area, in some cases faster than a human could, so they suit a first pass. None was close to exact, so every answer still needs checking against imagery.¹

Driving the news:

• Bellingcat ran the test after Anthropic's September post, which, in Bellingcat's words, claimed its models were approaching "superhuman capabilities for geolocating outdoor photos".¹
• Models had to give one set of coordinates, with no hedging, and could use their providers' built-in search, code execution and page-fetch tools.¹
• Gemini 4 Argon scored highest on Bellingcat's Mean Log Score, which rewards closing the last few kilometres. Claude's Opus and Fable led on either metric, and Claude Sonnet answered fastest.¹
• GeoSeer Fast refused four photos, Mistral Large 4 refused one, and Gemini 4 Argon repeatedly timed out on one.¹

By the numbers:

• 52 photos, none previously published, with metadata scrubbed.¹
• At least 10 within 1 km for several models; over half within 25 km for the top models.¹

The big picture: Bellingcat has benchmarked models since June 2025.¹ Anthropic's own evaluation used Flickr photos, no tools, and GeoGuessr duels as a proxy human baseline, because it had none for that dataset.² Bellingcat quotes Anthropic's Mythos medians of 37.0 km and 47.2 km across 6,000 photos, and Israel Defense reports a 151 km figure for top GeoGuessr players.¹³ Mythos is not fully released, so it was not tested.¹

The other side: Nuno Moniz of the University of Notre Dame, quoted by Bellingcat, criticises median-based comparisons and says training-data contamination is hard to rule out.¹ The two tests are not like for like: Anthropic's used no tools on online photos, while Bellingcat's allowed tools on unpublished ones.¹² Bellingcat calls Anthropic's example images easy and says Anthropic acknowledges the test does not emulate precise pinning in a sparsely populated battlefield.¹

Single-source note: the benchmark results come from Bellingcat's own tests and have not been independently replicated. Anthropic's figures are as quoted by Bellingcat and Israel Defense.¹³