How Google And AI Nearly Made A Seasoned Reporter Spiral

(Mis)Uses of Technology

from the truth-and-fiction-and-ai dept

This story was originally published by ProPublica. Republished under a CC BY-NC-ND 3.0 license.

Last month, my colleagues and I published an investigation into a Texas oil refinery startup, America First Refining, that had secretly gotten investment from Donald Trump Jr. We discovered a saga involving the Trump administration’s tariff policy, sanctioned Russian oil and an Indian billionaire family’s private zoo. 

At the center of the story was the CEO of the refinery company, Texas businessman John Calce. We’d spent weeks examining Calce — pulling old lawsuits, property records, corporate registry filings — and had pieced together a portrait of what appeared to be an obscure serial entrepreneur who’d for years tried and failed to secure funding for his long-shot refinery project.

Then, not long before our story was set to publish, we decided to do a scrub on a separate company he had incorporated called Brownsville Energy Storage Terminals.

Pulling up the company’s website, I felt a brief flash of panic: Had we somehow missed the existence of a major business owned by the man at the center of our next story? 

“From Houston to Rotterdam, Jurong to Fujairah. Our network connects the world’s most vital energy markets with speed, safety, and precision bulk oil storage,” announced the front page of the company’s website.

Brownsville Energy Storage Terminals, per the website, had more than 850 employees and 28 million barrels of oil storage capacity across six global hubs. This was puzzling: Our reporting had led us to believe Calce was struggling to raise enough money for a single project in the U.S., not overseeing a massive, multinational oil storage corporation. 

Had we been wrong? 

We turned to Google to learn more about the company’s top executives. Its CEO, Sarah Jenkins, had more than 20 years of experience at major energy firms. And its chief technology officer, David Chen, “built the company’s proprietary inventory management portal and integrated AI-driven predictive maintenance systems,” according to his bio. But we couldn’t find any trace of either of them online. Chalk it up to common names? 

We then Googled one of the more distinct names: Vice President for Sustainability Dr. Sofia Rossi, who had “spearheaded the ‘Future Fuels’ program, preparing assets for biofuels and hydrogen.” But, again, nothing. The links to their LinkedIn profiles were dead.

When we searched the company’s Texas phone numbers, we found the same numbers listed online for a Houston baklava caterer, a Dallas-area taxi service and an OB-GYN office.

We called the Texas numbers: dead. Then we tried the numbers for the company’s facilities in the Netherlands, Singapore and China. Also dead. 

We were beginning to suspect this company did not actually exist, at least as described on its website. 

What was going on with this website? We looked at the source code and noticed an odd notation, “This feature isn’t implemented yet, but don’t worry! You can request it in your next prompt!”

We checked the site’s domain registration, and we had our (apparent) answer: It was created this year and traced back to a company called Hostinger that offers an AI website builder for $2.99 per month. “Describe it, and AI builds it,” its homepage says. “Appear on Google and AI search automatically.”

Indeed, Google’s “AI Overview” search response, now thrust on users by default with more and more regularity, seemed to ratify the company’s bona fides:

When I searched for an award the company claimed on its website to have won, the Google AI Overview said that “Recent notable recipients include Brownsville Energy Storage Terminals, recognized for their rapid expansion in the independent oil and terminal operations sector.”

Brownsville Energy Storage Terminals is a real LLC. But everything on its website — from its history of the company, to its job postings, a diversity and inclusion policy — appears to be fictional. But perhaps more troubling is that Google, the proprietor of the world’s primary research tool, has rolled out AI Overviews that can indiscriminately take in fake material and authoritatively spit it back out as real.

In response to questions, a Google spokesperson said in a statement: “AI Overviews are rooted in our core Search ranking systems, surfacing reliable and high-quality information for the vast majority of queries. For uncommon search terms like these, there might not be high quality information published that matches the query — and we use these examples to improve our search systems.”

After we reached out to Hostinger, the company pulled down the site. “After receiving your inquiry, we carried out an internal review. Based on the violations identified, we suspended the website and the account behind it in line with our Terms of Service,” a spokesperson said in a statement.

What we encountered is a particular species of a larger problem that is beginning to be better understood. In April, The New York Times reported on an analysis that found Google’s AI Overviews were accurate approximately 9 out of 10 times, noting that that added up to “tens of millions of erroneous answers every hour” given vast search volumes. (A Google spokesperson told the Times that the study has “serious holes.” The company has acknowledged that AI Overviews “can make mistakes.”) 

A BBC reporter wrote a fictional article naming himself the best tech journalist at eating hot dogs, and Google’s AI as well as ChatGPT quickly picked it up and parroted it back.  

And the source material for the AI Overviews also appears eminently gameable, even when not trafficking in actual fiction. “It Is Trivially Easy to Use Reddit to Manipulate AI Search, Research Suggests,” ran a recent headline in 404 Media. 

The mystery website ended up as just a single paragraph in our story. But the larger implication is obvious: fakes, counterfeits and frauds that would have taken considerable effort to create just a few years ago can now be churned out pretty much instantly.

While preparing this piece, we reached out to Calce asking about the site. An attorney for his company, America First Refining, replied to us with a letter dated June 24 that the attorney sent to Hostinger. The attorney also addressed the letter to several email addresses listed on the Brownsville Energy Storage Terminals website.  

“I write to demand immediate removal from the brownsvilleenergyterminals.com website of all unauthorized references to America First’s office address on your website,” the letter said. “As you are aware, America First has no connection or affiliation with the brownsvilleenergyterminals.com website and has not authorized the use of its corporate address there.”

I’m left with lingering questions about the website: What was it for? Was it put up by some malicious actor who simply found the company’s LLC records and decided to create a website? Was it a test site that was mistakenly put online? Or could it have been designed for consumption by someone who was meant to think it was real? 

We don’t know, and our emails to the press contact listed on the website, media@brownsvilleenergyterminals.com, bounced back.

Filed Under: ai, ai overviews, hallucinations, john calce, reporting
Companies: brownsville energy storage terminals, google, hostinger