A Media Watchdog Is Helping To Train AI Models. It Says Chinese Propaganda Is More Reliable Than Many American News Sources.

In February 2023, a few months after ChatGPT's public release, the media watchdog NewsGuard announced a new tool for artificial intelligence companies. The company, which rates the credibility of news outlets on a 100-point scale, had been selling its data to advertisers with the goal of steering them away from "unreliable" sources. Now it would license the same data to AI companies in a "machine-readable" format, ideal for training chatbots to avoid "misinformation."

NewsGuard—whose cofounder, Steven Brill, suggested in 2020 that the New York Post's Hunter Biden laptop story was a "hoax"—said its ratings could help models "identify trustworthy news and information sources." It encouraged AI companies to use the data during the "supervised learning" process, a technique that influences how models evaluate information and aligns their outputs with developers' preferences.

"Without tools like NewsGuard for AI, which can be used during supervised learning to train transformer models, AI language models risk becoming … the 'next great misinformation superspreader,'" the company said in a press release. "To guard against political bias, these ratings are strictly apolitical."

Left out of the press release was a technical detail that might trouble truth-seeking AI developers: A model trained on NewsGuard's ratings would treat many American sources as less reliable than Chinese propaganda.

NewsGuard, which warns that "foreign disinformation" has "infected" the responses of Western chatbots, gives the state-owned China Daily a reliability score of 44.5—more than twice the score of the conservative outlet Newsmax, which earns just 20.

The company claims that China Daily "does not publish false or egregiously misleading content at a significant level," a benchmark worth 22 points in NewsGuard's rating system. By contrast, "Newsmax.com has advanced false or egregiously misleading claims on health, U.S. politics, and the Jan. 6, 2021, attack on the U.S. Capitol, such that a visitor to the site risks being seriously misled."

Visitors to The Federalist and One America News Network likewise risk "being significantly misled on important topics," scoring 17.5 and 22.5 respectively. China Global Television Network, on the other hand—which NewsGuard describes as a "mouthpiece of the Chinese Communist Party"—scores a 44.5, just 5 points shy of the Daily Wire (49.5).

The ratings are a stark example of how one of the top disinformation watchdogs of the Biden era is hoping to embed its worldview into generative AI, turning what critics say are biased judgments about credibility into notionally objective training data.

Founded in 2019 by Brill and Gordon Crovitz, a former publisher of the Wall Street Journal, NewsGuard has been accused of demonetizing conservative websites by flagging them as a brand risk for advertisers. The company also licenses its data to a content moderation platform, Safety Kit, used to flag "misinformation" on social media, and offers browser extensions for Safari, Microsoft Edge, and Google Chrome.

Now, as chatbots replace search engines as Americans' go-to research tool, NewsGuard is hoping to shape the outputs of a far more powerful technology—one that critics say is at risk of being distorted by what are inherently subjective judgments.

"The obvious danger of NewsGuard's entry into AI is that it induces the same censorious changes to AI that it did to media and social media," said Mike Benz, a former State Department official and the founder of the Foundation for Freedom Online, a nonprofit that advocates against digital censorship. "The fear every citizen should rightly have is that a highly biased private firm could influence or effectively control the sources from which AI derives its decisions, the same way NewsGuard's blacklists were used to control the media sources to which audiences could have access."

Gene Hamilton, the president of America First Legal, which in 2025 released documents showing ties between NewsGuard and the Biden State Department, said the question ultimately comes down to "who decides?"

"It's the same fundamental issue as before, but AI makes it even easier to silence dissenting views," said Hamilton. "Who decides what's reliable and what's not?"

At NewsGuard, the answer is usually Democrats. A Washington Free Beacon analysis found that 20 NewsGuard executives and advisers contributed to Democratic candidates between 2015 and 2022, while just 6 gave to Republicans. The company's ratings exhibit a similar tilt, with multiple studies finding that left-leaning outlets score higher on average than right-leaning ones, though there are plenty of exceptions.

NewsGuard, which received a $750,000 contract from the Pentagon in 2021, also cultivated ties with the Biden administration. After the White House secured a host of "trust and safety" commitments from top AI companies in 2023, NewsGuard launched an auditing service to help the companies meet their "White House Commitments," positioning the firm as an informal part of the Biden regulatory apparatus.

The company also unveiled "a suite of protections to safeguard AI products from election misinformation." "In a year with more than 40 major elections worldwide," a 2024 press release said, "the new tools will protect AI products from being exploited or causing inadvertent harm."

It is unclear which AI companies have purchased data from NewsGuard, which does not disclose its clients, though Anthropic and Microsoft both said that they do not use NewsGuard data to train their models. (Neither OpenAI nor Google Deepmind responded to requests for comment.) And while the ratings firm has made AI a centerpiece of its marketing, NewsGuard COO Matt Skibinski downplayed the ratings' impact on artificial intelligence models, stating in an email that "AI companies do not license our data about domestic news sites."

"The only AI-industry use of our data has been through our FailSafe for AI product," he said, referring to a subset of NewsGuard's data focused on Russian, Chinese, and Iranian sources. He added that "AI companies would be welcome to license our full dataset if they wished to do so."

In a lengthy exchange with the Free Beacon, Skibinski also accused this reporter of "cherry-picking," noting that many conservative outlets, including Fox News and the New York Post, score well above China Daily in NewsGuard's ratings.

"Your statement that NewsGuard gives 'lower ratings to rightwing American outlets than to Chinese state-run media' is not accurate, no, despite the cherry-picked set of sites you describe here," Skibinski wrote in an email. "If what you're instead asking is whether a particular website, Newsmax, had so much false or egregiously misleading content on its site that it somehow managed to score even lower than a Chinese state media outlet, that is true—but the fact that so many other conservative-leaning sites are rated credible is evidence enough that Newsmax's score is not based on its political leaning."

But for NewsGuard's critics, such determinations are themselves the product of cherry-picking. The ratings reflect a series of subjective criteria that, by the company's own admission, come down to judgment calls, such as whether an outlet takes "quotes out of context" or uses "opinionated language" in news stories.

The elasticity of those criteria leaves plenty of room for them to be applied unevenly. And it's what has allowed NewsGuard to portray privately owned, right-leaning outlets as less reliable than Chinese state media, an irony noted by the Federal Communications Commission in a letter to tech platforms about the company's ratings.

"NewsGuard's own track record raises questions about whether relying on the organization's products would constitute 'good faith' actions within the meaning of Section 230," FCC commissioner Brendan Carr wrote in the 2024 letter. "For one, reports indicate that NewsGuard has consistently rated official propaganda from the Communist Party of China as more credible than American publications."

The most important factor in those ratings, a pass/fail criterion worth 22 points, is also among the most subjective. Outlets fail the benchmark if, "in NewsGuard's judgment, a visitor to the site risks being significantly misled on important topics," due to the quantity of "false or egregiously misleading content" on the site.

It is not clear how NewsGuard defines "important topics." The company notes that China Daily, which is run by the Chinese Communist Party's Central Propaganda Department, has "published misleading and unsubstantiated content advancing the views of the Chinese government," in part by denying the "considerable evidence" of "human rights violations against Xinjiang's Uyghur community." But Skibinski said the "frequency of false or egregiously misleading claims … was not high enough for the site to fail that criterion."

By contrast, Skibinski continued, Newsmax had published "enough" false or misleading claims to incur the 22-point penalty in NewsGuard's latest review of the site. The alleged inaccuracies included an interview with a public safety consultant who concluded, based on video evidence, that Renée Good was "accelerating directly toward" an ICE agent before she was killed in Minneapolis, a conclusion that contradicted analyses by the Washington Post and New York Times.

"We use the term 'significant' mostly to separate false claims like these from minor factual errors such as spelling someone's name wrong or misstating a date," Skibinski said. "We do not rank the significance of each false claim in the way you seem to be implying."

Skibinski also said that the ratings only regard a claim as false if there is "clear evidence" contradicting it—meaning that countries that suppress such evidence can inflate the score of their state-run media.

"Due to China's stranglehold over information, it's of course possible that the site publishes some claims that are false, but which can't be proven false with available evidence," Skibinski conceded. "While you may see this as a weakness of our system, we see it as far better than the alternative: somehow venturing to guess at the veracity of claims rather than basing our assessments on evidence."

Hamilton, the president of America First Legal, said it would be better not to rate Chinese outlets at all.

"There's no measuring stick for a NewsGuard to measure the veracity of a state-controlled news outlet in China," Hamilton told the Free Beacon. "You're never going to get actual, unbiased information."

NewsGuard does warn that China Daily "generally fails to maintain basic standards of accuracy" and that readers should "proceed with caution." And its ratings don't always track the politics of its staffers: Fox News, the nation's most influential conservative TV network, scores 20 points higher than its left-leaning counterpart MSNBC, which, in NewsGuard's words, "has published some false or misleading claims."

But NewsGuard itself has made false claims about the websites it rates. Take the company's entry on the Free Beacon, which until recently claimed that the paper "has occasionally published false or egregiously misleading information."

The only example cited by NewsGuard was an August 2024 article about a nonprofit that helps doctors register their patients to vote, Vot-ER. The article stated that the group had partnered with Epic Systems, the primary health records software in the United States, to integrate questions about voter registration into patients' medical charts.

That statement was backed up by a hyperlinked page from Vot-ER's parent website. But NewsGuard claimed—falsely—that the partnership had no basis in reality, attributing the "false claim" to a "satirical post Epic posted" on April Fools Day. (The Free Beacon could not find any record of such a post.)

NewsGuard also quoted an Epic spokesman who claimed that the company's software "does not collect voter registration information." But Vot-ER's parent site had stated that "voter registration information is seamlessly integrated into the After Visit Summary … and is made easily accessible via MyChart," Epic's main software product.

Another webpage advertised an August 2024 workshop titled "MyChart, My Vote: Epic's Nonpartisan Voter Registration Build," which would be led by an "Epic expert," Philosophy Walker, "who built new voter registration tools."

Presented with both pages, Skibinski agreed to issue a correction to the Free Beacon's "Nutrition Label," the written summary that accompanies NewsGuard's ratings. But he refused to acknowledge that NewsGuard had made a factual error, insinuating, without evidence, that the webpages were misleading.

"To be clear, we have not seen evidence that Epic actually did add any feature for capturing voter status, which the article in question seemed to imply," Skibinski said. "But you've raised enough legitimate questions about this story that we think the best thing to do is to remove this from the Nutrition Label—especially since the example itself did not impact the site's score on any of our criteria."

The Free Beacon has an 80 on NewsGuard's ratings. It loses 7.5 points for disclosing its ownership on the terms of service page rather than "in a user friendly manner" and another 12.5 points over its "handling of opinion content."

"Because FreeBeacon.com does not disclose its conservative perspective, yet publishes content advancing that point of view, NewsGuard has determined that the website does not handle the difference between news and opinion responsibly," the site's Nutrition Label reads. "Such a disclosure does not appear on FreeBeacon.com," which features bombs and an American flag and states that it 'covers the enemies of freedom the way the mainstream media won't,' yet the site regularly criticizes Democrats while advancing a conservative perspective in its story selection and framing."

The post A Media Watchdog Is Helping To Train AI Models. It Says Chinese Propaganda Is More Reliable Than Many American News Sources. appeared first on .