Can AI Really Help Solve the Problem of Overclassification?
A new book overstates the extent to which government secrecy has become “an end in itself.”
What should we do about excessive governmental secrecy? The quantity of documents bearing a classified stamp is staggering, a number probably in the billions, with the U.S. government adding 50 million to the pile every year. President Joe Biden, former Vice President Mike Pence, and former President Donald Trump, have all been caught—the latter in his own very special way—in the coils of our plainly broken system of controlling classified information. With rampant overclassification widely acknowledged, and equally rampant leaking of some of our country’s most precious secrets, clearly reform is order. In what direction should we proceed?
One set of answers comes in a new book by Matthew Connelly, a professor of international relations at Columbia University and principal investigator at something called the “History Lab.” The objective of the lab, which has received funding from the National Science Foundation, is “to apply data science to the problem of preserving the public record and accelerating its release.” To that end, Connelly and his associates have amassed an enormous collection of declassified documents on which they have trained computers (via “big data” and artificial intelligence techniques) to develop an algorithm that can be used to determine which government records should be kept secret and which can safely be declassified. Hence, the title of Connelly’s volume, The Declassification Engine, the engine in question being the computer and associated program that will accomplish this ambitious goal.
Along the way to defending his contraption, Connelly makes an extended argument that as official secrecy has grown out of control our democratic government has devolved into an impenetrable conglomerate of overweening bureaucracies. This “dark state,” as he calls it, is the primary subject of his book.
In Connolly’s telling, the dark state was born of the Japanese surprise attack on Pearl Harbor. The “mishandling of secret intelligence,” he asserts, contributed to the success of the Japanese strike. Yet instead of our government acknowledging that secrecy had been “self-defeating,” authorities have created “ever-more-elaborate systems to conceal information, and to package what they made available to the public.” The doleful consequences of such excessive secrecy have persisted from Pearl Harbor to the present day: “Efforts to dismantle the dark state after World War II failed, and it continued to expand during the Cold War and the ‘Global War on Terror.’”
In a chapter devoted to atomic weapons, Connelly argues that secrecy has been used for perverted ends. The story he tells begins with the Manhattan Project, the massive clandestine bomb-building effort that began in 1942. “Clearly,” Connolly writes, “the secrecy surrounding the Manhattan Project was never just about winning World War II. It was about ensuring American pre-eminence over all potential rivals” and ensuring that a small group of officials within the U.S. government could wield unfettered power without interference from anyone, including “legislators elected to represent the American people.”
The power that came with such secrecy became “an end in itself,” and has become a persistent malady. Over the ensuing decades, “officials at the Pentagon have repeatedly tried to panic the American public by leaking what turned out to be exaggerated estimates of nuclear weapons in the hands of enemy states.” But it has all been a sham: “False alarms about the ‘bomber gap’ and the ‘missile gap’ were deliberate—and successful—efforts to scare up more money for ever-more-advanced weaponry, to say nothing of Iraqi WMDs and ‘regime change.’”
The problem, argues Connelly, lies in the “military-industrial complex.” Here he sees an abdication by civilian authorities in Congress and the White House in exercising their constitutional responsibilities. The upshot is that the military, in conjunction with private enterprise, has run roughshod over norms and laws:
Unaccountable spending, secret military plans, collusion with private industry, and the refusal to keep a record of how all this gets decided at the highest levels are all manifestations of a slow and silent coup—not Seven Days in May, but almost eighty years of deterioration in military discipline. The dirty secret of civil-military relations—that our elected civilian leadership is not in fact in charge of the military—is becoming an open secret.
What are we to make of Connelly’s argument? The book has received respectful reviews in both the New York Times and the Washington Post and earned blurbs from writers whom I admire, including Anne Applebaum and Nicholas Lemann. Some of the praise it has earned is deserved. Drawing on the vast trove of documents in his lab’s collection, Connelly amply illustrates some of the abuses and excesses of secrecy. His discussion of medical experiments performed by the CIA on mentally ill criminals and on its own personnel is particularly harrowing. In chapters that cover surveillance, cryptography, and what he calls “weird science,” he shows how secrecy has been used on countless occasions to cover up blunders, incompetence, and embarrassment.
That is all well and good, but what leaps out to this reviewer is that his accounting of our history is a kind of left-wing mirror of current right-wing memes and talking points. His volume is riddled with a kind of conspiratorial thinking throughout. What is his out-of-control “dark state” but a version of the out-of-control “deep state” of the MAGAverse?
To begin with Pearl Harbor, is it really true that excessive American secrecy is what enabled the Japanese to pull off the surprise attack? It is of course indisputable that the U.S. success at breaking Japanese codes was a closely held secret, and that secrecy interfered with the timely flow of intelligence among branches of the defense establishment, contributing to the debacle. But it is equally true that lax secrecy also contributed to Pearl Harbor. Though Connelly does not discuss it, a series of leaks in the 1920s and ’30s about the U.S. prowess in codebreaking by the American codebreaker Herbert Yardley led the Japanese to tighten their cryptographic security, slowing the pace at which the United States could read encrypted Japanese communications. Both excessive secrecy and lax secrecy played their part in leaving the United States vulnerable. Surprise attack is never a simple story, and the task of balancing secrecy and openness cuts two ways.
Connelly’s treatment of the Manhattan Project is similarly one-sided. Yes, the bomb-building effort was wrapped in extraordinary secrecy, famously excluding even members of Congress from knowledge of the project. But how could it be otherwise in a race with Nazi Germany to develop a weapon that could annihilate entire cities? As it turned out, if anything, the Manhattan Project was not secretive enough. As is well known, and as Connelly himself discusses, the bomb-building effort was extensively penetrated by Soviet intelligence operatives.
Attributing the “missile gap” and the “bomber gap” of the 1950s to “deliberate—and successful—efforts to scare up more money for ever-more-advanced weaponry” is a hoary left-wing staple. The real story is less sinister and more complex. In the pre-satellite, pre-U-2 era, the range of uncertainty about Soviet missile and bomber development was exceedingly wide. If U.S. estimates turned out to be high, that was partly the consequence of Soviet deception—Moscow’s attempt, for example, to make their bomber fleet appear larger than it was. Throughout the 1950s, there was an intense and good-faith debate within American intelligence about the true size of the Soviet Union’s bomber and missile force. If estimates came in high, that was more the result of a mixture of fear and prudence than an organized conspiracy to justify higher spending on military gadgetry.
Accompanying a one-sided historical narrative is sloppiness about essential matters. Connelly’s treatment of the Espionage Act of 1917 is a case in point. At one juncture, he pronounces that under the Espionage Act, “if an official passes along [secret] information, the person receiving it bears no risk of prosecution, even if they publish it on the front page of The New York Times.” But, of course, if the recipient of the secret information is a foreign agent, he or she would certainly be subject to prosecution. Furthermore, even when journalists receive secret information they are at risk for prosecution. Connelly himself cites the case of the Chicago Tribune, which during World War II received and then published ultra-sensitive information suggesting that the United States had cracked Japanese codes. Contradicting his own pronouncement, Connelly asserts that a grand jury investigation—and presumably therefore an ensuing prosecution under the Espionage Act—of the Tribune was “appropriate.”
Let’s return now to the question of whether AI can be used to declassify government secrets. In attempting to persuade government agencies to employ his lab’s algorithm, he has thus far run into a wall of skepticism. Of course, computers and AI are going to make untold progress over the next decades, and it is impossible to rule anything out. But as matters stand now, I would not entrust life-and-death decisions about sensitive national-security secrets to the whims of ChatGPT or Microsoft’s new version of Bing or any likeminded AI program. Not that human judgment about secrecy is flawless. Connelly’s book, intentionally—and unintentionally—illustrates that it is not.
Correction (Feb. 27, 2023, 7:30 a.m. EST): As originally published, this article included a passage misstating the nature of the Espionage Act of 1917 and its relationship to the Sedition Act of 1918, imputing to Matthew Connelly a factual error that he did not make. We apologize for the error.