There's something really interesting in here. Filtering through giant webs of data to find significant connections and patterns. Taking huge amounts of information and watching the dots connect to each other, resulting in useful info.
Taking Spying to Higher Level, Agencies Look for More Ways to Mine Data - New York TimesA small group of National Security Agency officials slipped into Silicon Valley on one of the agency's periodic technology shopping expeditions this month.
...
The tools they were looking for are new, but their application would fall under the well-established practice of data mining: using mathematical and statistical techniques to scan for hidden relationships in streams of digital data or large databases.
Supercomputer companies looking for commercial markets have used the practice for decades. Now intelligence agencies, hardly newcomers to data mining, are using new technologies to take the practice to another level.
But by fundamentally changing the nature of surveillance, high-tech data mining raises privacy concerns that are only beginning to be debated widely. That is because to find illicit activities it is necessary to turn loose software sentinels to examine all digital behavior whether it is innocent or not.
...
Data mining is already being used in a diverse array of commercial applications — whether by credit card companies detecting and stopping fraud as it happens, or by insurance companies that predict health risks. As a result, millions of Americans have become enmeshed in a vast and growing data web that is constantly being examined by a legion of Internet-era software snoops.
Technology industry executives and government officials said that the intelligence agency systems take such techniques further, applying software analysis tools now routinely used by law enforcement agencies to identify criminal activities and political terrorist organizations that would otherwise be missed by human eavesdroppers.
One such tool is Analyst's Notebook, a crime investigation "spreadsheet" and visualization tool developed by i2 Inc., a software firm based in McLean, Va.
The software, which ranges in price from as little as $3,000 for a sheriff's department to millions of dollars for a large government agency like the Federal Bureau of Investigation, allows investigators to organize and view telephone and financial transaction records. It was used in 2001 by Joyce Knowlton, an investigator at the Stillwater State Correctional Facility in Minnesota, to detect a prison drug-smuggling ring that ultimately implicated 30 offenders who were linked to Supreme White Power, a gang active in the prison.
Ms. Knowlton began her investigation by importing telephone call records into her software and was immediately led to a pattern of calls between prisoners and a recent parolee. She overlaid the calling data with records of prisoners' financial accounts, and based on patterns that emerged, she began monitoring phone calls of particular inmates. That led her to coded messages being exchanged in the calls that revealed that seemingly innocuous wood blocks were being used to smuggle drugs into the prison.
"Once we added the money and saw how it was flowing from addresses that were connected to phone numbers, it created a very clear picture of the smuggling ring," she said.
...
Mr. Arquilla, who was a consultant on Admiral Poindexter's Total Information Awareness project, said that the $40 billion spent each year by intelligence agencies had failed to exploit the power of data mining in correlating information readily available from public sources, like monitoring Internet chat rooms used by Al Qaeda. Instead, he said, the government has been investing huge sums in surveillance of phone calls of American citizens.
"Checking every phone call ever made is an example of old think," he said.
He was alluding to databases maintained at an AT&T data center in Kansas,
which now contain electronic records of 1.92 trillion telephone calls, going back decades. The Electronic Frontier Foundation, a digital-rights advocacy group, has asserted in a lawsuit that the AT&T Daytona system, a giant storehouse of calling records and Internet message routing information, was the foundation of the N.S.A.'s effort to mine telephone records without a warrant.
An AT&T spokeswoman said the company would not comment on the claim, or generally on matters of national security or customer privacy.
But the mining of the databases in other law enforcement investigations is well established, with documented results. One application of the database technology, called Security Call Analysis and Monitoring Platform, or Scamp, offers access to about nine weeks of calling information. It currently handles about 70,000 queries a month from fraud and law enforcement investigators, according to AT&T documents.
A former AT&T official who had detailed knowledge of the call-record database said the Daytona system takes great care to make certain that anyone using the database — whether AT&T employee or law enforcement official with a subpoena — sees only information he or she is authorized to see, and that an audit trail keeps track of all users. Such information is frequently used to build models of suspects' social networks.
The official, speaking on condition of anonymity because he was discussing sensitive corporate matters, said every telephone call generated a record: number called, time of call, duration of call, billing category and other details. While the database does not contain such billing data as names, addresses and credit card numbers, those records are in a linked database that can be tapped by authorized users.
New calls are entered into the database immediately after they end, the official said, adding, "I would characterize it as near real time."
According to a current AT&T employee, whose identity is being withheld to avoid jeopardizing his job, the mining of the AT&T databases had a notable success in helping investigators find the perpetrators of what was known as the Moldovan porn scam.
[o]
