Dark Data

A while back there was a lively debate on Broken Spines about whether or not it is possible to succeed without failing.  In the end, the pugilists involved resorted to name calling and feelings were hurt.  So, we didn’t get to explore a more interesting idea: do we put too much emphasis on success – when learning from our failures is what’s truly valuable?

Google, in their Palimpsest project, will soon provide a home for terabytes of open-source scientific datasets.  This will hopefully correct “publication bias”, where science gets skewed because only positive correlations see the light of day.  Thomas Goetz sums this up best in a Wired article called  Freeing the Dark Data of Failed Scientific Experiments”.  Here are some exerpts:

So what happens to all the research that doesn’t yield a dramatic outcome — or, worse, the opposite of what researchers had hoped? It ends up stuffed in some lab drawer. The result is a vast body of squandered knowledge that represents a waste of resources and a drag on scientific progress. This information — call it dark data — must be set free.

in this data-intensive age, those apparent dead ends could be more important than the breakthroughs. After all, some of today’s most compelling research efforts aren’t one-off studies that eke out statistically significant results, they’re meta-studies — studies of studies — that crunch data from dozens of sources, producing results that are much more likely to be true. What’s more, your dead end may be another scientist’s missing link, the elusive chunk of data they needed. Freeing up dark data could represent one of the biggest boons to research in decades, fueling advances in genetics, neuroscience, and biotech.

0 Responses to “Dark Data”

  1. Leave a Comment

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


Top Posts

Blog Stats

  • 91,917 hits

%d bloggers like this: