Big Data, Small Data

"Big Data" has become the new darling of the IT world. Since May, 2011 Google references to it have jumped by a factor of five. "Data Mining" on the other hand has fallen by a factor of three since its peak in 2004.





The IT industry has a way of reinventing itself, and changing the language to suit. One could argue that these two topics are the same; the difference may be one of scale.

The challenge for non-profits with respect to modern views of data are on two ends of a long spectrum. On the one hand, is the opportunity to gain insights and take actions based on mining big data trends (Google.org Flu Trends data comes to mind). On the other hand, is what I'd call the challenge of small data, namely the lack of good data (input and impact measures come to mind) and the relatively small, siloed data that is not being shared. 

There are two sets of motivation problems at each end of this spectrum. On the big data side is how to motivate non-profits to look at the possibilities in mining what are external data, and as Gisli Olafsson recognizes, is often at the wrong "zoom level" to be actionable. On the small data side, what is the incentive to share data and develop common standards? For corporations, it's the expectation of accessing larger consumer markets. I doubt altruism alone will drive NGOs to share. There has to be a hunger and scarcity on the one hand (what drove the NetHope members to collaborate) or a large economic benefit (also a NetHope driver, through its member deals and discounts). Otherwise, I don't see non-profits sharing data in a meaningful, aggregated way.

For further thoughts about Big Data, see the Wikipedia entry on Big Data and the Gartner definition: "Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization." That's a mouthful definition.

On the small data side, look at the case of Blackbaud's donor data products and how they have motivated many NGOs to share their most private data: about donors and their giving transactions. Chuck Longfield, Blackbaud's Chief Scientist solved the problem of "what's in it for me" by creating a huge economic benefit to sharing data: learning how my fundraising prowess stacks up against an aggregate of my peers and what I can therefore do differently.

Whether big or small data, motivation to share and use is the key. I suspect the small data issues need to be solved for NGO's before we will see the benefits of big data extending to our sector.

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."