NIST Big Data Definitions and Taxonomy Generates Probing Questions

Many interesting questions came up in the NIST Definitions and Taxonomy Big Data group meeting today. Brilliant minds are hard at work to stabilize the language around Big Data, but some fundamental questions have been posed that the marketplace seems to believe we have already solved.

  • How do we differentiate Big Data from traditional big data like sensor feeds, credit card processing, and financial transactions. What makes it different? One noted professional taxonomist asserted that a basic differentiator may exist in the variability and variety of data.
  • Has data lifecycle changed with BD? The subgroup lead Nancy Grady made a compelling argument that the position of storage in the workstream may be of interest. She pointed out that traditional decision support transforms and stores the data prior to analysis, whereas the Big Data paradigm frequently stores data raw and applies structure later (schema on read).
  • Should there be an obsolescence characteristic attached to data definitions? Ubiquitous sensors (The Internet of Things) may present disposable data with immediate obsolescence which climate monitoring sensors only provide value at a future date.
  • Data cleanliness may be less important than traditional BI.
  • Are there certain enablers to Big Data that should be assumed in planning such as (perhaps) cloud computing?

At this point it is obvious there is no consensus on these questions, but what do we as a community of practitioners think about these questions?

 

The working group meetings are highly compelling and I encourage anyone who wishes to become involved to go to the group site, http://bigdatawg.nist.gov/

NIST Big Data Reference Architecture Sub-Group

I’m excited to report I have been selected as a co-chair of the National Institute of Standards and Technology (NIST) Big Data Working Group’s Big Data Reference Architecture sub-group.  I had the opportunity to observe the Definitions and Taxonomy sub-group meeting this morning and the two-hour teleconference was of inestimable value.  The minds Dr. Wo has arranged to lead and participate in these groups is stunning and I am honestly humbled by my inclusion.  I’ll be posting highlights on this site and www.analyticbridge.com (one of Dr. Vincent Granville’s contributions to the community).