Social Networks, Big Data, Data Quality and You!

Not only is social data Big Data, its Quality Data, too!

For the better part of the last two decades I have advised my clients and employers to push hard on their suppliers to adopt either EDI (Electronic Data Interchange) or, in certain more recent cases XML, instead of trading using phone, fax, email or snail mail.  The costs – both in time and in bad data – are too great to continue executing business transations in a completely manual way.  I remember years ago when the web first made it viable, I also started recommending that when suppliers would not trade electronically with them, companies should push the use of web portals my clients could off-load the data entry – and responsibility for bad data – onto their suppliers.  Unfortunately many companies (not my clients, mind you) abandoned electronic trading altogether when their IT shops found supporting portals was easier than supporting electronic trading…but that is a completely different story.

The thing here is the idea of having your supplier handle the entry and validation of data.  They have a vested interest in making sure the invoice information they provide you is complete and accurate or they might not get paid.  In the same way, I see the the social networks, and other companies, that are aggregating information on all their users leveraging the same model…but it is even better for them.

Every one of us must fill out web forms when signing up for almost any service on line.  Information we share includes name, address, email address, age and many other personal tidbits.  The information becomes the basis of the “consumer as a product” business that these companies have adopted where our virtual selves are sold to the highest bidder – either directly for cash or indirectly through targeted advertising and routes.  In a March 11th column for the Huffington Post, former Federal Trade Commissioner Pamela Jones Harbour wrote “To Google, users of its products are not consumers, they are the product.”

Imagine having a “self-building” product.  It is self-aware and concerned about itself so it naturally wants to make sure the information about it is accurate so on a daily basis it makes itself better.  From a data quality standpoint, there are few better ways to make sure your data is accurate.  Make sure the products – er, those that enter the data – have a vested interest in it.  But the data quality aspect of social data is even better!  For the most part, the data collected from on-line searches, click-throughs, web browsing and everything else you do online – whether collected overtly or covertly – will pretty much be accurate – it is near impossible not to be.

Social data, then, is a data managers dream.  The real challenge is what to do with all the data.  With few data quality issues, data managers responsible for social data are left with working with their business counterparts to figure out the best ways to exploit the data they collect.  Which leads me to an old quote from Google’s ex-CEO Eric Schmidt when speaking with The Atlantic: “Google policy is to get right up to the creepy line and not cross it.”  With Chloe Albanesius at PCMag.com reporting that at least one recent company defector thinking that Google+ has ruined Google, perhaps they’ve stuck their nose just across that line afterall.  Quality Big Data can be a scary thing.

All I can say is “Stay behind creepy line, please!”

Data Quality Doesn’t Get Any Respect

Halloween Costumes have some fuming.  But should stakeholders be the most upset?

David Loshin of DataFlux recently asked “is Data Quality Recession Proof?“ I think the answer is “yes”.  There is no doubt that data quality is still a problem in an economic downturn. In fact, I think it is recession irrelevant.  It continues to have a bad impact and is demanding attention, but I wonder if it is getting any more attention than it has in non-recessionary times.  I believe it should because – as Charles Blyth points out in a comment on David’s blog – organizations are more efficient with greater data accuracy. The fact that so much attention has been paid to new ways to move data (see The Long Tail of B2B Standards by Steve Keifer) that we miss the need to get the data right! 

Is Good Data Really Such an Alien Issue at Target?

I’d say it is time to stop focusing on XML and other means of moving data and spend more time getting the processes, culture and technology in place to assure accuracy of the data.  It does little good to move it quickly or in a new format if the data is bad. 

So, any time is a good time for data quality.  The problem is getting enough enterprise focus on the issue.  Yet when you do get corporate attention, is it for the wrong reasons?  Take Target, for instance.  They were the first to respond to the controversy regarding the “illegal alien” Halloween costumes they (along with Wal-Mart, Walgreens and others) sell.  This weekend some people started suggesting the costumes are not “politically correct” and that caused the risk-adverse retailers to pull the web pages for those products from their online stores.  In Target’s case, the fact that they are (or were) selling the costumes was explained as a “data entry” error (Hey, they got first dibs on this excuse – wonder what Wal-Mart and Walgreens will say?). 

If a data entry error (meaning data quality) is really the cause, was that error in the ordering of the multiple SKUs – like “We didn’t mean to enter in these SKU numbers and quantities along with distribution centers to deliver them to and the dates on which to deliver them”?  Or did they want to order them but not post them on the web site or put them in the store for sale?  In this case was the data entry error in revealing they had a horde of the products that they weren’t planning to sell?  I can see the CEO in a staff meeting:  “Let’s order these things and not sell them, thus keeping everyone from getting them because we think they are not politically correct!”

Let’s face it: Data Quality becomes a convenient scapegoat even when it probably isn’t the problem.  Yet, when it really is a problem it is too often ignored.  I know how bad the data is because in my last role I was able to analyze the quality of data being sent to retailers from their suppliers.  The data crossed numerous industry sectors so we know the problems are persistent and numerous.  We know the companies using the GXS Product Data Quality tool are experiencing better data, however, I suspect that only the retailers are really making use of this data.  Most suppliers don’t take the cleansed data and refresh their own systems with it.  Thus, retailers are ordering from good data and it won’t line up with the data in supplier systems.  The suppliers have to know their data is bad.  Do they just not care? 

In the end, a data quality program that fixes data on one end but not the other of a strategic relationship really is little better than having the same bad data on both ends.  The only difference is that retailers can wield a heavier compliance stick when they know their own data is good.  I think it is time to divest from those suppliers and their retailers.  Since supply chains with divergent data on each end are destined to be less competitive, they aren’t a good place to invest nor a good place to continue holding stocks.  Unfortunately most retail supply chains are like that. 

Getting back to the costumes, these retailers have sophisticated, automated systems.  Even for seasonal products Wal-Mart has decided against seasonal suppliers so they don’t have to manage occasional, manual business relationships.  They want automation with repetitive, reliable and accurate processes with correspondingly good data.  If the information about the costumes in Target’s systems  was wrong I could see it being a data quality problem.  But the fact they were sold in the first place?  Don’t blame it on errors in data – keystroke or otherwise.  There are enough of those without adding additional, fictitious fault.

Data quality just doesn’t get any respect.