Blog Post

Data super friends: can social media and enterprise applications team up?

Everywhere I go in Silicon Valley, people are talking about social media and big data, but rarely do I hear real-world success stories about social media data in a business-to-business context. We all know about the innovations of the consumer Web over the past ten years, including the amazing experiences that companies like Amazon (s AMZN), Apple (s AAPL), Facebook, Google (s GOOG), Yahoo (s YHOO) and others have created by crunching massive amounts of consumer data for personalization. But how are companies in other industries leveraging the big data that’s erupting from social media services?

Social media presents an opportunity to make customer service organizations more responsive and proactive. It gives product development organizations an immediate feedback loop regarding product issues or key features that customers want. With 74 percent of consumers relying on social networks to guide purchase decisions (according to Gartner), marketers drool over the chance to target influencers in social media channels, and to gain a more holistic 360-degree view of each customer. Finally, analyzing social media conversations can be a powerful tool for corporate strategists to uncover market opportunities and other new types of competitive and business intelligence.

While much has been said about these opportunities, few have acknowledged that for this data to be most valuable, it must be integrated with relevant enterprise data from companies’ sales, financial, marketing, customer support, or e-commerce systems (just to name a few).  Say a company correlated social media data with e-commerce sales records, for example.  Looking at this information together, it could learn which social media conversations or reviews correlated most to increases in product sales, and start triggering marketing offers to further capitalize on these types of social media activities as they’re happening.

The roadblocks

Few companies have successfully integrated social media data into their business applications or analysis like this because it’s just not easy.  I’ve been in the integration space for nearly 20 years, and the roadblocks to an effective enterprise data ecosystem have never been greater. As I see it, the five most prominent hurdles to integrating social media data into your business are:

  • Current systems are overwhelmed by constant streams
    The majority of applications in enterprises still run in batch mode, quite different from the real-time stream of the Twitter fire hose.  Even companies that are used to processing frequent batches of data aren’t necessarily prepared for the performance hit they might get to their corporate networks from merging those waves of information with rip tides of data from the ocean of social media.
  • Social media data requires more structure
    Most IT organizations are great at managing structured, relational data in SQL or Oracle databases. However, even the largest consumer Web players are still mastering the semi-structured, non-relational data that exists in the wild Web.  Social media data may not have a well defined data model or schema that defines field names or field types, so it typically requires extra steps when being integrated with traditional relational data.
  • New access methods can be difficult to grasp
    Many companies are using Web APIs for the first time as they reach outside the walls of their enterprise to grab social data.  This requires a whole different knowledge set than the expertise gained from work with on-premise databases – to be successful, all IT departments must have some understanding of Web standards like REST and HTTPS.
  • Data quality is imperative
    With more diverse data ecosystems comes more work in cleansing and parsing “messy” data, which often exists in social media.  These efforts include data matching, de-duplication, standardization, enrichment, etc.  It’s more crucial than ever for companies to address data quality right at the earliest point of entry into the enterprise, and prevent bad data from polluting downstream business applications.
  • Big data crunching is no easy task
    It’s no secret that all the social media sources out there generate massive volumes of data.  For companies to crunch this information, they need to know which data to pull in and which to ignore.  But even after filtering out the noise, we’re talking about data volumes that would have been unimaginable in the 90s.  Now businesses need the power of complex solutions like Hadoop – which are not easy for your average IT person to use.

How to get around those roadblocks

So now that I’ve laid out the roadblocks, let me just say that I still believe there’s a huge untapped potential in connecting social media data with enterprise applications, and it can be done – even if you’re an emerging company without big brain data scientists on staff.  Here are three simple steps I recommend for any business getting started with social media integration:

  •  Prioritize the social media sites
    Consider the popular or up-and-coming social media sites, and think about where your customers and competitors are most likely to be hanging out – Facebook, Google+, LinkedIn, Twitter, Yelp?  Sit down with your marketing team to sort through the characteristics of social media influencers in your ecosystem of customers, partners, suppliers, etc., and compile a list of keywords or phrases you want to track across social media conversations.
  •  Choose your connections
    Identify opportunities for connecting these data sources with other valuable information in your enterprise. Will social media activity be most valuable in the context of your CRM or marketing solutions, when it’s combined with your sales or Web analytics tools, within your business intelligence or master data management systems – or all of the above?
  •  Pick a connection strategy
    Carefully weigh the pros and cons of various connection approaches. You could allocate IT resources to laboriously hand-code each connection in your IT portfolio, you could retrofit a traditional data integration or enterprise application integration tool to accommodate your changing integration needs, or you could evaluate new cloud-based connection technologies that were built with today’s data environment in mind.

Has your company taken any steps to integrate social media into your business operations and systems yet?  If so, how are you tackling this challenge, and do you feel you are achieving the full value of social media data in the enterprise?

An early investor in SnapLogic, Gaurav Dhillon joined as CEO in 2009 when he saw the tremendous potential for the company’s cloud and on-premise integration products, strategy, and unique business model. Gaurav has spearheaded the rapid growth of the company and manages its financing, products, strategic relationships, and operations. Gaurav is the former CEO of Informatica. 

Image courtesy of Flickr user buddawiggi.

8 Responses to “Data super friends: can social media and enterprise applications team up?”

  1. Eric Giles

    Nice post. I’m interested to find out if anyone is aware of activity in and around the semantic markup of social media data. Are their any emerging standards in this space that we should be aware of? Who is already marking up their own web content with semantic tags ready for more automated consumption?

  2. james ferguson

    Integrated Social to Enterprise – no, but integrating – definitely.

    As a start-up in Building Energy Intelligence and a B2B SaaS solution we see huge advantages over our “dinosaurs”.

    New technologies are truly disruptive…

    Firstly, established manufacturers in our space (Siemens, Honeywell, Cisco et al.), often rely on established local capital infrastructure and must maintain compatibility to now obsolete technologies – with smart-meters rolling out the puck moving ahead faster than they can skate (even if they have got the direction about right).

    Energy use to new entrants is just a formatted data stream (it could be a twitter feed) but for the old-guard it is an on-site maintenance nightmare involving big enterprise clients.

    For this reason many SME’s are now getting ahead of big enterprise in basic energy management, watch the space for VC and opportunistic M&A action (Google “Building Energy Intelligence Startup” and see what comes up)

    Second, energy managers of buildings have three constituencies, the team that operates plant and equipment that create an environment, the management of the local core-business operation (who if leasing the building may not even carry energy risk), and most importantly users of the building – PEOPLE.

    So called “building services”are big lumps of steel, cables and ducts (you see actors crawl though them in just about every spy or hostage movie – why are they so clean inside ?)

    As must be obvious, the energy manager of a portfolio of buildings has very little clue of how happy his clients are, and so no idea of the energy being wasted.

    Social media can carry indicative metrics (e.g of how many people are in the building ?) over arbitrary networks. The established B2B utility industry has not even begun to think about how these issues integrate. Their current focus is to identify where existing clients actually keep their buildings. To them for some time, buildings have been totally disconnected from account holders. Show them a zero gas reading in the middle of summer and they do not know if they are dealing with a school that is closed, or a hotel kitchen where a meter reading device fell off the wall.

    Exciting, disruptive and to be settled – yes, but the Enterprise Social network will be bridged – and from the bottom up !

  3. Enjoyed your article. You started off with the Gartner statistic on the influence of social media in driving purchase decisions. We’ve built a company – SocialTwist – around that (I’m the head of Marketing). We help our clients (large brands lik Sara Lee, Kraft and others) make clever use of social media any their marketing campaigns focused on customer acquisition. The points of integration of this type of social outreach is far easier to manage and more simply structured. It is primarily their CRM systems, used to drive further outreach, loyalty and other customer-facing engagements.

  4. Great points. There are challenges in this space that I don’t think can be met with existing tools alone, even with awesome integration software. For those situations, I have started a project where enterprise software developers and integrators can build enterprise social collaboration and microblogging functionality into their own software. It’s a toolkit called Collabinate ( It is still early stage, but I am hoping it helps companies trying to manage their entry into the fray.

  5. Deepak Nayal

    I do not think the question is can they team up, I think it is when will they. This will require good integration between social media analytics and MIS. Not too hard (considering the advancements in big data), but as you mentioned above in your post, it is important to separate the noise from the useful stuff.


  6. Jon Samsel

    Guarav- Nice post with lots of valuable insights for CMO’s, CTO’s and CIO’s to consider. I found myself muttering “Yes” under my breath several times when you made an excellent point. Not sure if you were aware of this or not but connecting big web data to brands is what is all about. Full disclosure, I am the company’s CEO. Give it a spin sometime and you can see all the free competitive insights we provide. Customers are now asking us for API’s so they can integrate our competitive intel, such as social media data, into their enterprises. Exciting times, for sure. Keep up the super posts!

  7. Daria Golubeva

    Gaurav, you’ve raised some really interesting points in this post! It feels that in the past few years social media really changed the way we (knowledge workers) search and perceive information and gain new insights. With the right amount of credibility put into them, social media might be a very valuable source of information not only for individuals, but for businesses as well. Social media give people and easy and friendly way to get connected and generate knowledge together.
    The way social media handle an enormous flow of real-time data and never let people drown in it is the lesson that enterprise software has to learn.