While it is common knowledge that Big Data challenges exist for companies such as Google, Facebook, and Cisco, the problem is also facing GE in a very significant manner.  A number of GE product lines require the ability to consume and process massive amounts of data.  These include remote monitoring and diagnostics, electronic medical records, smart grid, and credit card processing. But the landscape of possible solutions is a daunting and highly fluid space, with a plethora of emerging technology players and approaches targeting needs not met by traditional players.   No one approach is a hands-down winner; each technology offers its own tradeoff choices between several key performance and maintenance factors.  The combination and augmentation of a variety of technologies, working together, are required to address our needs. Staying on top of the latest developments in the Big Data space is a critical element of our success.

To track the Big Data space we utilized the WISDOM tool that we developed as part of work sponsored by Lockheed Martin. WISDOM is a lightweight information retrieval and analysis architecture that addresses the complex task of gathering, combining, and storing documents to enable in-depth analysis.  WISDOM allows us to specify a topic of interest via a few keywords and then retrieves web posting on that topic for a specified time period. WISDOM provides a seamless integration of two core technologies: information retrieval and text mining. While search engines and technologies like RSS make gathering information easier, they still require a significant amount of consideration when applying them as different search engines may retrieve irrelevant results and it is difficult to impossible to know that all relevant results have been found. Also, doing significant analysis of such documents will usually require the fusion of other information sources (like blog type, inlink counts etc) – a task that most search engines, at least, do not support.  WISDOM is lightweight in that it does not crawl the web but rather relies on making queries to search engines to retrieve web postings. While we can go back in time and retrieve postings on a topic, our typical usage is to make the daily queries to retrieve all posting for that day. Some of the out-of-the-box analysis capability present in WISDOM is the ability to summarize and provide overview of the data, identify top sites and authors and the networks that they form between each other through citations and other references,  show how influence spreads over time, identify named entities, sentiment, online roles and geo-visualization. For additional details see my colleagues and my’s publication in IEEE Second International Conference on Social Computing.

An illustration of the use of the WISDOM technology to assess the Big Data landscape is presented in Figure 1.   WISDOM counted the mentions of company names in titles where the article contained our keyword sets from 1/1/2011 to 11/11/2011 (37,000 posts).  The mentions were aggregated by week and presented using a stream plot.  Significant acquisitions, product rollouts, and partnerships are quickly revealed.  We manually added captions to indicate the event that triggered significant “buzz” for the given company.

By restricting to those articles that contain our acquisition keywords within the title, we gain a clearer picture of the flurry of merger and acquisition activity: giants teaming up with other giants and large players moving quickly to gobble up their small and nimble competitor startups.

As was the case in 2011, 2012 promises to be another year of significant mergers and acquisitions in the Big Data storage and analytics space. By using WISDOM, GE can keep a birds-eye view of this large and quickly evolving ecosystem.