The comparison contains 103 criteria

The matrix and comparison contains 103 selection criteria and all the major BI tools in the market like Oracle BI, IBM Cognos, QlikView, SAS, Business Objects, WebFOCUS and so forth. All these tools are thoroughly reviewed and all the answers of the vendors were validated by live demonstrations, usage or official manuals.

A useful BI tools comparison-matrix

The matrix, displayed below, can be used for different purposes like selecting one or more BI tools that met specific requirements or it can be used to see how the BI tools score on different aspects like core functionality, infrastructure, performance issues, connectivity and so on. Sixteen vendors are included in the BI tools comparison matrix.

The complete list of business intelligence tools displays all the major tools available in the market and the tools which are covered in the BI Tool Survey.

A research paper with good testimonials

The BI Tool Survey is a research paper with all the information you need to know for selecting the right BI & reporting tool. All the important features of Business Intelligence tools are covered, also their strengths and weaknesses.

“The BI Tool Survey has been immediately useful to me” says Richard Silverstein, Strategic Planning Manager of the Sweet & Maxwell Group. And Vera Jonkers, consultant to Parsons Brinckerhoff Africa, said the following: ”the BI Tool Survey report is excellent“. Discover the power of this report for yourself and order the report today. We are happy to welcome you as our customer.

CAMBRIDGE, Mass. – 08 Nov 2011: New research released today by MIT Sloan Management Review and the IBM Institute for Business Value (NYSE: IBM) reports that organizational challenges, more so than technology hurdles, are holding companies back from fully integrating analytics across their enterprises.

According to a global survey of more than 4,500 executives, managers and analysts from more than 120 countries and 30 industries, 44 percent of organizations say cultural barriers to enterprise-wide analytics adoption, such as the requirement for new leadership competencies and organizational resistance to new ideas, are the primary barriers.  In contrast, only 24 percent point to technology concerns.

Org Challenges Graphic:  New research from IBM and MIT Sloan Management Review reports that cultural barriers, more so than technology hurdles, are holding organizations back from fully integrating analytics across their enterprises.  This chart shows the percentage of respondents who rate these challenges as extremely difficult to resolve.

The new report, entitled “Analytics: The Widening Divide,” builds on the findings from the original study by MIT SMR and IBM in 2010 to understand how companies are embedding analytics in more of the enterprise’s processes and operations.  The 2010 study found organizations fall into one of three levels of sophistication: basic users referred to as Aspirationals, followed by the more Experienced users, and the most advanced users referred to as Transformed.  Year-to-year comparisons reveal that the more sophisticated users are expanding their deployment of analytics and widening the performance gap over their peers.

For instance, from 2010 to 2011 the percentage of respondents who cited a competitive advantage using analytics grew 23 percent for Transformed and 66 percent for Experienced organizations.  These same organizations are more than twice as likely to substantially outperform their competitive peers. In contrast, Aspirational organizations lost ground in competitiveness, falling 5 percent since last year.

“Our new research shows that the early and aggressive adopters of analytics make significant gains in both performance and overall competitiveness,” said Fred Balboni, IBM’s global leader, Business Analytics and Optimization.  ”These indicators point to an urgent need for organizations to foster a data-oriented culture and drive an analytics strategy that embeds fact-based insights into decisions and processes at every level of the business.”

Competitive Advantage Graphic:  The ability of organizations to create a competitive advantage with analytics has surged in the past 12 months, according to new research from IBM and MIT Sloan Management Review.  This chart shows the percentage of respondents who cited a competitive advantage using analytics, year over year, grouped by analytic sophistication levels

“We’ve found that there are three legs to the competitive analytics stool: a data-oriented culture, information management competency, and analytics expertise,” said David Kiron, executive editor for MIT Sloan Management Review.  ”Companies that have all three use analytics to deliver advantage in the marketplace.”

The study found that the majority of organizations are using analytics to manage their financial and operational activities, but are less likely to rely on analytics-based insights for decisions in other key areas.  On average, less than 25 percent of Aspirational organizations, and one-half of Transformed organizations, say they rely on data and analytics to make decisions involving customers, business strategy and human resources.  Even Transformed organizations are not using analytics to their fullest potential, indicating ample opportunities for advanced users to do more and for less sophisticated organizations to create a competitive advantage by targeting analytics at key strategic activities.

While Transformed organizations use analytics more broadly across the organization than their peers, they differentiate themselves by intensely focusing on applying analytics to three areas:

• Increasing the speed of decisions – Transformed organizations are more than three times more likely than Aspirationals to focus intensely on making better decisions, faster.
• Managing enterprise risks – Eighty-six percent of Transformed organizations are addressing the full range of organizational risks that can impact their business, while none of the Aspirational organizations have the same level of focus.  Transformed organizations are using analytics to not only mitigate, but also anticipate risks.
• Engaging customers – Transformed organizations are outpacing their peers in leveraging the enormous amounts of data available today to understand and engage with their customers in new ways.  Two-thirds of them are putting analytical insights into the hands of customer-facing employees to drive sales and productivity — compared to one-fourth of Aspirationals.

The study examines how Transformed organizations are creating an advantage in the marketplace.  The analysis shows that of all the characteristics exhibited by this group, their proficiency in six areas (represented by the percentage of Transformed companies that say they possess these characteristics) distinguished them the most:

• Ability to analyze data – 78%
• Ability to capture and aggregate data – 77%
• Culture open to new ideas – 77%
• Analytics as a core part of business strategy and operations – 72%
• Embed predictive analytics into process – 66%
• Insights available to those who need them – 65%

To access the full report, visit MIT SMR or IBM.  Go here for more information on the MIT SMR/IBM joint New Intelligent Enterprise project.

About MIT Sloan Management Review

A media company based at the MIT Sloan School of Management, MIT Sloan Management Review’s mission is to lead the conversation among research scholars, business executives and other thought leaders about advances in management practice that are transforming how people lead and innovate. MIT Sloan Management Review captures for thoughtful managers the creativity, excitement and opportunity generated by rapid organizational, technological and societal change. Source: IBM.

Just as QlikTech disrupted the BI industry by empowering all business users with interactive data analysis instead of static reports, the company is now leading the next transformation to collaborative decision-making that leverages the power of the collective intelligence of a group, organization or department to accelerate better decisions with greater alignment and transparency.

QlikView 11 also marks the next generation of associative search pioneered by QlikView with the introduction of comparative analysis. Comparative analysis extends the QlikView associative experience to enable interactive comparison of user-defined groupings. Other key benefits of the release include making application development easier and faster, and enhancing QlikView’s manageability and security capabilities for large enterprises.

Customers are realizing the value and ease of QlikView upgrades as illustrated in the recently-announced BI Survey 10, conducted by the Business Application Research Center (BARC). With the largest number of participants in this year’s BI Survey, BARC analysts report “remarkably, all those who specified they used [QlikView] stated that they used Version 10, the current version.”

“Our vision for QlikView 11 builds on the fact that decisions aren’t made in isolation, but through social exchanges driven by real-time debate, dialog, and shared insight,” says Anthony Deighton, CTO and senior Vice President, Products at QlikTech. “QlikView 11’s social business discovery approach allows workgroups and teams to collaborate and make decisions faster by collectively exploring data, anywhere, anytime, on any device. Business users are further empowered with new collaborative and mobile capabilities, and IT managers will appreciate the unified management functionality that allows them to keep control and governance at the core while pushing usage out to the edges of the organization.”

Social Business Discovery Accelerates Decision Making through Collaboration

QlikView 11 puts the social and collaborative experience front and center. With social business discovery, workgroups and teams can collaborate by collectively exploring data in real time, as well as jointly creating analysis to quickly answer business questions and arrive at informed decisions.

• Co-Create with Collaborative Workspaces – QlikView users can invite others – even those who do not have a license – to participate in live, interactive, shared sessions. All participants in a collaborative session interact with the same analytic app and can see others’ interactions live. From their tablet or desktop or laptop computer, users can share selections and test scenarios together to better uncover insights, solve problems, and make decisions.
• Provide In-Context Commentary – Users can now post and respond to notes right in the context of any QlikView app. During the creation of an app, this could include communication about changes to the app’s design, or new or easier ways to view data. Throughout the usage of an app it could include questions, answers, and insights about the data. Notes and comments can be added to charts, graphics, and visualizations, and are accessible as threaded discussions.

Comparative Analysis Drives Deeper Insight

With QlikView users are not limited to predefined paths they must follow, or questions they must formulate ahead of time. They ask what they need to ask, and explore up, down, and sideways and now even side-by-side.

Business users can now create multiple selection states in a QlikView app; they can create graphs, tables, or sheets that are based on different selections, then compare multiple selections of data inside one document, one chart, or one object, to spot trends, outliers, or differences. This makes it easier to gain new insights into patterns of use, opportunities and threats.

Comparative analysis is used for contrasting multiple data sets like market basket comparisons in a retail setting or stock performance analysis in financial services. For instance, to compare the performance of two or more groupings of products, users simply select multiple products for each group and QlikView automatically visually contrasts their performance—across regions, time periods, or any other measure.

Mobile Users Gain Insight “On Location”

QlikView 11 on Mobile delivers the complete business discovery experience – including interactive analysis, rich visualization, associative search, and the ability to share apps, data and decision trails. QlikView takes full advantage of the immersive, interactive potential of touchscreen tablets like the Apple iPad, Android tablets, and BlackBerry Playbook. QlikView 11 makes optimal use of small touch screens such as the Apple iPhone or Android phones by presenting a single chart or graph at a time. By leveraging advanced HTML5 browser technology, QlikView delivers a touch-driven interface that makes BI as easy and compelling to use as consumer apps, while IT retains centralized security and manageability.

Enhanced Enterprise Capabilities for Control at the Core

With QlikView 11, IT can serve the business like never before – all while assuring strict data security, quality, and governance. Customer-driven enhancements include extending support for very large deployments with improved scalability, ease of administration and security management. Automatic load balancing ensures maximum performance even with thousands of users while extensive APIs broaden integration with business applications and system management software.

Rapid App Building Marks the End of the End User

In just a fraction of the time it takes with typical Business Intelligence solutions, QlikView 11 lets users create apps through a simple process that instantly delivers relevant analysis, reports, and dashboards critical to decision-making and operations. New functionality improves team development through integration with third-party source controls like Microsoft Team Foundation Server (TFS) and additional layout and analysis capabilities enhance application design.

With QlikView 11, users have point and click app building capabilities to embed UI elements on the fly – no keystrokes needed. A QlikView app can be even more quickly and easily enhanced and extended to meet changing business needs. New data sources and related listboxes, visualizations and images can be integrated with the app auto-calibrating to reflect the additions. QlikView 11 includes connection wizards and out-of-box connectors to SAP, Informatica, and Salesforce.com.

The business case for mobile BI is very strong

This is hardly a strange outcome since the business case for mobile BI is very strong. In addition, managers love gadgets like smartphones and tablets. The future of BI is mobile. We asked visitors: ”Business intelligence will become completely mobile in the near future”.

Possibly they believe that mobile BI will become a commodity

A small number of voters (17%) did not (completely) agree with the statement in the poll. Possibly they believe that mobile BI will become a commodity, but that a small part of BI will be consumed in traditional ways: viewing reports and dashboards on desktops and laptops. And I guess they are right, being able to do thorough analysis requires people to have devices like big screens, a mouse and the possibility to switch easily between windows.

Increase the usage of key information

Mobile BI will clearly increase the usage of key information by managers and employees. With BI apps on smartphones and tablets, users of these devices are only limited by their internet connections. Figures can be viewed from everywhere in almost any situation. From a BI study among 389 organizations it became clear that one of the biggest success factors for business intelligence is the use of information for analysis, process improvement, innovation and action. When managers consistently do the above and mobile BI may well support this, it is hardly surprising that the performance of the company will increase. Assuming they take the right decisions and actions!

Do BI vendors support mobile BI?

The question is whether all business intelligence vendors are ready for mobile BI. The answer is in our BI Tool Survey 2012 where we compared 16 tools for business intelligence on 103 selection criteria.

In our new poll we are asking if people are considering moving to BI in the cloud. On the left side of this webpage you can see the poll. Please let your voice be heard: vote.

Pentaho, a leading worldwide provider of software for BI and data integration software, today reaffirmed its commitment to support Big Data with a major expansion of native Big Data sources, including the latest Hadoop distributions, NoSQL sources, as well as native support for many analytic databases and traditional OLTP databases. Pentaho’s native connection to Big Data platforms makes it easier and faster than ever to analyze the enormous data volumes generated by today’s organizations.

Pentaho BI is changing the way that organizations analyze big data. Pentaho’s Big Data support enables:

• Fastest time to solution for Big Data analysis and data integration problems;
• Optimized performance and functionality through native integration;
• Dramatically reduced development costs compared to traditional tools for BI.

Big Data Platforms supported by Pentaho:

Hadoop Distributions- After announcing its initial support for Apache Hadoop and Hive in May 2010, Pentaho has made strides to expand and update its native support to the latest distributions and releases of Hadoop including: Apache Hadoop 0.20.x.x, Cloudera’s Distribution including Apache Hadoop 3.0 (CDH) and EMC Greenplum HD 1.0.

NoSQL Sources – The new release of Pentaho BI 4 includes production support for a variety of NoSQL sources that are designed to provide high-performance for Big Data workloads including: HBase, MongoDB, LexisNexis HPCC, elastic search, and XML streaming.

Analytic Databases - Pentaho BI offers unparalleled native OLAP analysis and data integration support for a variety of Big Data optimized analytic databases through native SQL generation for fast analytics or native bulk loader integration for fast data integration. Analytic databases with Pentaho native support include: EMC Greenplum, HP NonStop SQL/MX, HP Vertica, IBM Netezza, Infobright, Ingres Vectorwise, LucidDB, MonetDB and Teradata.

Traditional OLTP Databases - While traditional OLTP databases are typically not considered “Big Data” platforms, Pentaho maximizes their performance and scalability through native SQL dialect generation for fast businesz analytics, or native bulk loader integration for fast data integration.  OLTP databases with native Pentaho support include: Apache Derby, Firebird, HyperSQL, IBM DB2, IBM Informix, Ingres, Interbase, Microsoft Access, Microsoft SQL Server, MySQL, Oracle, and PostgreSQL.

Quotes and Multimedia

“We recognized early on the complexity and diversity of Big Data, and the growing need to support Big Data volumes by organizations of all sizes. As a result, Pentaho now offers deeper and more comprehensive support for Big Data sources than any other BI vendor.”

- Richard Daley, Co-Founder and CEO, Pentaho Corporation

“Although we evaluated many BI vendors for our next generation analytics service, what tipped the scale in favor of Pentaho was their support for big data analytics, especially Hadoop. Their offering was far superior to those of other venders in Big Data support and vision. For us, the two biggest selling points for us were support for ETL against Hadoop clusters and the ability to drill down from reports into data stored in Hadoop. Pentaho will dramatically reduce the development time necessary for us to bring our new analytic solution to market.

-Michael Ciancio-Bunch, Vice President, Chief Architect, ExactTarget

“Ideeli needs to analyze our massive web log data sets in the context of the rich data available in our relational transaction systems. Ideeli chose Pentaho for Hadoop because the built-in Hadoop, HBase and Map/ReduceElastic functionality in the Pentaho Data Integration application makes it possible to combine these sources to see the true picture of our business.”

-Paul Zanis, Director, Data Services, Ideeli, Inc.

“Hortonworks and Pentaho share a vision whereby Apache Hadoop becomes the de facto platform for storing, managing and analyzing big data. We are focused on accelerating the development and adoption of Apache Hadoop and are excited to be working with Pentaho to further simplify the development and deployment of Big Data projects.”

-Eric Baldeschwieler, CEO, Hortonworks.

“Infobright and Pentaho excel at helping companies quickly analyze one of the highest growth areas within Big Data: machine generated data such as Web logs, telecom call records, stock tick data and sensor data. To demonstrate the power of the combined products, the companies have partnered to offer an integrated open source solution that can be downloaded and implemented in minutes”

-Susan Davis, VP Marketing and Product Management, Infobright

“With MongoDB usage growing in the enterprise, there is a clear need for integration with business intelligence (BI) tools. The combination of Pentaho and MongoDB will give users the ability to rapidly analyze and extract value from their data stored in MongoDB. We’re excited to work with Pentaho to bring the full power of their suite to the MongoDB community.”

Max Schireson, President, 10gen the company that develops MongoDB

“As a system integrator, OpenBI uses Hadoop and Pentaho with clients like Exact Target to create analytical insights from previously inaccessible and untrackable data. When working with Pentaho for Hadoop we see a significant increase in productivity versus hand coding, and in many cases execution time and data passes are greatly reduced. The fact that Pentaho natively integrates with multiple data points reduces the complexity of the overall integration and gives us the flexibility to create a 100% custom integration without it being 100% custom.”

-Bryan Senseman, Partner and CTO, OpenBI

“The business opportunities emerging from big data analysis are increasing at a rapid pace as companies of all shapes and sizes discover ways to not only improve their business performance, but create new revenue streams from their data. Real-time, user-driven business intelligence is a key component in making big data initiatives true game changers. The combination of VectorWise and Pentaho gives users the speed, flexibility and agility that they require, pushing big data analytics even further into organizations.”

- Fred Gallagher, General Manager of VectorWise

“Vertica offers the only analytic platform developed in the 21^st century to support 21^st century big data challenges. The combined Pentaho and Vertica solution enables customers in every major vertical to explore, analyze, and visualize all of their data, at scale and in real-time.  The tight integration between Pentaho and Vertica delivers a decided advantage in today’s ultra-competitive business environment.”

-Scott Howser, VP of Product Marketing, Vertica

Now with the new release 10 Cognos has returned to its status as one of the leading products in the marketplace. SAS Enterprise BI Server and IBM Cognos score the most points in the 2012 BI Tool Survey.

New sections in the BI Survey

The 2012 survey now includes sections on Performance Management, Predictive Analysis (Data Mining) and technical performance issues like use of aggregates and caching.

Big data

These have been included to help take account of various trends in the market – specifically “Big Data”, the increasing use of management systems like Balanced Scorecard and the interest in trying to predict future performance instead of just reporting what has happened in the past.

Companies have different requirements

Creating successful BI systems is quit complex, because different types of companies have different requirements. In our view it would be a mistake to restrict the initial choice to one or two companies just because they have a large market share. Read more in the section ‘ranking by category’ or download the BI Tool Survey Report.

Tough times have always engendered sudden growth spurts of Open Source Software. The last time it occurred was soon after the Dot Com bust. This was the time when a lot of Open Source software like Linux and Java related ones got their sudden boost!

A similar thing could again happen to Open Source Business Intelligence! This is especially true of subscription based Open Source BI and those companies that were on the verge of finishing their basic IT projects automating functions that needed automation.

Once basic functional software like Enterprise Resource Planning (ERP) or CRM systems are in place automating daily functions, companies have the budgets and the face time for Business Intelligence solutions. With the Banking, Finance and Insurance Services (BFSI) sector in big trouble, with lots of uncertainties about the Global economy, many companies may dip their toes in the Open Source BI waters.

Of course, Open Source Software is not by any means free, since it always costs money for support and ongoing maintenance of the BI effort. With the recent downturn in Sun Microsystems performance and their announcements of layoffs, I wonder where their purchase of MySql open source product is! Would there be any cutbacks in their business model for MySQL support? What would this mean for those Open Source BI solutions based on MySQL

If anything, touch times are actually good for companies to experiment wih new technologies, especially if it is open source, and it does not cost too much to get something going!

Will be interesting to see if this is true and Open Source BI takes off during these touch times for the global economy!

We are all faced with a series of great opportunities brilliantly disguised as impossible situations. – Charles R. Swindoll.

Source:

“Customers should not have to choose between functionality and cost savings. Pentaho and Netezza are working together to give customers superior business intelligence and data warehousing capabilities at a lower TCO than other approaches on the market,” said Matt Rollender, director of strategic alliances for Netezza. “We anticipate that our combined offerings will set the standard for solutions that help customers effectively analyze and understand their data.”

Businesses of all sizes are increasingly looking for a scalable business intelligence strategy to help them improve decision making and increase business efficiency. Unfortunately, the upfront and recurring costs associated with some of the solutions from larger business intelligence providers translate into an unacceptably high total cost of ownership. The integrated solutions from Pentaho BI and Netezza break those cost barriers down with proven and scalable offerings that run at a fraction of the cost of other alternatives.

“There is a growing need, regardless of company size, for business intelligence and data warehousing capabilities that are both feature rich and affordable,” said Rajat Bhakhri, president and CEO of Aeturnum, Inc., an implementation partner for both Pentaho and Netezza. “By combining our best practices with the capabilities of this integrated solution, users can quickly discover high-value information within large volumes of data, while substantially lowering their total cost of ownership.”

Source:

By Jonathan Young, Attivio, Special to ZDNet

Commentary–It seems as though there is a watershed event in the search industry every ten years or so. Although Lexis-Nexis first commercialized search in the 1970s, it took a decade of indexing advances such as skip lists and index compression to make indexing practical, and another decade of computing advances to give us billions of searchable documents on the Internet.
Ten years ago, Google totally changed the face of what was then the emerging concept of Web search using better ranking algorithms based on website popularity. This precipitated the bifurcation of the search market into two segments: Web search and enterprise search. As the Web search space came to be dominated by Google, the old guard (e.g. Verity, Autonomy, AltaVista, and FAST) turned to enterprise search.

Google brought two changes to the industry. First, it raised the standard and importance of ease of use for the end user. By establishing the search box standard for the Web, it set the bar for enterprise search as well. Enterprise search in general does not have a strong track record: stories of multi-year deployments, cost overruns, and dissatisfied users are rife in the community.

Second, Google brought search to the forefront of information access as a strategy in the enterprise. The statement, “Why can’t I find information in my company as easy as I can find it on Google?” became all too common, echoing from the cubicles of knowledge workers. The average knowledge worker uses Google search more than their own internal knowledge applications. Not surprising, perhaps, there are signs that the enterprise search market is entering another watershed:

• Significant dissatisfaction with current solutions: according to AIIM, 85 percent of respondents in a recent survey said findability is significantly critical to their organization’s goals and success, yet 85 percent also said less than 50 percent of their enterprise information is searchable online.
• Increased innovation: especially in data transformation, analysis, and visualization.
• Unmet potential: current estimates state that more than 80 percent of corporate information assets reside in unstructured content sources, including contracts, white papers, research documents, emails, PDFs, and beyond, yet 75 percent of IT resources are dedicated to structured data.
• Increased mergers and acquisitions: business intelligence and data warehousing (BI/DW) vendors are purchasing search and analytics technologies (SAP/Pilot) while search vendors are buying analytical applications (Autonomy/Zantaz). Some of the largest acquisitions in the software industry have occurred in the last 15 months (SAP/Business Objects, IBM/Cognos, Microsoft/FAST).
• Industry convergence: analysts are heralding the convergence of enterprise search with business intelligence as the ultimate answer to unified information access.
• Legacy architectures: attempts to bend search engines to accept structured data and to force databases to understand unstructured content have both proven to be failed approaches. Most legacy enterprise search engines (such as the AltaVista toolkit) were built from the same code base as the web search engines, and until recently, the fundamental technology used in search engines had not changed for decades.

The basic approach is still based on matching words typed into the search box against words in documents in the index. Incremental improvements to document retrieval and ranking have slowly been added, including:

• TF/IDF ranking (better documents contain more of the rare words in a query)
• Boolean filtering (Boston +Celtic +Music -Celtics)
• Stop word elimination (common words don’t help search)
• Query expansion (using synonyms or related terms)
• Custom results ranking (e.g. by freshness, availability, or profit margin)

These techniques have generally resulted in small marginal improvements on the performance of information retrieval in the research lab, but whether or not they really improve the search experience for the end user can be questioned.

Changes in the search landscape

Competition in the enterprise search space has led to many novel, often wasteful, and sometimes confusing features. Most vendors now agree that scalability is a key requirement, but traditional search solutions require that the entire index be rebuilt each time additional resources are added to the configuration. This approach not only requires an infallible crystal ball, it ties up precious system resources. For true linear scalability, you must be able to add hardware as needed without impacting the running system, resulting in significantly lower total cost of ownership (TCO).

Also contributing to the historically high TCO of search is the cost of achieving good relevancy. It is commonplace to expect poor relevancy for an out-of-the-box installation of a legacy search engine. The cost of tuning the system is generally quite high, requiring significant adjustment of numerous configuration parameters. Obtaining a good relevance profile still remains largely a black art. Only recently has a new generation of search technologies enabled good out-of-the-box relevancy.

What about features that are touted to improve findability, such as “conceptual search” or “semantic search”? One version of semantic search involves using a large matrix (representing the frequency of terms in documents) to find latent semantic dimensions in the content, and then approximating documents by points in this alternative “semantic dimensional space”. The computational demands to this approach can be high; both when building the index and when matching documents to queries, and the results from searches can be unpredictable and even unrepeatable.

Other approaches extract “concepts” (key words and phrases) from documents using a variety of techniques, including lists of words and phrases, regular expressions, word bigrams (or trigrams), and advanced named entity extraction techniques created by Natural Language Processing (NLP) researchers. If the quality is high enough, extracted concepts can be used to enable corpus statistics, query enhancements, faceted browsing, and other forms of exploratory search. Unfortunately, the historical approach to faceted browsing has required that you pre-define your facets before indexing starts. If you want to change them, you likely will have to re-index the content again. This can take weeks.

Open source to the rescue?

What does the open source community bring to the party? The news is decidedly mixed. Several university labs make their research code available on liberal terms, but the systems do not have polished user interfaces. SourceForge has over 500 open-source “search engine” projects; most are incomplete. In simple terms, what Lucene and other open source solutions do, they do quite well, but they do little of what is actually needed to roll out a professional application. Lucene states clearly on its home page that it is merely a “library” and not a complete solution. In spite of some successes, building a search engine based on open-source components is still a task for experts.

One interesting recent development is openpipeline.org, which offers “open source software for crawling, parsing, analyzing and routing documents … for enterprise search and document processing”. While the effort is to be commended, the results are less than stellar: most of the text extraction modules are proprietary, and the pipelines are statically configured. While documents can be processed in parallel, there is no support for conditional execution of components or for cycles in the pipeline, which are essential for processing complex documents such as emails and archives in zip format. In formal terms, the architecture is not Turing complete.

This is an issue for most of the commercially available engines as well. They follow a linear pipeline that does not support branching, conditional logic, or parallel processing. Processing zip files and emails with attachments is problematic for such as system. A far better approach is a looping workflow that indexes the container first and then the contained items, while maintaining the child/parent relationship.

A search engine that only produces results when the user types a query into a search box is very limited in today’s modern knowledge enterprise. Newer systems allow additional forms of input, including drilling down into datasets based on filters created by clicking a faceted UI, “geo-search” near the user’s current location, and even personalized search based on the user’s role, access rights, and prior search history. Some enterprise search systems have begun to offer APIs which enable end users (or their IT departments) to build standalone alerting applications with some effort. Why don’t the workflows support this natively?

Source and more information: ZDnet.com

One theme of the release is ensuring the consistency and quality of data that resides throughout a company and is used for BI. The upgraded suite includes integration with IBM InfoSphere Business Glossary, which contains standard business terms defined by a company; another feature lets data modelers trace back the lineage of data using InfoSphere master-data management technology. They could see where data warehouse information used in a Cognos report came from, for example, and where that warehouse got the data from.

Source: Information Week