What is Big Data Analytics


What is big data - Big data analytics, software, tools + trends

Big data is currently regarded as the IT trend par excellence in the IT industry and is therefore discussed controversially.

The almost inflationary use of the term big data goes hand in hand with the risk that it will degenerate into a fashion or plastic word without clear outlines. In addition, the IT industry is a burnt child when it comes to fashion topics: Green IT, SOA, EAI, dot-com bubble, ... It is therefore not surprising that the word hype is used again and again in the discussion about big data.

Is big data really just one of those fast-moving buzzwords, initially pushed by marketing strategists, but then just as quickly dropped again until it more or less disappears into oblivion?

Relevant questions about big data, which are discussed in more detail in the following text:

  1. What is big data
  2. What is Big Data Analytics?
  3. What challenges does the selection of big data software pose?
  4. Which big data solutions do software manufacturers offer?
  5. Which big data factors do companies have to consider?
  6. Summary: What do companies have to consider when choosing big data software?

What is big data

So far, there has been no generally applicable, clear definition of big data. In general, Big Data can be defined as any dataset that exceeds the limits and possibilities of conventional IT. Big data is about anything that no longer works with conventional technology due to the size of the data, i. H. to collect, store, search, distribute, analyze and visualize large amounts of data. Standard databases and tools are increasingly having problems coping with the increasing flood of data: Relative databases fail because of their volume, ETL processes are too slow and have difficulties with the various data formats, traditional BI is therefore too slow and can handle the No longer effectively processing masses of unstructured data.

The emergence of big data

The background to the discussions about big data is the sharp increase in the global volume of data. A large number of different sources are responsible for this: sensor data, machine data, log data, WordWideWeb or RFID chips. In 2011, the global data volume cracked the zettabyte barrier (1 with 21 zeros) and there is no end to growth in sight. By 2020 it should be 35 zettabytes.

But it's not just the gigantic amounts of data that make up the big data problem. The lack of structure - there is talk of poly-structured data - and the different formats are extremely problematic for conventional business software. Conventional BI software is based on a data warehouse, in the core of which clearly structured and standardized data must be stored. This requires complex extraction, loading and transaction processes (ETL) in advance. This is the only way to further process the data in a profitable way. Under the conditions of an increasing data volume and a lack of structuring, the data can no longer be efficiently mapped in the relational databases of the data warehouses.

Amount of dataAnalyticsspeedData diversity
Processing of large volumes of dataCreation of modelsretrieve data fasterStructuring the data
Processing of different data setsData miningselect data fasterManagement of different data types


What is Big Data Analytics?

Big data is particularly relevant for the area of ​​Business Intelligence (BI), which deals with the analysis of data (acquisition, evaluation, presentation). Big data analytics describes the systematic evaluation / analysis of large amounts of data with the help of newly developed software. In contrast to conventional software solutions, big data software includes special functions and techniques that enable a large amount of data to be processed in parallel.

  • Processing of many records
  • fast import of data
  • quick search and query of data
  • simultaneous processing of several queries
  • Analysis of different types of information

Big Data Analysis represents one of the hottest trends in the business intelligence software industry.

At this point, many companies ask themselves what exactly is the difference between big data analytics and big data analysis? You can find helpful explanations on this topic in our question area: What is the difference between Big Data Analytics and Big Data Analysis?

Here you will find recommendations from SAP for the effective processing of big data: Evaluate big data quickly and easily

Big data software

Big data software can form the basis for big data analysis. A software program can execute the applications listed under Big Data Analytics.

What challenges does the selection of big data software pose?

The company's situation: increased requirements, growing challenges

As already mentioned, the phenomenon of growing data volumes and the multiplication of data sources is not entirely new. The really new thing about big data seems to come from the corporate environment. It is the increased requirements on the part of companies that give big data a new dimension. BI software has gained increasing strategic importance in companies in recent years. Accordingly, the number of users has continued to rise, as have the expectations of the topicality and short-term availability of the data as well as the query performance of the system with a simultaneous need for more complex analysis.

The increased requirements reflect the increased challenges in the business world. In view of the increasingly fierce global economic competition, the following applies more than ever: time is money. Those companies that react fastest to current market developments and can align their internal process landscape to market requirements create a decisive competitive advantage for themselves. In addition to the important factor of time, it is essential for companies to be able to easily understand the increasingly complex structures and their interrelationships in the company. Effective countermeasures can only be initiated if you know exactly where and what is going wrong in your own company.

In addition, an awareness of the strategic value of data has now established itself in large parts of the corporate world. This awareness is reflected in the fact that medium-sized companies are now using BI software almost as standard. This emerges from the current study by the consulting and market analyst company SoftSelect on the subject of business intelligence. Those who manage to successfully analyze the enormous data material with regard to initially hidden patterns and relationships are often one step ahead of their competitors. In order to take into account the time and complexity aspects of day-to-day business, high-performance processing of the huge mountains of data is required.

Which big data solutions do software manufacturers offer?

The software providers: a wide range of solutions

This new constellation has of course not gone unnoticed by the software manufacturers. Analogous to the classic BI architecture, new methods and technologies for capturing, storing, processing, analyzing and displaying large, poly-structured amounts of data have long been available on the market. However, the software offered is just as diverse as the problems raised by big data. There is a multitude of providers on the market who provide a wealth of solutions for all of the areas mentioned. It is often very difficult for companies to see through this confusing market.

In the area of ​​data integration, the main problem lies in the speed and manageability of the poly-structured data. Software providers are currently trying to combine big data functions with established data integration tools such as Informatica, Pentaho or Pervasive, and there are also specialists for integrating polystructured data sources such as Hadoop, Chukwa, Flume or Sqoop.

Special file systems such as HDFS from Hadoop, but also so-called NoSQL (not-only SQL) databases, are ideal for storing and high-performance processing of big data. It is important here that these techniques are harmonized with the classic analytical databases, which continue to take on important functions. This is the only way to maintain the consistency of the data and to carry out typical relational operations without problems.

The MapReduce approach developed by Google is at the center of the fast processing of big data. This is based on the following mechanism: A task is broken down into the smallest possible parts, then distributed to as many computers as possible for parallel processing and then merged again as a result. A high level of parallel processing of poly-structured data is thus possible. Another tool that enables big data to be processed in seconds is in-memory computing, such as SAP HANA. The main memory of a computer is used as data storage. In contrast to data that is stored on a hard drive, this enables a much higher access speed to the data. There are also solutions that rely on analytical databases. These are mostly column-oriented databases that break with the common concept of classic row-oriented databases. They filter out unneeded areas and thus enable flexible and, above all, quick access. With all these technologies, huge amounts of data can be processed at such a speed that one can quite accurately speak of real-time analysis.

In the area of ​​the analysis of polystructured data, the modeling based on detailed data can be observed. Especially the open source provider R, but also other data mining tools from EMC, SAS, or SPSS have established themselves on the market. In addition, there are tools that, due to their ability to process large amounts of data, can cover completely new areas of application such as text mining or location intelligence.

Big Data Analytics Tools and Trends 2015

In 2015, new trends in the use of big data are emerging. The most important are:

  • Big data management via the cloud
  • Improved data integration via ETL (extraction, transformation, loading)
  • Optimization of SQL databases
  • Optimizing data storage

Here is some helpful information from Tableau on Big Data usage and visualization: Big Data - Optimization

Which big data factors do companies have to consider?

In summary, the current phenomenon of big data can be characterized as a combination of the following factors:

  • increased data volume
  • growing number of data sources
  • Polystructuring of the data
  • very different data formats
  • more and more users of BI software
  • higher expectations of the analysis of complex relationships and query performance of the system
  • Evaluations almost in real time

The combination of all these factors is increasingly overwhelming classic databases and analysis tools, which is why it is to be expected that the need for new, more powerful software solutions in companies will gradually increase.

Since the phenomenon of big data is linked to very real business challenges, it is rather unlikely that big data is just a matter of hype. Even if the market for big data software is still at an early stage of development and companies are still exploring worthwhile areas of application, IT providers are already offering promising solutions for the problems described.

Big data topics

Summary: What do companies have to consider when choosing big data software?

Even if big data is not just a trendy topic, companies have to look very carefully if big data is to help companies succeed:

  1. First of all, your own IT infrastructure must be checked carefully:

    • How do you best handle your data? Which data really need to be kept, for which data is short or medium-term storage sufficient? What storage options are there? How much computing power must actually be available? What software do you need? Where do traditional databases, hardware components, applications, etc. need support from big data technologies? How can these be meaningfully expanded with big data software?
  2. In addition, a very precise analysis should take place where exactly big data solutions offer actual business added value:

    • In which areas does Big Data really make sense and where are “classic” solutions completely sufficient? What specific application scenarios are there for big data technologies? A thorough evaluation must take place in advance, especially with regard to the sometimes very cost-intensive acquisition costs of big data solutions.
  3. Furthermore, specially trained data scientists are required in the company who are able to make productive use of the results provided by the big data tools.

    • It's not just about analyzing as much data as possible as quickly as possible, but about what the numbers actually mean and which decision has to be made from the company's perspective. Future developments, for example, can only be forecast with the greatest possible degree of certainty through the interaction of high-performance IT and well-trained specialists.

As a result, it is advisable to deal with the topic of big data very carefully if big data is to actually develop into an important factor in business success.

If you are looking for software for your industry, you will find our extensive software selection here:

Software selection


Author: Michael Gottwald / SoftSelect GmbH


Here you will find further knowledge of our experts on the subject of what is big data? - Big data analytics, software, tools + trends

| Big Data |