Cornerstone Big Data Analysis through Apache Spark

The systematic analysis of ever-increasing data collection presents companies with ever-greater challenges. Many companies simply lack the know-how to handle big data projects. Following to the motto “Let’s do a Data Lake first”, they bring together all available data in one system. Because often they are subject to the misconception that you should put as much data as possible in the system in order to gain the maximum insight and the most flexible evaluations. Unfortunately this does not work, because we can expect performance problems here. Therefore, the company must also think about meaningful evaluations for big data analytics in advance, that offer added value considering the cost-benefit ratio. An upstream potential analysis is recommended and can provide insight and foresight here.

Within the research and development project PRO-OPT, which is funded within the framework of the technology program “Smart Data –  Innovationen aus Daten” of the Federal Ministry for Economic Affairs and Energy of Germany, camLine has taken on this problem. The aim of the PRO-OPT project is to enable companies in decentralized cooperative structures (smart ecosystems) to effectively and intelligently analyze large amounts of data. Digitalization and automation in particular are generating ever larger amounts of data in production. The data sources are distributed at different locations of a participant or at different, economically independent participants of the ecosystem, especially because not only parts from own manufacturing are used in the products. Overarching analysis must be broken down, taking into account access permissions to these sources. Big Data strategies should help to make these analyses more efficient.