What is Apache Spark
What is Apache Spark ?
Apache Spark is an open-source distributed general-purpose cluster-computing framework. It provides an interface for programming the entire clusters with implicit data parallelism and fault tolerance. Apache Spark is an unified analytics engine for the big data processing, with built-in modules for streaming, SQL, machine learning and graph processing.
Spark is a fast and general engine for large-scale data processing. It is an open-source, distributed processing system which is used for big data workloads. Spark utilizes in-memory caching and optimized query execution for fast queries against data of any size.
Apache Spark is not a language or a DB, but it is a framework. It supports Java, Python, Scala, and R programming languages making it well suited for a plethora of use cases. It is incorporated by the Developers and Data Scientists into applications for rapid query, analysis, and to transform data.
Apache Spark has its architectural foundation in the Resilient Distributed Dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.
Spark facilitates the implementation of both the iterative algorithms, which visit their data set multiple times in a loop, and interactive or exploratory data analysis, i.e., the repeated database-style querying of data.
The latency of such applications will be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation. Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.
Spark is designed in a way that it integrates with all the Big data tools. For example, Spark can access any Hadoop data source and can run on Hadoop clusters. It extends Hadoop MapReduce to the next level which includes the iterative queries and stream processing.
Many believe that Apache Spark is an extension of Hadoop, but it is not true. Apache Spark is an independent of Hadoop because it has its own cluster management system. It uses Hadoop for storage purpose only.
Apache Spark enables the Hadoop clusters applications to run faster in memory while running on the disk. It lets you write the applications in Java, Scala, or Python much faster.
