What is RDD? Apache spark Resilient distributed datasets ( RDD ) Explained , Demystified

What is RDD?

Resilient distributed datasets(RDD) - Is a collection of elements, partitioned across the nodes of a cluster and can be operated in parallel.

RDD Life cycle:

How to create an RDD?

1) Starting from a file in any of the Hadoop distributed FS, shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat.

2) From an existing Scala collection in the driver program, and transforming it by calling the paralyze method on the sparkcontext

What are the features of an RDD?

1. Users can also ask Spark to persist the RDD in memory, allows efficient re-use across parallel operations.
2. RDD will automatically recover from node failures.

What are the operations supported by RDD?

supports two types of Operations:

1. Transformations - creates new dataset from existing RDD
2. Actions - Runs a function on the RDD, aggregates and returns a result

Roughly above operations can be linked to Map and reduce, but exception being reduceByKey  which creates a new RDD

All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program. 

Apache spark program life cycle

Spark Application —> driver program —> main function —> parallel operations on cluster -> result

Orchestrating Hadoop Workflows - What your workflow tool should look like.

This post illustrates what are the most important qualities to look for while deciding a Big data workflow tool.

1. Schedule the ETL process to ingest incremental data from various streams.
   
   
   Ability to schedule is the most elementary requirement of any orchestration tool, and the tool
   must be abel to ingest and process data simultaneously form various sources to various endpoints. It should support most common sources and endpoints out of the box.
   
   
2. Notifications on failure of ETL process.
   
   
   Notifications and Alerts are a must, a failure can setback you to backfill and can drastically increase the infrastructure costs (especially if u pay by the hour for the cluster). So a good reliable notification mechanism is essential and a must have.
   
   
3. Optimization on resources usage for daily ingestion.
   
   
   Not directly supported by the workflow tool, but the tool should persist the execution time and trends for a task, which can be used to optimize your tasks. This data becomes important when you have run the workflow sufficiently and can understand the time trends.
   
   
4. Retry on failures, make ur tasks as Idempotent incremental loads
   
   
   The ownership of making the workflow idempotent is on the developer and not the tool. The tool should support retires.
   
   
5. Zero/Minimal Manual intervention
   
   
6. Self recovering/ rescheduling system
   
   
7. Pause the workflow
   
   
   The tool should support you , when needed to pause the execution after the current running task. This enables you to fix any issue with the next task in your workflow without having to fail the workflow.
   
   
8. Automatic Data audits, for incremental loaded data.
   
   
   Should support audits on tasks and help make decision in the workflow.