Apache Spark RDD Operation
Apache Spark RDD Operation refers to the various transformations and actions that can be applied to Resilient Distributed Datasets (RDDs) in Apache Spark. RDD operations enable efficient parallel processing of large datasets in a distributed environment.
1 Types of RDD Operations[편집 | 원본 편집]
RDD operations are classified into two types:
- Transformations: Lazy operations that return a new RDD without immediate execution.
- Actions: Operations that trigger computation and return results or store data.
2 Transformations[편집 | 원본 편집]
Transformations are applied to RDDs to produce new RDDs. They are lazy, meaning they are not executed until an action is performed.
2.1 Common Transformations[편집 | 원본 편집]
| Transformation | Description | Example |
|---|---|---|
| map(func) | Applies a function to each element in the RDD. | Converting temperatures from Celsius to Fahrenheit. |
| filter(func) | Retains elements that satisfy a condition. | Filtering even numbers from an RDD. |
| flatMap(func) | Similar to map but allows multiple outputs per input. | Tokenizing sentences into words. |
| union(rdd) | Merges two RDDs. | Combining two datasets. |
| distinct() | Removes duplicate elements. | Getting unique elements from an RDD. |
| groupByKey() | Groups data by key (for (K,V) pairs). | Grouping sales data by region. |
| reduceByKey(func) | Merges values for each key using a function. | Summing sales per region. |
| sortByKey() | Sorts RDD by key. | Sorting word count results. |
2.2 Example Transformation[편집 | 원본 편집]
rdd = sparkContext.parallelize([1, 2, 3, 4, 5])
squared_rdd = rdd.map(lambda x: x * x) # [1, 4, 9, 16, 25]
3 Actions[편집 | 원본 편집]
Actions compute and return results or store RDD data. They trigger the execution of all previous transformations.
3.1 Common Actions[편집 | 원본 편집]
| Action | Description | Example |
|---|---|---|
| collect() | Returns all elements of the RDD to the driver. | Retrieving data for small datasets. |
| count() | Returns the number of elements. | Counting lines in a file. |
| reduce(func) | Aggregates elements using a binary function. | Summing all numbers in an RDD. |
| first() | Returns the first element. | Retrieving a sample record. |
| take(n) | Returns the first n elements. | Getting a preview of data. |
| foreach(func) | Applies a function to each element. | Writing records to an external database. |
3.2 Example Action[편집 | 원본 편집]
rdd = sparkContext.parallelize([1, 2, 3, 4, 5])
sum_result = rdd.reduce(lambda x, y: x + y) # Output: 15
4 Lazy Evaluation in RDDs[편집 | 원본 편집]
RDD transformations are lazy, meaning they do not execute immediately. Instead, Spark builds a DAG (Directed Acyclic Graph) representing operations, which is only executed when an action is called.
Example:
rdd = sparkContext.textFile("data.txt") # No execution yet
words = rdd.flatMap(lambda line: line.split()) # Still not executed
word_count = words.count() # Now execution starts
5 Persistence and Caching[편집 | 원본 편집]
RDDs can be cached in memory to speed up iterative computations:
- cache() – Stores the RDD in memory.
- persist(storage_level) – Stores the RDD using different storage levels (e.g., memory, disk).
Example:
rdd = sparkContext.textFile("data.txt").cache()
print(rdd.count()) # RDD is now cached in memory
6 Comparison with DataFrames[편집 | 원본 편집]
| Feature | RDD | DataFrame |
|---|---|---|
| Abstraction Level | Low (Resilient Distributed Dataset) | High (Table-like structure) |
| Performance | Slower (No optimizations) | Faster (Uses Catalyst Optimizer) |
| Storage Format | Unstructured | Schema-based |
| Ease of Use | Requires functional transformations | SQL-like API |
7 Advantages of RDDs[편집 | 원본 편집]
- Fault Tolerant: Uses lineage to recompute lost partitions.
- Parallel Execution: Automatically distributes computations across nodes.
- Immutable and Lazy Evaluation: Optimizes execution by avoiding unnecessary computations.
8 Limitations of RDDs[편집 | 원본 편집]
- Higher Memory Usage: No schema-based optimizations.
- Verbose API: Requires functional programming.
- Less Optimized than DataFrames: Lacks query optimizations found in Spark DataFrames.
9 Applications[편집 | 원본 편집]
- Processing large-scale unstructured data.
- Complex transformations requiring fine-grained control.
- Iterative machine learning computations.
