Apache Spark RDD Operation 편집하기

'''Apache Spark RDD Operation''' refers to the various transformations and actions that can be applied to [[Resilient Distributed Datasets (RDDs)]] in [[Apache Spark]]. RDD operations enable efficient parallel processing of large datasets in a distributed environment.
==Types of RDD Operations==
RDD operations are classified into two types:
*'''Transformations:''' Lazy operations that return a new RDD without immediate execution.
*'''Actions:''' Operations that trigger computation and return results or store data.
==Transformations==
Transformations are applied to RDDs to produce new RDDs. They are lazy, meaning they are not executed until an action is performed.
===Common Transformations===
{| class="wikitable"
!Transformation!!Description!!Example
|-
|'''map(func)'''||Applies a function to each element in the RDD.||Converting temperatures from Celsius to Fahrenheit.
|-
|'''filter(func)'''||Retains elements that satisfy a condition.||Filtering even numbers from an RDD.
|-
|'''flatMap(func)'''||Similar to map but allows multiple outputs per input.||Tokenizing sentences into words.
|-
|'''union(rdd)'''||Merges two RDDs.||Combining two datasets.
|-
|'''distinct()'''||Removes duplicate elements.||Getting unique elements from an RDD.
|-
|'''groupByKey()'''||Groups data by key (for (K,V) pairs).||Grouping sales data by region.
|-
|'''reduceByKey(func)'''||Merges values for each key using a function.||Summing sales per region.
|-
|'''sortByKey()'''||Sorts RDD by key.||Sorting word count results.
|}
===Example Transformation===
<syntaxhighlight lang="python">
rdd = sparkContext.parallelize([1, 2, 3, 4, 5])
squared_rdd = rdd.map(lambda x: x * x)  # [1, 4, 9, 16, 25]
</syntaxhighlight>
==Actions==
Actions compute and return results or store RDD data. They trigger the execution of all previous transformations.
===Common Actions===
{| class="wikitable"
!Action!!Description!!Example
|-
|'''collect()'''||Returns all elements of the RDD to the driver.||Retrieving data for small datasets.
|-
|'''count()'''||Returns the number of elements.||Counting lines in a file.
|-
|'''reduce(func)'''||Aggregates elements using a binary function.||Summing all numbers in an RDD.
|-
|'''first()'''||Returns the first element.||Retrieving a sample record.
|-
|'''take(n)'''||Returns the first n elements.||Getting a preview of data.
|-
|'''foreach(func)'''||Applies a function to each element.||Writing records to an external database.
|}
===Example Action===
<syntaxhighlight lang="python">
rdd = sparkContext.parallelize([1, 2, 3, 4, 5])
sum_result = rdd.reduce(lambda x, y: x + y)  # Output: 15
</syntaxhighlight>
==Lazy Evaluation in RDDs==
RDD transformations are '''lazy''', meaning they do not execute immediately. Instead, Spark builds a '''DAG (Directed Acyclic Graph)''' representing operations, which is only executed when an action is called.

Example:<syntaxhighlight lang="python">
rdd = sparkContext.textFile("data.txt")  # No execution yet
words = rdd.flatMap(lambda line: line.split())  # Still not executed
word_count = words.count()  # Now execution starts
</syntaxhighlight>
==Persistence and Caching==
RDDs can be cached in memory to speed up iterative computations:
*'''cache()''' – Stores the RDD in memory.
*'''persist(storage_level)''' – Stores the RDD using different storage levels (e.g., memory, disk).
Example:<syntaxhighlight lang="python">
rdd = sparkContext.textFile("data.txt").cache()
print(rdd.count())  # RDD is now cached in memory
</syntaxhighlight>
==Comparison with DataFrames==
{| class="wikitable"
!Feature!!RDD!!DataFrame
|-
|Abstraction Level||Low (Resilient Distributed Dataset)||High (Table-like structure)
|-
|Performance||Slower (No optimizations)||Faster (Uses Catalyst Optimizer)
|-
|Storage Format||Unstructured||Schema-based
|-
|Ease of Use||Requires functional transformations||SQL-like API
|}
==Advantages of RDDs==
*'''Fault Tolerant:''' Uses lineage to recompute lost partitions.
*'''Parallel Execution:''' Automatically distributes computations across nodes.
*'''Immutable and Lazy Evaluation:''' Optimizes execution by avoiding unnecessary computations.
==Limitations of RDDs==
*'''Higher Memory Usage:''' No schema-based optimizations.
*'''Verbose API:''' Requires functional programming.
*'''Less Optimized than DataFrames:''' Lacks query optimizations found in Spark DataFrames.
==Applications==
*Processing large-scale unstructured data.
*Complex transformations requiring fine-grained control.
*Iterative machine learning computations.
==See Also==
*[[Apache Spark]]
*[[Resilient Distributed Datasets (RDDs)]]
*[[Spark DataFrame]]
*[[Big Data Processing]]
*[[Parallel Computing]]