> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/h2oai/h2o-3/llms.txt
> Use this file to discover all available pages before exploring further.
# Hadoop & Spark
> Run H2O on YARN/Hadoop clusters and Apache Spark with Sparkling Water, including cloud integrations for AWS EMR, Azure HDInsight, and GCP Dataproc.
H2O can run as a YARN application on Hadoop clusters, giving it access to HDFS data and cluster resources. For Spark environments, Sparkling Water embeds H2O inside a Spark context.
## Prerequisites
* At least **6 GB of memory** allocated per H2O node
* Open communication on the H2O ports (default **54321** and **54322** TCP)
* The `h2odriver.jar` for your Hadoop distribution
Each H2O node runs as a single mapper. Run only one mapper per physical host — YARN may place two mappers on the same host if resources are tight, so open a range of at least 20 ports around 54321 to allow adaptive port selection.
***
## Running H2O on YARN/Hadoop
Download the H2O build for your Hadoop version from [h2o.ai/download](https://www.h2o.ai/download). Supported distributions include CDH 5.x, HDP 2.x, and MapR 4.x/5.x, as well as AWS EMR.
```bash theme={null}
unzip h2o--*.zip
cd h2o--*
```
```bash theme={null}
hadoop jar h2odriver.jar -nodes 3 -mapperXmx 6g -output hdfsOutputDirName
```
Key flags:
* `-nodes` — number of H2O nodes (mappers) to launch
* `-mapperXmx` — heap size per node (minimum `6g`; recommended: 4× your dataset size)
* `-output` — unique HDFS output directory name (must be unique per run)
When the cluster forms, you will see:
```
H2O cluster (3 nodes) is up
Open H2O Flow in your web browser: http://172.16.2.184:54321
```
```python Python theme={null}
import h2o
h2o.init(ip="172.16.2.184", port=54321)
```
```r R theme={null}
library(h2o)
h2o.init(ip = "172.16.2.184", port = 54321)
```
***
## Hadoop launch parameters reference
| Parameter | Description |
| ------------------------- | ------------------------------------------------ |
| `-nodes ` | Number of H2O nodes to launch |
| `-mapperXmx ` | Heap size per mapper (minimum `6g`) |
| `-extramempercent <0-20>` | Extra JVM memory as a percentage of `-mapperXmx` |
| `-nthreads ` | CPUs per node; `-1` uses all available |
| Parameter | Description |
| -------------------------------- | ------------------------------------------------------- |
| `-baseport ` | Starting port for H2O nodes (default `54321`) |
| `-driverport ` | Port for mapper-to-driver callback |
| `-driverportrange ` | Allowed range for driver callback port |
| `-network ` | Bind H2O to a specific IPv4 network, e.g. `10.1.2.0/24` |
| Parameter | Description |
| --------------------------------- | ---------------------------------------------------------------- |
| `-principal ` | Kerberos principal for authentication |
| `-keytab ` | Path to Kerberos keytab |
| `-run_as_user ` | Start cluster on behalf of a Hadoop user (non-Kerberos clusters) |
| `-jks ` | Java keystore for TLS |
| `-jks_pass ` | Keystore password (default `h2oh2o`) |
| Parameter | Description |
| -------------------- | -------------------------------------------------------- |
| `-output ` | Required. HDFS output directory (must be unique per run) |
| `-flow_dir ` | Directory for saving H2O Flow notebooks on HDFS |
| `-notify ` | Write the cluster's IP:port to a file when ready |
| `-disown` | Exit the driver after the cluster forms |
| `-timeout ` | Time to wait for cluster formation (default `120`) |
***
## Accessing S3 data from Hadoop
H2O launched on YARN can read from S3 as well as HDFS. Configure S3 access in Hadoop's `core-site.xml` and set `HADOOP_CONF_DIR` to point to that directory before launching H2O.
No changes to H2O's launch command are needed — S3 access flows through Hadoop's standard credential chain once `core-site.xml` is configured.
***
## Sparkling Water: H2O on Apache Spark
[Sparkling Water](https://github.com/h2oai/sparkling-water) runs an H2O cluster inside a Spark application. H2O and Spark share the same JVMs, enabling zero-copy data exchange between H2O frames and Spark DataFrames.
### Starting Sparkling Water
```bash theme={null}
$SPARK_HOME/bin/sparkling-shell \
--conf spark.executor.memory=4g \
--conf spark.driver.memory=2g
```
From the Spark shell:
```scala Scala theme={null}
import org.apache.spark.h2o._
val hc = H2OContext.getOrCreate()
println(hc.flowURL()) // Opens H2O Flow UI
```
### Data exchange between H2O and Spark
```scala Spark DataFrame → H2O Frame theme={null}
// Convert Spark DataFrame to H2O Frame
val sparkDF = spark.read.parquet("hdfs://data/train.parquet")
val h2oFrame = hc.asH2OFrame(sparkDF)
```
```scala H2O Frame → Spark DataFrame theme={null}
// Convert H2O Frame back to Spark DataFrame for downstream use
val predictions = model.score(h2oFrame)
val sparkPredictions = hc.asDataFrame(predictions)
```
Train with H2O for its algorithm performance (especially GBM and AutoML), then convert predictions back to Spark DataFrames to integrate with your existing Spark pipelines.
### TLS with Sparkling Water
```bash theme={null}
$SPARK_HOME/bin/spark-submit \
--class water.SparklingWaterDriver \
--conf spark.ext.h2o.jks=/path/to/h2o.jks \
--conf spark.ext.h2o.jks.pass=mypassword \
sparkling-water-assembly-all.jar
```
***
## Cloud integrations
### AWS EMR
H2O provides a pre-built distribution for Amazon EMR. Download the EMR-specific build from the H2O download page, then launch with the standard `hadoop jar h2odriver.jar` command from the EMR master node.
For S3-backed data, assign an IAM role with S3 read permissions to the EMR cluster — no additional configuration is needed.
### Azure HDInsight
Use the HDInsight-compatible H2O distribution. Launch from the HDInsight head node with the `hadoop jar` command. Azure Blob Storage (WASB) can be accessed the same way as S3 once the cluster's `core-site.xml` is configured with the WASB connector.
### GCP Dataproc
H2O runs on Dataproc as a YARN application. Use a Dataproc initialization action script to install H2O on all nodes, then submit the `hadoop jar h2odriver.jar` command from the master.
```bash Dataproc init action theme={null}
#!/bin/bash
HADOOP_VERSION=$(hadoop version | head -1 | awk '{print $2}')
wget -q "https://h2o-release.s3.amazonaws.com/h2o/latest_stable_Hadoop.zip" -O /tmp/h2o-hadoop.zip
unzip -q /tmp/h2o-hadoop.zip -d /opt/h2o-hadoop
```
GCS data access uses the Cloud Storage connector bundled with Dataproc — no extra H2O configuration needed beyond standard Hadoop GCS setup.