Elastic MapReduce (EMR)

Introduction

Amazon Elastic MapReduce (EMR) is a fully managed big data processing service that allows developers to effortlessly create, deploy, and manage big data applications. EMR supports various big data processing frameworks, including Hadoop MapReduce, Apache Spark, Apache Hive, and Apache Pig. Developers can leverage these frameworks and their rich ecosystem of tools and libraries to perform complex data transformations, machine learning tasks, and real-time data processing.

LocalStack supports EMR and allows developers to run data analytics workloads locally. EMR utilizes various tools in the Hadoop and Spark ecosystem, and your EMR instance is automatically configured to connect seamlessly to LocalStack’s S3 API. LocalStack also supports EMR Serverless to create applications and job runs, to run your Spark/PySpark jobs locally.

The supported APIs are available on the API coverage section for EMR and EMR Serverless, which provides information on the extent of EMR’s integration with LocalStack.

Note

To utilize the EMR API, certain additional dependencies need to be downloaded from the network (including Hadoop, Hive, Spark, etc). These dependencies are fetched automatically during service startup, hence it is important to ensure a reliable internet connection when retrieving the dependencies for the first time. Alternatively, you can use one of our *-bigdata Docker image tags which already ship with the required libraries baked in and may provide better stability (see here for more details).

Getting started

This guide is designed for users new to EMR and assumes basic knowledge of the AWS CLI and our lstk aws command.

Start your LocalStack container using your preferred method. We will create a virtual EMR cluster using the AWS CLI. To create an EMR cluster, run the following command:

lstk aws emr create-cluster \
          --release-label emr-5.9.0 \
          --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large

You will see a response similar to the following:

Output

{
    "ClusterId": "j-A2KF3EKLAOWRI"
}

You can also specify startup commands using the --steps=... command line argument to the CreateCluster API.

Examples

The following code snippets and sample applications provide practical examples of how to use EMR in LocalStack for various use cases:

• Running data analytics jobs using EMR
• Running EMR Serverless Jobs with Java

API Coverage

33 of 65 operations implemented

Available from the Ultimate plan. Licensing details

Formats for tools and AIPage MarkdownCoverage MarkdownCoverage JSONCoverage catalog

Find an API

Search the full operation list, then sort the table to compare current support.

Loading operations…

Search by operation name

Operation ▲	Implemented ▼	Verified on Kubernetes
Loading operations…

PreviousNext

Complete static API list All 65 operations and their current support status

Operation	Status
AddInstanceFleet	Implemented
AddInstanceGroups	Implemented
AddJobFlowSteps	Implemented
AddTags	Implemented
CancelSteps	Not implemented
CreatePersistentAppUI	Not implemented
CreateSecurityConfiguration	Implemented
CreateStudio	Not implemented
CreateStudioSessionMapping	Not implemented
DeleteSecurityConfiguration	Implemented
DeleteStudio	Not implemented
DeleteStudioSessionMapping	Not implemented
DescribeCluster	Implemented
DescribeJobFlows	Implemented
DescribeNotebookExecution	Not implemented
DescribePersistentAppUI	Not implemented
DescribeReleaseLabel	Not implemented
DescribeSecurityConfiguration	Implemented
DescribeStep	Implemented
DescribeStudio	Not implemented
GetAutoTerminationPolicy	Implemented
GetBlockPublicAccessConfiguration	Implemented
GetClusterSessionCredentials	Not implemented
GetManagedScalingPolicy	Not implemented
GetOnClusterAppUIPresignedURL	Not implemented
GetPersistentAppUIPresignedURL	Not implemented
GetSession	Not implemented
GetSessionEndpoint	Not implemented
GetStudioSessionMapping	Not implemented
ListBootstrapActions	Implemented
ListClusters	Implemented
ListInstanceFleets	Implemented
ListInstanceGroups	Implemented
ListInstances	Implemented
ListNotebookExecutions	Not implemented
ListReleaseLabels	Implemented
ListSecurityConfigurations	Not implemented
ListSessions	Not implemented
ListSteps	Implemented
ListStudioSessionMappings	Not implemented
ListStudios	Not implemented
ListSupportedInstanceTypes	Implemented
ModifyCluster	Implemented
ModifyInstanceFleet	Implemented
ModifyInstanceGroups	Implemented
PutAutoScalingPolicy	Implemented
PutAutoTerminationPolicy	Implemented
PutBlockPublicAccessConfiguration	Implemented
PutManagedScalingPolicy	Not implemented
RemoveAutoScalingPolicy	Implemented
RemoveAutoTerminationPolicy	Implemented
RemoveManagedScalingPolicy	Not implemented
RemoveTags	Implemented
RunJobFlow	Implemented
SetKeepJobFlowAliveWhenNoSteps	Not implemented
SetTerminationProtection	Implemented
SetUnhealthyNodeReplacement	Not implemented
SetVisibleToAllUsers	Implemented
StartNotebookExecution	Not implemented
StartSession	Not implemented
StopNotebookExecution	Not implemented
TerminateJobFlows	Implemented
TerminateSession	Not implemented
UpdateStudio	Not implemented
UpdateStudioSessionMapping	Not implemented

API Coverage (EMR Serverless)

14 of 22 operations implemented

Available from the Ultimate plan. Licensing details

Formats for tools and AIPage MarkdownCoverage MarkdownCoverage JSONCoverage catalog

Find an API

Search the full operation list, then sort the table to compare current support.

Loading operations…

Search by operation name

Operation ▲	Implemented ▼	Verified on Kubernetes
Loading operations…

PreviousNext

Complete static API list All 22 operations and their current support status

Operation	Status
CancelJobRun	Implemented
CreateApplication	Implemented
DeleteApplication	Implemented
GetApplication	Implemented
GetDashboardForJobRun	Not implemented
GetJobRun	Implemented
GetResourceDashboard	Not implemented
GetSession	Not implemented
GetSessionEndpoint	Not implemented
ListApplications	Implemented
ListJobRunAttempts	Not implemented
ListJobRuns	Implemented
ListSessions	Not implemented
ListTagsForResource	Implemented
StartApplication	Implemented
StartJobRun	Implemented
StartSession	Not implemented
StopApplication	Implemented
TagResource	Implemented
TerminateSession	Not implemented
UntagResource	Implemented
UpdateApplication	Implemented