pcf.particle.aws.emr package

Submodules

pcf.particle.aws.emr.emr_cluster module

class pcf.particle.aws.emr.emr_cluster.EMRCluster(particle_definition, session=None)

Bases: pcf.core.aws_resource.AWSResource

This is the implementation of Amazon’s ECS Cluster.

FILTERED_UPDATE_PARAMS = {'InstanceCount', 'TargetOnDemandCapacity', 'TargetSpotCapacity'}
UNIQUE_KEYS = ['aws_resource.Name']
UPDATE_PARAM_CONVERSIONS = {'RequestedInstanceCount': 'InstanceCount', 'TargetOnDemandCapacity': 'TargetOnDemandCapacity', 'TargetSpotCapacity': 'TargetSpotCapacity'}
_get_cluster_id()

Get’s the cluster Id of the EMR cluster based on cluster name

Returns:ClusterId
_set_unique_keys()

Logic that sets keys from state definition that are used to uniquely identify the EMR Cluster

_start()

Calls boto3 run_job_flow()

Returns:boto3 run_job_flow() response
_stop()

Calls _terminate()

_terminate()

Calls boto3 terminate_job_flows()

Returns:boto3 terminate_job_flows() response
_update()

Logic that updates the particle. This is implemented for each individual particle

equivalent_states = {<State.running: 1>: 1, <State.stopped: 2>: 0, <State.terminated: 3>: 0}
flavor = 'emr_cluster'
get_status()

Calls boto3 describe_clusters using arn.

Returns:“missing”}
Return type:status or {“status”
is_state_definition_equivalent()

Determines if current state is equivalent to the desired state.

Returns:bool
is_state_equivalent(state1, state2)
Parameters:
Returns:

bool
state_lookup = {'bootstrapping': <State.pending: 4>, 'missing': <State.terminated: 3>, 'running': <State.running: 1>, 'starting': <State.pending: 4>, 'terminated': <State.terminated: 3>, 'terminated_with_errors': <State.terminated: 3>, 'terminating': <State.pending: 4>, 'waiting': <State.running: 1>}
sync_state()

Calls get_status() and updates the current_state_definition and the state.

Module contents