Class cascading.DistanceUseCasePlatformTest

4

tests

0

failures

0

ignored

8.979s

duration

100%

successful

Tests

Test Duration Result
testEuclideanDistance 1.230s passed
testEuclideanDistanceComposite 5.137s passed
testEuclideanDistanceShort 1.099s passed
testPearsonDistanceComposite 1.513s passed

Standard output

2016-01-14 15:21:55,144 INFO  platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(195)) - found 1 test platforms from classpath
2016-01-14 15:21:55,178 INFO  platform.PlatformRunner (PlatformRunner.java:addPlatform(244)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-01-14 15:21:55,214 INFO  tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-01-14 15:21:55,603 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-14 15:21:55,968 INFO  platform.TestPlatform (TestPlatform.java:getGlobalProperties(73)) - platform property overrides: 
2016-01-14 15:21:56,136 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:21:56,138 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:21:56,217 INFO  property.AppProps (AppProps.java:getAppID(177)) - using app.id: 86ED42454B414342AD844F975AE040CD
2016-01-14 15:21:56,515 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.160
2016-01-14 15:21:56,560 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.206
2016-01-14 15:21:56,562 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:21:56,562 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:21:56,563 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:21:56,693 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:21:56,911 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 15:21:57,187 INFO  util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.0.4
2016-01-14 15:21:57,189 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:21:57,190 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:57,191 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 15:21:57,191 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 15:21:57,191 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 15:21:57,191 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 15:21:57,192 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-01-14 15:21:57,193 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:21:57,213 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:21:57,395 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:21:57,402 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:21:57,402 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 15:21:57,447 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001 doesn't exist and is created
2016-01-14 15:21:57,520 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:21:57,521 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,522 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001
2016-01-14 15:21:57,553 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784917401_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:21:57,605 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:21:57,621 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-01-14 15:21:57,621 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-01-14 15:21:57,622 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,626 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:21:57,649 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:21:57,653 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:21:57,656 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir
2016-01-14 15:21:57,657 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/history.txt.appattempt_1452784917401_0001_000000, maxErrors=10
2016-01-14 15:21:57,657 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:21:57,660 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784917401_0001_000000, appSubmitTime=1452784917539, launchTime=1452784917552
2016-01-14 15:21:57,665 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:21:57,666 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:21:57,670 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:21:57,670 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:21:57,722 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 15:21:57,722 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 15:21:57,722 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,728 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:56013
2016-01-14 15:21:57,770 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:21:57,823 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 15:21:57,823 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 15:21:57,823 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,845 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:21:57,845 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784917401_0001_000000, startTime=1452784917844
2016-01-14 15:21:57,845 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:21:57,923 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:21:57,940 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:21:57,941 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001, dagName=[CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,163 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,264 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784917401_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/dag_1452784917401_0001_1.dot
2016-01-14 15:21:58,268 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/dag_1452784917401_0001_1-tez-dag.pb.txt
2016-01-14 15:21:58,408 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784917401_0001_1
2016-01-14 15:21:58,408 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:58,409 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58 Running Dag: dag_1452784917401_0001_1
2016-01-14 15:21:58,459 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:21:58,459 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784917401_0001_1, submitTime=1452784918163
2016-01-14 15:21:58,490 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:58,500 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:21:58,503 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:21:58,503 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784917401_0001_1, initTime=1452784918461
2016-01-14 15:21:58,504 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:21:58,507 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:21:58,507 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_STARTED]: dagID=dag_1452784917401_0001_1, startTime=1452784918506
2016-01-14 15:21:58,507 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:21:58,508 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:21:58,509 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E41B0DA6047647CD86110E309795E83D : {E2A47943811E44E4B9DEB0E1C823D43E={InputName=E2A47943811E44E4B9DEB0E1C823D43E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:21:58,509 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: E2A47943811E44E4B9DEB0E1C823D43E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:21:58,509 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,519 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,519 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,521 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001, dagName=[CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,524 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]. Starting root input initializers: 1
2016-01-14 15:21:58,529 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: E2A47943811E44E4B9DEB0E1C823D43E on vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,540 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:21:58,543 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,552 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784917401_0001, with dag id: dag_1452784917401_0001_1
2016-01-14 15:21:58,563 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,564 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,567 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:21:58,569 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input E2A47943811E44E4B9DEB0E1C823D43E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:21:58,570 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:21:58,570 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:21:58,583 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,586 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,586 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A07D163EA427438690C0188ECA117939, vertexId=vertex_1452784917401_0001_1_01, initRequestedTime=1452784918541, initedTime=1452784918585, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,587 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,591 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,628 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,629 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,629 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,630 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, vertexId=vertex_1452784917401_0001_1_02, initRequestedTime=1452784918591, initedTime=1452784918630, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,630 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,631 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,632 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,635 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:21:58,640 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: E2A47943811E44E4B9DEB0E1C823D43E on vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,652 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,652 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,653 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,653 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,654 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:21:58,657 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,668 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=8E92A936D8E747F4AE5E78A4F1908DF1, vertexName=D317F954092243EE989F4AA89C4E0B92, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:21:58,676 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,678 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,679 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D317F954092243EE989F4AA89C4E0B92, vertexId=vertex_1452784917401_0001_1_03, initRequestedTime=1452784918632, initedTime=1452784918678, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,680 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,681 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:21:58,682 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {E2A47943811E44E4B9DEB0E1C823D43E=forAllWorkUnits=true, update=[1]}
2016-01-14 15:21:58,683 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] parallelism set to 1
2016-01-14 15:21:58,683 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:21:58,692 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_00, eventCount=1
2016-01-14 15:21:58,693 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,694 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E41B0DA6047647CD86110E309795E83D, vertexId=vertex_1452784917401_0001_1_00, initRequestedTime=1452784918509, initedTime=1452784918693, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:21:58,695 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] with distanceFromRoot: 0
2016-01-14 15:21:58,695 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:21:58,697 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:58,697 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,699 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,700 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_00, startRequestedTime=1452784918696, startedTime=1452784918696
2016-01-14 15:21:58,702 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,739 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:58,741 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskId=task_1452784917401_0001_1_00_000000, scheduledTime=1452784918739, launchTime=1452784918739
2016-01-14 15:21:58,742 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:58,744 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_00 for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,744 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] with distanceFromRoot: 1
2016-01-14 15:21:58,746 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,751 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E41B0DA6047647CD86110E309795E83D in vertex: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,754 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A07D163EA427438690C0188ECA117939 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,755 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_01, startRequestedTime=1452784918746, startedTime=1452784918746
2016-01-14 15:21:58,755 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,755 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,765 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E41B0DA6047647CD86110E309795E83D, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:58,767 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:58,767 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_01 for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,768 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] with distanceFromRoot: 2
2016-01-14 15:21:58,768 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:58,769 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A07D163EA427438690C0188ECA117939 in vertex: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:58,769 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E5DC0B0707E8449C9745B6C3C1E14658 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,771 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,772 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_02, startRequestedTime=1452784918768, startedTime=1452784918768
2016-01-14 15:21:58,772 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,773 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_02 for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,773 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] with distanceFromRoot: 3
2016-01-14 15:21:58,774 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E5DC0B0707E8449C9745B6C3C1E14658 in vertex: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:58,774 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D317F954092243EE989F4AA89C4E0B92 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,776 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,777 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_03, startRequestedTime=1452784918774, startedTime=1452784918774
2016-01-14 15:21:58,777 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,820 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:21:58,825 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:21:58,825 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:21:58,830 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000001
2016-01-14 15:21:58,830 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:58,837 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784917401_0001_00_000001
2016-01-14 15:21:58,837 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:21:58,845 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:58,845 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:58,846 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000001, launchTime=1452784918845
2016-01-14 15:21:58,848 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000001
2016-01-14 15:21:58,852 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_00_000000_0] to container: [container_1452784917401_0001_00_000001]
2016-01-14 15:21:58,852 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:58,854 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_00_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:21:58,855 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000001 given task: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,855 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,856 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, startTime=1452784918853, containerId=container_1452784917401_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:21:58,856 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:58,857 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:58,858 INFO  counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters:  GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-01-14 15:21:58 Starting to run new task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,862 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,863 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:58,863 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:58,864 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:58,865 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:58,865 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:58,866 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:58,894 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:58,894 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E41B0DA6047647CD86110E309795E83D, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:58,897 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:58,901 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,901 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:58,902 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,908 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:21:58,907 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:21:58,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:58,911 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:58,912 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:58,913 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,916 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,917 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,923 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:58,923 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,924 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,929 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:58,932 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:21:58,933 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,933 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,957 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E41B0DA6047647CD86110E309795E83D, ordinal: 0
2016-01-14 15:21:58,958 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E41B0DA6047647CD86110E309795E83D, mem on start (mb), free: 223, total: 281, max: 672
2016-01-14 15:21:58,958 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,959 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:58,959 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:58,959 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:58,960 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:58,967 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:21:58,967 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:21:58,967 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:58,968 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:58,968 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:21:58,969 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E2A47943811E44E4B9DEB0E1C823D43E, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:21:58,969 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A07D163EA427438690C0188ECA117939, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:21:58,969 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:58,970 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,970 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:21:58,970 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,971 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E2A47943811E44E4B9DEB0E1C823D43E being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:58,971 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:58,971 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:58,972 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:58,973 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,975 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:21:58,984 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@68f6db2f
2016-01-14 15:21:58,997 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@68f6db2f
2016-01-14 15:21:58,998 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:21:58,998 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:21:59,015 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:59,049 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:59,051 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,051 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,069 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@5620e5a8; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@9c5a4c5; comparator=cascading.tuple.hadoop.util.TupleComparator@ec48686; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,072 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,074 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,075 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,186 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:21:59,187 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:21:59,188 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:21:59,189 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:21:59,190 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:21:59,198 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:59,198 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:21:59,201 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E41B0DA6047647CD86110E309795E83D, all 1 inputs ready in: 00:00:00.002
2016-01-14 15:21:59,214 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,214 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E41B0DA6047647CD86110E309795E83D, mem on close (mb), free: 111, total: 281, max: 672
2016-01-14 15:21:59,215 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,216 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,216 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:21:59,217 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:21:59,237 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,260 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,261 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_00, eventCount=1
2016-01-14 15:21:59,267 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:21:59,267 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,270 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, startTime=1452784918853, finishTime=1452784919262, timeTaken=409, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:21:59,279 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:21:59,279 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,279 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,283 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,284 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,286 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,289 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,295 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskId=task_1452784917401_0001_1_00_000000, startTime=1452784918853, finishTime=1452784919288, timeTaken=435, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:21:59,296 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,296 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,297 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] : 1
2016-01-14 15:21:59,298 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:59,319 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,330 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, vertexId=vertex_1452784917401_0001_1_00, initRequestedTime=1452784918509, initedTime=1452784918693, startRequestedTime=1452784918696, startedTime=1452784918696, finishTime=1452784919298, timeTaken=602, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784918853, firstTasksToStart=[ task_1452784917401_0001_1_00_000000 ], lastTaskFinishTime=1452784919262, lastTasksToFinish=[ task_1452784917401_0001_1_00_000000 ], minTaskDuration=409, maxTaskDuration=409, avgTaskDuration=409.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_00_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,331 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,333 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,334 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000001, stoppedTime=1452784919333, exitStatus=0
2016-01-14 15:21:59,335 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,335 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] attempt: attempt_1452784917401_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,335 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,336 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,338 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:21:59,338 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:21:59,338 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:59,339 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=A07D163EA427438690C0188ECA117939, taskId=task_1452784917401_0001_1_01_000000, scheduledTime=1452784919338, launchTime=1452784919338
2016-01-14 15:21:59,339 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:59,339 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,340 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:21:59,340 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A07D163EA427438690C0188ECA117939, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,341 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:59,341 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:59,344 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000002
2016-01-14 15:21:59,344 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:59,346 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:21:59,346 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000001: 63 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:21:59,346 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000001, Exiting
2016-01-14 15:21:59,347 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,347 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,353 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:21:59,353 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000001 completed successfully
2016-01-14 15:21:59,354 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:21:59,355 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784917401_0001_00_000002
2016-01-14 15:21:59,361 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:21:59,366 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:59,370 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:59,370 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000002, launchTime=1452784919366
2016-01-14 15:21:59,374 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,375 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_01_000000_0] to container: [container_1452784917401_0001_00_000002]
2016-01-14 15:21:59,376 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:59,376 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000002 given task: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,377 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,377 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_01_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:21:59,377 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:59,377 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A07D163EA427438690C0188ECA117939, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, startTime=1452784919376, containerId=container_1452784917401_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:21:59,378 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:59,378 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,379 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,379 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,385 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:59,385 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:59,386 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:59,386 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:59,386 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:59,389 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A07D163EA427438690C0188ECA117939, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,389 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:59,390 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,392 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:59,392 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,395 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:21:59,395 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,400 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:21:59,405 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:59,409 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:59,411 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:59,411 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,409 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:59,417 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,425 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:59,426 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,427 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,440 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:21:59,441 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:21:59,441 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,441 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,446 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A07D163EA427438690C0188ECA117939, ordinal: 1
2016-01-14 15:21:59,447 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A07D163EA427438690C0188ECA117939, mem on start (mb), free: 99, total: 281, max: 672
2016-01-14 15:21:59,447 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,447 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:59,447 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:59,447 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:59,447 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:59,448 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:59,448 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:21:59,450 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:59,451 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:21:59,451 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E5DC0B0707E8449C9745B6C3C1E14658, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:21:59,451 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E41B0DA6047647CD86110E309795E83D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:21:59,451 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:59,452 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,458 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E41B0DA6047647CD86110E309795E83D being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:59,458 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:59,460 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:59,467 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,468 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:21:59,472 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E41B0DA6047647CD86110E309795E83D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:21:59,474 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:21:59,480 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:21:59,484 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:21:59,487 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,488 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:59,492 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,494 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_00_000000_0_10003, runDuration: 324320, ]
2016-01-14 15:21:59,499 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 15:21:59,505 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_00_000000_0_10003] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784919505, TimeTaken=3, Rate=0.35 MB/s
2016-01-14 15:21:59,508 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,509 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:21:59,513 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E41B0DA6047647CD86110E309795E83D] #1 in 12ms
2016-01-14 15:21:59,514 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,514 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,513 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E41B0DA6047647CD86110E309795E83D] #1, status:false, isInterrupted:false
2016-01-14 15:21:59,515 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:21:59,517 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:21:59,518 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,525 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:21:59,525 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,534 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@9aa1ae3; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@7ecb3209; comparator=cascading.tuple.hadoop.util.TupleComparator@4e422de7; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,534 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,537 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,537 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,540 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:21:59,541 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,605 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:21:59,606 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:21:59,606 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:21:59,607 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:21:59,607 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:21:59,607 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:21:59,608 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] CC67493F06964370956C01CE4B630E7C, for 1 inputs
2016-01-14 15:21:59,617 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:21:59,617 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:21:59,625 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A07D163EA427438690C0188ECA117939, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:21:59,628 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@253217dd; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7bbcc13e
2016-01-14 15:21:59,667 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,667 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A07D163EA427438690C0188ECA117939, mem on close (mb), free: 91, total: 345, max: 672
2016-01-14 15:21:59,667 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,668 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:21:59,668 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:21:59,668 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,669 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:21:59,670 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:21:59,671 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,671 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,671 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:21:59,671 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:21:59,703 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,707 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,709 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_01, eventCount=1
2016-01-14 15:21:59,709 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,712 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12]]
2016-01-14 15:21:59,714 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, startTime=1452784919376, finishTime=1452784919709, timeTaken=333, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-01-14 15:21:59,714 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,714 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,716 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,714 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:21:59,716 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,717 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,718 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, taskId=task_1452784917401_0001_1_01_000000, startTime=1452784919376, finishTime=1452784919716, timeTaken=340, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-01-14 15:21:59,718 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,718 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] : 1
2016-01-14 15:21:59,719 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,719 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,721 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,726 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, vertexId=vertex_1452784917401_0001_1_01, initRequestedTime=1452784918541, initedTime=1452784918585, startRequestedTime=1452784918746, startedTime=1452784918746, finishTime=1452784919719, timeTaken=973, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12, vertexStats=firstTaskStartTime=1452784919376, firstTasksToStart=[ task_1452784917401_0001_1_01_000000 ], lastTaskFinishTime=1452784919709, lastTasksToFinish=[ task_1452784917401_0001_1_01_000000 ], minTaskDuration=333, maxTaskDuration=333, avgTaskDuration=333.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_01_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,729 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,731 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,732 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000002, stoppedTime=1452784919732, exitStatus=0
2016-01-14 15:21:59,733 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:21:59,733 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,734 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:21:59,734 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000002, Exiting
2016-01-14 15:21:59,734 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,734 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,734 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,735 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] attempt: attempt_1452784917401_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,735 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,736 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,736 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:21:59,737 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:21:59,738 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:21:59,738 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:59,739 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000002 completed successfully
2016-01-14 15:21:59,738 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskId=task_1452784917401_0001_1_02_000000, scheduledTime=1452784919738, launchTime=1452784919738
2016-01-14 15:21:59,740 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:59,741 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:21:59,742 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,743 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:21:59,745 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E5DC0B0707E8449C9745B6C3C1E14658, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,745 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:59,746 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:59,748 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000003
2016-01-14 15:21:59,748 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:59,751 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784917401_0001_00_000003
2016-01-14 15:21:59,752 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:21:59,754 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:59,754 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000003, launchTime=1452784919754
2016-01-14 15:21:59,754 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:59,755 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000003
2016-01-14 15:21:59,756 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_02_000000_0] to container: [container_1452784917401_0001_00_000003]
2016-01-14 15:21:59,756 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:59,757 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000003 given task: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,757 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,757 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_02_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:21:59,758 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,758 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,758 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,759 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:59,759 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:59,759 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:59,759 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:59,759 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:59,760 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E5DC0B0707E8449C9745B6C3C1E14658, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,759 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, startTime=1452784919757, containerId=container_1452784917401_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:21:59,760 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:59,761 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:59,761 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:59,763 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:21:59,764 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:59,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,766 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:21:59,766 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:59,766 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,768 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:59,768 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,767 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:59,768 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,771 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:59,771 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,771 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,772 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:59,772 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:21:59,772 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:21:59,773 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,773 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,787 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, ordinal: 2
2016-01-14 15:21:59,787 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, mem on start (mb), free: 84, total: 345, max: 672
2016-01-14 15:21:59,787 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,788 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:59,788 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:59,788 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:59,789 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:59,789 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:59,789 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:21:59,790 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:59,790 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:21:59,791 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D317F954092243EE989F4AA89C4E0B92, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:21:59,791 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A07D163EA427438690C0188ECA117939, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:21:59,791 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:59,792 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A07D163EA427438690C0188ECA117939 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:59,792 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,794 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:59,796 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:59,797 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,798 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:21:59,798 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A07D163EA427438690C0188ECA117939 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:21:59,799 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:21:59,799 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:21:59,799 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:21:59,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:59,802 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,805 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_01_000000_0_10006, runDuration: 292878, ]
2016-01-14 15:21:59,808 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_01_000000_0_10006] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784919808, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:21:59,808 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,809 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:21:59,811 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A07D163EA427438690C0188ECA117939] #2 in 5ms
2016-01-14 15:21:59,811 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A07D163EA427438690C0188ECA117939] #2, status:false, isInterrupted:false
2016-01-14 15:21:59,811 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:21:59,814 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:21:59,814 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:21:59,814 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,815 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:21:59,816 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:21:59,816 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,819 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:21:59,821 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,825 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:21:59,826 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:21:59,827 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@46423a2d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@25f98464; comparator=cascading.tuple.hadoop.util.TupleComparator@2062d4b6; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,828 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,828 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,829 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,918 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:21:59,919 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:21:59,919 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:21:59,920 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:21:59,920 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:21:59,921 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 4AC4B50191CE4F079527F017D3591EC2, for 1 inputs
2016-01-14 15:21:59,922 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:21:59,923 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@75789cb0; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@41772310
2016-01-14 15:21:59,958 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,958 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, mem on close (mb), free: 79, total: 408, max: 672
2016-01-14 15:21:59,958 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,959 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:21:59,960 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:21:59,961 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,961 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:21:59,961 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:21:59,962 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,962 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,963 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 15:21:59,963 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:21:59,973 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,979 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,980 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 15:21:59,980 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_02, eventCount=1
2016-01-14 15:21:59,981 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,982 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, startTime=1452784919757, finishTime=1452784919981, timeTaken=224, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:21:59,983 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,983 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,983 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,985 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:21:59,986 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,987 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,988 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskId=task_1452784917401_0001_1_02_000000, startTime=1452784919757, finishTime=1452784919983, timeTaken=226, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:21:59,990 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,990 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] : 1
2016-01-14 15:21:59,991 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,992 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,990 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000003
2016-01-14 15:21:59,996 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, vertexId=vertex_1452784917401_0001_1_02, initRequestedTime=1452784918591, initedTime=1452784918630, startRequestedTime=1452784918768, startedTime=1452784918768, finishTime=1452784919991, timeTaken=1223, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452784919757, firstTasksToStart=[ task_1452784917401_0001_1_02_000000 ], lastTaskFinishTime=1452784919981, lastTasksToFinish=[ task_1452784917401_0001_1_02_000000 ], minTaskDuration=224, maxTaskDuration=224, avgTaskDuration=224.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_02_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,996 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,996 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,997 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000003, stoppedTime=1452784919997, exitStatus=0
2016-01-14 15:21:59,997 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,998 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,999 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] attempt: attempt_1452784917401_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,999 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:22:00,000 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,000 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:00,001 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:00,001 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:00,001 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskId=task_1452784917401_0001_1_03_000000, scheduledTime=1452784920001, launchTime=1452784920001
2016-01-14 15:22:00,002 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:00,002 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,003 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:00,003 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: D317F954092243EE989F4AA89C4E0B92, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:00,003 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:00,004 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:00,004 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:00,005 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:00,005 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000003, Exiting
2016-01-14 15:22:00,005 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000003
2016-01-14 15:22:00,005 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000003
2016-01-14 15:22:00,006 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:00,007 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000003 completed successfully
2016-01-14 15:22:00,008 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000004
2016-01-14 15:22:00,008 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:00,008 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:00,011 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784917401_0001_00_000004
2016-01-14 15:22:00,012 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:00,016 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:00,016 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000004, launchTime=1452784920016
2016-01-14 15:22:00,016 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:00,017 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,018 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_03_000000_0] to container: [container_1452784917401_0001_00_000004]
2016-01-14 15:22:00,018 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:00,018 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000004 given task: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,018 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00 Starting to run new task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,019 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,019 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,019 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:00,019 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:00,019 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:00,019 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:00,020 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:00,020 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: D317F954092243EE989F4AA89C4E0B92, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:00,020 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:00,021 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,023 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:00,023 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,024 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_03_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:00,025 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:00,026 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:00,024 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,026 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, startTime=1452784920018, containerId=container_1452784917401_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:00,032 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:00,032 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:00,031 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:00,027 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:00,027 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:00,036 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:00,037 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,037 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:00,037 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,039 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:00,039 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,042 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:00,043 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:00,043 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,045 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,060 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D317F954092243EE989F4AA89C4E0B92, ordinal: 3
2016-01-14 15:22:00,061 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D317F954092243EE989F4AA89C4E0B92, mem on start (mb), free: 73, total: 408, max: 672
2016-01-14 15:22:00,061 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,062 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:00,062 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:00,072 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:00,072 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,074 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:00,074 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:00,075 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:00,078 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:00,078 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:00,078 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:00,078 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:00,079 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 8E92A936D8E747F4AE5E78A4F1908DF1, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:00,079 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E5DC0B0707E8449C9745B6C3C1E14658, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:00,080 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:00,081 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E5DC0B0707E8449C9745B6C3C1E14658 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:00,081 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,082 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:00,082 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:00,082 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E5DC0B0707E8449C9745B6C3C1E14658 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:00,083 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:00,083 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:00,083 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:00,084 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:00,085 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:00,085 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,086 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:00,087 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,089 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_02_000000_0_10009, runDuration: 207486, ]
2016-01-14 15:22:00,092 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-01-14 15:22:00,092 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_02_000000_0_10009] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452784920092, TimeTaken=2, Rate=2.07 MB/s
2016-01-14 15:22:00,093 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,093 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:00,094 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E5DC0B0707E8449C9745B6C3C1E14658] #3 in 4ms
2016-01-14 15:22:00,094 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E5DC0B0707E8449C9745B6C3C1E14658] #3, status:false, isInterrupted:false
2016-01-14 15:22:00,094 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:22:00,094 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,096 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:00,096 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 15:22:00,096 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:00,096 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:00,097 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 15:22:00,097 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,097 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:00,098 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 15:22:00,099 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 1F5BA0AA0DD54EC78DF32C09F8E9E519, for 1 inputs
2016-01-14 15:22:00,100 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D317F954092243EE989F4AA89C4E0B92, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:00,103 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6aff5bd4; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@22e5f22d
2016-01-14 15:22:00,117 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:00,117 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:00,118 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,119 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784917401_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:00,120 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,121 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D317F954092243EE989F4AA89C4E0B92, mem on close (mb), free: 69, total: 408, max: 672
2016-01-14 15:22:00,121 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,122 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:00,123 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:00,123 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:00,124 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:00,124 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:00 Completed running task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,126 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:00,127 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:00,129 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, startTime=1452784920018, finishTime=1452784920127, timeTaken=109, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:00,130 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:00,130 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,130 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:00,131 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:00,133 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskId=task_1452784917401_0001_1_03_000000, startTime=1452784920018, finishTime=1452784920130, timeTaken=112, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:00,133 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:00,133 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] : 1
2016-01-14 15:22:00,133 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,132 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:00,133 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:00,135 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,135 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, vertexId=vertex_1452784917401_0001_1_03, initRequestedTime=1452784918632, initedTime=1452784918678, startRequestedTime=1452784918774, startedTime=1452784918774, finishTime=1452784920133, timeTaken=1359, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784920018, firstTasksToStart=[ task_1452784917401_0001_1_03_000000 ], lastTaskFinishTime=1452784920127, lastTasksToFinish=[ task_1452784917401_0001_1_03_000000 ], minTaskDuration=109, maxTaskDuration=109, avgTaskDuration=109.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_03_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:00,136 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:00,137 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:00,137 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000004, stoppedTime=1452784920137, exitStatus=0
2016-01-14 15:22:00,138 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:00,138 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:00,138 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784917401_0001_1
2016-01-14 15:22:00,136 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,140 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:00,137 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:00,143 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:00,144 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:00,144 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000004, Exiting
2016-01-14 15:22:00,144 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,144 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,144 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784917401_0001_1
2016-01-14 15:22:00,145 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:00,147 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:22:00,147 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000004 completed successfully
2016-01-14 15:22:00,147 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 8E92A936D8E747F4AE5E78A4F1908DF1 for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,162 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:22:00,162 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:22:00,172 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784917401_0001_1, queueSize=0
2016-01-14 15:22:00,172 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:00,173 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784917401_0001_1, startTime=1452784918506, finishTime=1452784920163, timeTaken=1657, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14984, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1513619456, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=65, MERGE_PHASE_TIME=106, FIRST_EVENT_RECEIVED=49, LAST_EVENT_RECEIVED=49, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139678849, Process_Duration=110, Process_End_Time=5811139678959, Read_Duration=23, Tuples_Read=303, Tuples_Written=317, Write_Duration=22, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:00,176 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784917401_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:00,176 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:00,176 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784917401_0001_00_000004
2016-01-14 15:22:00,176 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:00 Completed Dag: dag_1452784917401_0001_1
2016-01-14 15:22:00,179 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784917401_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:00,179 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784917401_0001_1_post
2016-01-14 15:22:00,180 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,181 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:00,187 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001
2016-01-14 15:22:00,192 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:00,192 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:00,192 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:00,192 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:00,193 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:00,193 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:00,196 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:00,196 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:00,197 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:00,197 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:00,198 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:00,199 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:00,199 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:00,208 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001
2016-01-14 15:22:00,210 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:03.020
2016-01-14 15:22:00,346 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 15:22:00,386 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:00,386 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:00,487 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.069
2016-01-14 15:22:00,534 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.115
2016-01-14 15:22:00,535 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:00,536 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:00,536 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:00,560 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:00,590 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 15:22:00,784 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting
2016-01-14 15:22:00,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:00,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 15:22:00,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  parallel execution of steps is enabled: false
2016-01-14 15:22:00,786 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  executing total steps: 1
2016-01-14 15:22:00,786 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  allocating management threads: 1
2016-01-14 15:22:00,786 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,787 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:00,801 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tez session mode enabled: true
2016-01-14 15:22:00,806 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:00,806 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:00,807 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:00,817 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001 doesn't exist and is created
2016-01-14 15:22:00,838 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:00,838 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:00,840 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001
2016-01-14 15:22:00,844 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784920806_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:00,859 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:00,860 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:00,863 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:00,863 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:00,864 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir
2016-01-14 15:22:00,864 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/history.txt.appattempt_1452784920806_0001_000000, maxErrors=10
2016-01-14 15:22:00,864 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:00,865 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784920806_0001_000000, appSubmitTime=1452784920843, launchTime=1452784920843
2016-01-14 15:22:00,871 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:00,875 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:00,877 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:00,877 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:00,881 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:41215
2016-01-14 15:22:00,884 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:00,888 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784920806_0001_000000, startTime=1452784920888
2016-01-14 15:22:00,888 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:00,888 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:00,939 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:00,940 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:00,940 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001, dagName=[A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,963 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,966 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784920806_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/dag_1452784920806_0001_1.dot
2016-01-14 15:22:00,967 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/dag_1452784920806_0001_1-tez-dag.pb.txt
2016-01-14 15:22:01,004 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784920806_0001_1
2016-01-14 15:22:01,004 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,004 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:01 Running Dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,020 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784920806_0001_1, submitTime=1452784920963
2016-01-14 15:22:01,020 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:01,022 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,022 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:01,023 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784920806_0001_1, initTime=1452784921020
2016-01-14 15:22:01,024 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:01,025 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:01,026 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_STARTED]: dagID=dag_1452784920806_0001_1, startTime=1452784921025
2016-01-14 15:22:01,026 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:01,026 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:01,026 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: C58271B0BFDE40AEA813831159DCF987 : {CB71003DE96947C99F8E2532DE037A51={InputName=CB71003DE96947C99F8E2532DE037A51}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:01,026 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: CB71003DE96947C99F8E2532DE037A51, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:01,026 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:01,027 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]. Starting root input initializers: 1
2016-01-14 15:22:01,029 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:01,029 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,029 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001, dagName=[A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:01,031 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: CB71003DE96947C99F8E2532DE037A51 on vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,037 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:01,038 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input CB71003DE96947C99F8E2532DE037A51 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:01,038 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,038 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,041 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,042 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:01,042 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, vertexId=vertex_1452784920806_0001_1_01, initRequestedTime=1452784921029, initedTime=1452784921042, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,042 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,043 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,043 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,045 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:01,048 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:01,048 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,048 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,048 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,049 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: CB71003DE96947C99F8E2532DE037A51 on vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,049 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,050 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D4533A9745A740C398F52479AA7C29A1, vertexId=vertex_1452784920806_0001_1_02, initRequestedTime=1452784921043, initedTime=1452784921049, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,050 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,050 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:01,051 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {CB71003DE96947C99F8E2532DE037A51=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:01,051 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] parallelism set to 1
2016-01-14 15:22:01,051 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:01,054 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_00, eventCount=1
2016-01-14 15:22:01,055 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,055 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C58271B0BFDE40AEA813831159DCF987, vertexId=vertex_1452784920806_0001_1_00, initRequestedTime=1452784921026, initedTime=1452784921055, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:01,055 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] with distanceFromRoot: 0
2016-01-14 15:22:01,056 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:01,056 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,065 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] submitted tez dag to app master: application_1452784920806_0001, with dag id: dag_1452784920806_0001_1
2016-01-14 15:22:01,067 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,067 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:01,069 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,071 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=AA5BC7EE4A07438280C31731A446F7FB, vertexName=B383C94B7B3947F0A2BD91A29DE0815D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:01,071 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,073 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, vertexId=vertex_1452784920806_0001_1_03, initRequestedTime=1452784921056, initedTime=1452784921073, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,073 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,073 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,074 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,074 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,075 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_00, startRequestedTime=1452784921074, startedTime=1452784921074
2016-01-14 15:22:01,075 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,076 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,077 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskId=task_1452784920806_0001_1_00_000000, scheduledTime=1452784921077, launchTime=1452784921077
2016-01-14 15:22:01,077 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,077 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,078 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_00 for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,079 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] with distanceFromRoot: 1
2016-01-14 15:22:01,079 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,080 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C58271B0BFDE40AEA813831159DCF987 in vertex: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,080 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A25E8F44D00D4AD7A993BB5C20FE0E40 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,080 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,080 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_01, startRequestedTime=1452784921080, startedTime=1452784921080
2016-01-14 15:22:01,081 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,081 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: C58271B0BFDE40AEA813831159DCF987, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,082 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,082 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_01 for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,082 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] with distanceFromRoot: 2
2016-01-14 15:22:01,082 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,083 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A25E8F44D00D4AD7A993BB5C20FE0E40 in vertex: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,083 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:01,083 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D4533A9745A740C398F52479AA7C29A1 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,084 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,084 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_02, startRequestedTime=1452784921083, startedTime=1452784921083
2016-01-14 15:22:01,084 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,085 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:01,086 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:01,086 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000001
2016-01-14 15:22:01,087 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,087 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_02 for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,087 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] with distanceFromRoot: 3
2016-01-14 15:22:01,091 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784920806_0001_00_000001
2016-01-14 15:22:01,091 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:01,093 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D4533A9745A740C398F52479AA7C29A1 in vertex: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,093 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B383C94B7B3947F0A2BD91A29DE0815D with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,093 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,093 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_03, startRequestedTime=1452784921092, startedTime=1452784921092
2016-01-14 15:22:01,093 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000001, launchTime=1452784921093
2016-01-14 15:22:01,093 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,093 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,094 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,096 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,097 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_00_000000_0] to container: [container_1452784920806_0001_00_000001]
2016-01-14 15:22:01,097 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,098 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000001 given task: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,098 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,098 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_00_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,099 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,099 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, startTime=1452784921098, containerId=container_1452784920806_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:01,099 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,099 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,100 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,100 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,100 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,101 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,101 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,101 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,101 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,102 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: C58271B0BFDE40AEA813831159DCF987, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,102 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,103 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,103 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,105 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,105 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:01,107 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:01,108 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,108 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,109 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,109 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,110 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,112 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,112 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,139 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,141 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,142 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,143 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,144 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:01,145 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,145 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,153 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C58271B0BFDE40AEA813831159DCF987, ordinal: 0
2016-01-14 15:22:01,153 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C58271B0BFDE40AEA813831159DCF987, mem on start (mb), free: 260, total: 516, max: 672
2016-01-14 15:22:01,154 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,154 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,154 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,154 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,154 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,154 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:01,155 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:01,155 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,155 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,155 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:01,155 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CB71003DE96947C99F8E2532DE037A51, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:01,155 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A25E8F44D00D4AD7A993BB5C20FE0E40, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:01,155 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,155 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CB71003DE96947C99F8E2532DE037A51 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,155 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,156 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:01,156 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,156 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,157 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,157 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,157 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,157 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:01,159 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@49557cef
2016-01-14 15:22:01,160 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@49557cef
2016-01-14 15:22:01,160 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:01,161 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:01,163 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:01,164 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,164 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,164 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,166 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@1ce0f06c; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@73c85143; comparator=cascading.tuple.hadoop.util.TupleComparator@644f4117; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,166 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,167 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,167 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,181 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:01,182 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:01,182 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:01,182 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:01,183 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:01,183 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:01,184 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:01,184 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C58271B0BFDE40AEA813831159DCF987, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,189 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,189 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C58271B0BFDE40AEA813831159DCF987, mem on close (mb), free: 158, total: 516, max: 672
2016-01-14 15:22:01,189 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,189 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,190 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:01,190 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:01,194 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,199 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,200 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:01,200 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_00, eventCount=1
2016-01-14 15:22:01,201 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:01,202 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,202 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,203 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, startTime=1452784921098, finishTime=1452784921202, timeTaken=104, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:01,204 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,204 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,205 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskId=task_1452784920806_0001_1_00_000000, startTime=1452784921098, finishTime=1452784921204, timeTaken=106, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:01,205 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,206 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] : 1
2016-01-14 15:22:01,206 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,204 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,205 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,208 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, vertexId=vertex_1452784920806_0001_1_00, initRequestedTime=1452784921026, initedTime=1452784921055, startRequestedTime=1452784921074, startedTime=1452784921074, finishTime=1452784921206, timeTaken=132, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784921098, firstTasksToStart=[ task_1452784920806_0001_1_00_000000 ], lastTaskFinishTime=1452784921202, lastTasksToFinish=[ task_1452784920806_0001_1_00_000000 ], minTaskDuration=104, maxTaskDuration=104, avgTaskDuration=104.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_00_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,209 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,210 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,213 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,214 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] attempt: attempt_1452784920806_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,214 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,214 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,215 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,215 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,215 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000001, stoppedTime=1452784921215, exitStatus=0
2016-01-14 15:22:01,215 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,215 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,218 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,218 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskId=task_1452784920806_0001_1_01_000000, scheduledTime=1452784921218, launchTime=1452784921218
2016-01-14 15:22:01,218 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,218 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,219 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,219 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,219 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: A25E8F44D00D4AD7A993BB5C20FE0E40, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,219 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,219 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,220 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000002
2016-01-14 15:22:01,221 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,225 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784920806_0001_00_000002
2016-01-14 15:22:01,225 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:01,226 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,226 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,226 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000002, launchTime=1452784921226
2016-01-14 15:22:01,230 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,231 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_01_000000_0] to container: [container_1452784920806_0001_00_000002]
2016-01-14 15:22:01,231 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,232 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_01_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,232 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, startTime=1452784921232, containerId=container_1452784920806_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:01,233 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,233 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,233 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,233 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000002 given task: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,233 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,234 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,234 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,234 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,234 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,234 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,235 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,235 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,235 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,235 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,236 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000001, Exiting
2016-01-14 15:22:01,236 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,236 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,236 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: A25E8F44D00D4AD7A993BB5C20FE0E40, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,238 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,238 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000001 completed successfully
2016-01-14 15:22:01,239 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,239 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,240 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,240 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,240 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,241 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,241 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,243 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,244 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,246 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:01,241 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,247 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,248 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,249 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,250 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,250 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,250 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,247 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,254 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,254 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,255 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,263 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, ordinal: 1
2016-01-14 15:22:01,263 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, mem on start (mb), free: 148, total: 516, max: 672
2016-01-14 15:22:01,263 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,263 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,263 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,263 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,263 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,264 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,264 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,264 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,264 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:01,264 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C58271B0BFDE40AEA813831159DCF987, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:01,265 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D4533A9745A740C398F52479AA7C29A1, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:01,265 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,265 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,265 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C58271B0BFDE40AEA813831159DCF987 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,266 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,267 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,267 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C58271B0BFDE40AEA813831159DCF987 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,267 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:01,267 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,268 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,268 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,272 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,272 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,273 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,273 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,273 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_00_000000_0_10015, runDuration: 83630, ]
2016-01-14 15:22:01,278 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_00_000000_0_10015] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784921278, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:01,278 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,278 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:01,278 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [C58271B0BFDE40AEA813831159DCF987] #4 in 1ms
2016-01-14 15:22:01,278 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 15:22:01,279 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C58271B0BFDE40AEA813831159DCF987] #4, status:false, isInterrupted:false
2016-01-14 15:22:01,280 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,280 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:01,280 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,281 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,281 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:01,281 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,282 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,282 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,283 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,283 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,285 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@332803fd; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@81df148; comparator=cascading.tuple.hadoop.util.TupleComparator@65cbb221; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,285 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,285 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,286 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,337 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:01,337 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:01,337 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:01,338 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:01,338 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:01,339 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 8EF3D351182B4FC9B104498D5B49B7BB, for 1 inputs
2016-01-14 15:22:01,340 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:01,340 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:01,341 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,342 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7f814a00; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1218766
2016-01-14 15:22:01,371 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,371 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, mem on close (mb), free: 81, total: 516, max: 672
2016-01-14 15:22:01,371 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,372 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,372 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,372 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,372 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,373 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01,373 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,373 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,373 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:01,374 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:01,394 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,398 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,400 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_01, eventCount=1
2016-01-14 15:22:01,402 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,404 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10]]
2016-01-14 15:22:01,404 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, startTime=1452784921232, finishTime=1452784921402, timeTaken=170, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10
2016-01-14 15:22:01,404 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:01,404 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,404 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,405 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,406 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,406 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskId=task_1452784920806_0001_1_01_000000, startTime=1452784921232, finishTime=1452784921405, timeTaken=173, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10
2016-01-14 15:22:01,406 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,407 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] : 1
2016-01-14 15:22:01,407 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,405 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,405 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,410 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, vertexId=vertex_1452784920806_0001_1_01, initRequestedTime=1452784921029, initedTime=1452784921042, startRequestedTime=1452784921080, startedTime=1452784921080, finishTime=1452784921407, timeTaken=327, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10, vertexStats=firstTaskStartTime=1452784921232, firstTasksToStart=[ task_1452784920806_0001_1_01_000000 ], lastTaskFinishTime=1452784921402, lastTasksToFinish=[ task_1452784920806_0001_1_01_000000 ], minTaskDuration=170, maxTaskDuration=170, avgTaskDuration=170.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_01_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,410 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,410 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,413 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] attempt: attempt_1452784920806_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,413 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,414 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,414 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,415 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,415 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000002, stoppedTime=1452784921415, exitStatus=0
2016-01-14 15:22:01,416 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,416 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,417 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskId=task_1452784920806_0001_1_02_000000, scheduledTime=1452784921417, launchTime=1452784921417
2016-01-14 15:22:01,417 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,417 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,418 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,418 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,419 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,419 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,419 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,420 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000002, Exiting
2016-01-14 15:22:01,420 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,420 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,420 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: D4533A9745A740C398F52479AA7C29A1, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,420 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,420 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,421 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,421 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000002 completed successfully
2016-01-14 15:22:01,421 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,422 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000003
2016-01-14 15:22:01,423 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,426 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784920806_0001_00_000003
2016-01-14 15:22:01,428 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:01,430 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,430 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,431 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000003, launchTime=1452784921430
2016-01-14 15:22:01,432 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,433 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_02_000000_0] to container: [container_1452784920806_0001_00_000003]
2016-01-14 15:22:01,433 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,434 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000003 given task: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,434 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,435 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_02_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,435 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,435 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, startTime=1452784921434, containerId=container_1452784920806_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:01,436 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,436 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,437 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,437 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,437 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,437 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,438 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,438 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,438 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,438 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: D4533A9745A740C398F52479AA7C29A1, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,439 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,441 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,443 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,443 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,443 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:01,445 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,445 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,445 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,445 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,446 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,446 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,446 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,448 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,448 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,448 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,449 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,449 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,449 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,450 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,453 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,464 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D4533A9745A740C398F52479AA7C29A1, ordinal: 2
2016-01-14 15:22:01,464 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4533A9745A740C398F52479AA7C29A1, mem on start (mb), free: 73, total: 516, max: 672
2016-01-14 15:22:01,464 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,465 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,465 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,465 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,465 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,466 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,466 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,467 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,467 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:01,467 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A25E8F44D00D4AD7A993BB5C20FE0E40, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:01,468 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B383C94B7B3947F0A2BD91A29DE0815D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:01,468 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,468 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A25E8F44D00D4AD7A993BB5C20FE0E40 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,468 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,468 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,470 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,470 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,471 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,472 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A25E8F44D00D4AD7A993BB5C20FE0E40 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,472 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:01,472 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,473 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,475 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,475 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,475 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,477 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_01_000000_0_10017, runDuration: 151311, ]
2016-01-14 15:22:01,478 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_01_000000_0_10017] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784921478, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:22:01,479 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,479 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:01,480 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A25E8F44D00D4AD7A993BB5C20FE0E40] #5 in 3ms
2016-01-14 15:22:01,480 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-01-14 15:22:01,480 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A25E8F44D00D4AD7A993BB5C20FE0E40] #5, status:false, isInterrupted:false
2016-01-14 15:22:01,481 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,482 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,482 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,482 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:01,482 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,482 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,484 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,484 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@59d75340; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@479abcd4; comparator=cascading.tuple.hadoop.util.TupleComparator@7fe32705; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,486 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,486 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,487 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,487 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:01,487 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,549 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:01,549 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,550 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:01,550 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:01,550 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:01,550 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:01,551 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CDC24F3386C2421B89475874E56F482D, for 1 inputs
2016-01-14 15:22:01,552 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D4533A9745A740C398F52479AA7C29A1, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,553 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3b3bb7fb; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4dabde1e
2016-01-14 15:22:01,573 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,573 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4533A9745A740C398F52479AA7C29A1, mem on close (mb), free: 70, total: 580, max: 672
2016-01-14 15:22:01,573 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,573 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,574 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,574 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,574 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,574 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01,574 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,575 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,575 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 15:22:01,575 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:01,579 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,584 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,584 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_02, eventCount=1
2016-01-14 15:22:01,585 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,586 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, startTime=1452784921434, finishTime=1452784921585, timeTaken=151, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:01,586 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,586 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 15:22:01,587 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,587 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:01,587 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,589 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,589 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskId=task_1452784920806_0001_1_02_000000, startTime=1452784921434, finishTime=1452784921588, timeTaken=154, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:01,590 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,590 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] : 1
2016-01-14 15:22:01,590 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,588 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,593 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, vertexId=vertex_1452784920806_0001_1_02, initRequestedTime=1452784921043, initedTime=1452784921049, startRequestedTime=1452784921083, startedTime=1452784921083, finishTime=1452784921590, timeTaken=507, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452784921434, firstTasksToStart=[ task_1452784920806_0001_1_02_000000 ], lastTaskFinishTime=1452784921585, lastTasksToFinish=[ task_1452784920806_0001_1_02_000000 ], minTaskDuration=151, maxTaskDuration=151, avgTaskDuration=151.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_02_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,593 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,593 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,594 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000003, stoppedTime=1452784921593, exitStatus=0
2016-01-14 15:22:01,594 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,594 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] attempt: attempt_1452784920806_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,593 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,594 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,594 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,595 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,594 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,595 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,595 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,596 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,596 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000003, Exiting
2016-01-14 15:22:01,596 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,596 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,596 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskId=task_1452784920806_0001_1_03_000000, scheduledTime=1452784921596, launchTime=1452784921596
2016-01-14 15:22:01,596 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,597 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,598 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,598 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,598 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000003 completed successfully
2016-01-14 15:22:01,599 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,599 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,599 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: B383C94B7B3947F0A2BD91A29DE0815D, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:01,599 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,600 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,602 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000004
2016-01-14 15:22:01,603 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,599 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,605 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784920806_0001_00_000004
2016-01-14 15:22:01,606 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:01,608 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000004, launchTime=1452784921608
2016-01-14 15:22:01,608 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,608 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,608 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,610 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_03_000000_0] to container: [container_1452784920806_0001_00_000004]
2016-01-14 15:22:01,610 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,611 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000004 given task: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,611 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,611 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,612 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,612 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,612 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,612 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,612 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,612 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,613 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_03_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,613 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: B383C94B7B3947F0A2BD91A29DE0815D, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:01,613 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, startTime=1452784921612, containerId=container_1452784920806_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:01,613 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,613 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,613 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,614 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,614 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,616 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,617 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,616 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:01,618 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,618 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:01,620 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,620 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,621 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:01,621 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,623 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,624 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,625 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,625 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,631 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,632 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,632 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,634 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,641 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:01,642 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,649 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, ordinal: 3
2016-01-14 15:22:01,649 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, mem on start (mb), free: 64, total: 580, max: 672
2016-01-14 15:22:01,650 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,650 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,651 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,651 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,651 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,652 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:01,652 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:01,653 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,653 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,654 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:01,654 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, AA5BC7EE4A07438280C31731A446F7FB, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:01,654 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D4533A9745A740C398F52479AA7C29A1, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:01,655 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,655 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,656 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,656 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,657 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D4533A9745A740C398F52479AA7C29A1 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,657 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:01,657 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,657 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,658 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D4533A9745A740C398F52479AA7C29A1 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,659 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,661 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,662 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,662 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,663 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,663 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_02_000000_0_10021, runDuration: 135175, ]
2016-01-14 15:22:01,665 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_02_000000_0_10021] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452784921665, TimeTaken=1, Rate=4.14 MB/s
2016-01-14 15:22:01,665 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,666 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:01,666 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D4533A9745A740C398F52479AA7C29A1] #6 in 2ms
2016-01-14 15:22:01,666 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D4533A9745A740C398F52479AA7C29A1] #6, status:false, isInterrupted:false
2016-01-14 15:22:01,667 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,667 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 15:22:01,667 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,667 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,668 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 15:22:01,668 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,668 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,669 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-01-14 15:22:01,670 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,671 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,671 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 15:22:01,671 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 63CCBAE3A50F4E7A8F917368CBE80CF6, for 1 inputs
2016-01-14 15:22:01,672 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,673 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@23b2412a; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@bd43eb5
2016-01-14 15:22:01,692 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:01,692 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:01,693 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,693 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784920806_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:01,694 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,695 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, mem on close (mb), free: 60, total: 580, max: 672
2016-01-14 15:22:01,695 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,695 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,696 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,696 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,696 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,697 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,698 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,700 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, startTime=1452784921612, finishTime=1452784921699, timeTaken=87, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:01,700 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,700 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,700 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:01,701 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,701 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:01,701 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,701 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,701 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,702 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskId=task_1452784920806_0001_1_03_000000, startTime=1452784921612, finishTime=1452784921701, timeTaken=89, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:01,702 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,703 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,703 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] : 1
2016-01-14 15:22:01,703 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,705 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,706 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, vertexId=vertex_1452784920806_0001_1_03, initRequestedTime=1452784921056, initedTime=1452784921073, startRequestedTime=1452784921092, startedTime=1452784921092, finishTime=1452784921704, timeTaken=612, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784921612, firstTasksToStart=[ task_1452784920806_0001_1_03_000000 ], lastTaskFinishTime=1452784921699, lastTasksToFinish=[ task_1452784920806_0001_1_03_000000 ], minTaskDuration=87, maxTaskDuration=87, avgTaskDuration=87.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_03_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,706 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,707 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,707 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,707 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000004, stoppedTime=1452784921707, exitStatus=0
2016-01-14 15:22:01,708 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,708 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,708 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,709 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784920806_0001_1
2016-01-14 15:22:01,709 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: AA5BC7EE4A07438280C31731A446F7FB for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,711 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,712 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,712 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000004, Exiting
2016-01-14 15:22:01,712 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,712 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,713 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,713 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000004 completed successfully
2016-01-14 15:22:01,723 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,723 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,723 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,728 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784920806_0001_1, queueSize=0
2016-01-14 15:22:01,728 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:01,730 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784920806_0001_1, startTime=1452784921025, finishTime=1452784921723, timeTaken=698, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14417, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2299527168, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=42, FIRST_EVENT_RECEIVED=23, LAST_EVENT_RECEIVED=23, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139685750, Process_Duration=77, Process_End_Time=5811139685827, Read_Duration=20, Tuples_Read=303, Tuples_Written=317, Write_Duration=16, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:01,730 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784920806_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:01,731 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:01,731 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784920806_0001_00_000004
2016-01-14 15:22:01,732 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01 Completed Dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,733 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784920806_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:01,734 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784920806_0001_1_post
2016-01-14 15:22:01,734 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,735 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:01,740 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001
2016-01-14 15:22:01,742 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:01,742 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:01,742 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:01,743 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:01,743 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:01,744 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:01,744 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:01,745 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:01,745 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:01,745 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:01,745 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:01,746 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:01,746 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:01,752 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001
2016-01-14 15:22:01,754 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  completed in: 00:00.968
2016-01-14 15:22:01,862 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 15:22:01,893 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:01,894 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:01,975 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.053
2016-01-14 15:22:02,007 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.085
2016-01-14 15:22:02,008 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:02,009 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:02,009 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:02,030 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:02,056 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 15:22:02,198 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:22:02,198 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,199 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 15:22:02,199 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 15:22:02,200 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 15:22:02,200 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 15:22:02,201 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,201 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:02,215 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:22:02,220 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:02,221 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:02,221 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:02,231 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001 doesn't exist and is created
2016-01-14 15:22:02,249 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:02,249 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:02,250 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001
2016-01-14 15:22:02,253 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784922220_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:02,266 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:02,267 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:02,269 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:02,269 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:02,269 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir
2016-01-14 15:22:02,270 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/history.txt.appattempt_1452784922220_0001_000000, maxErrors=10
2016-01-14 15:22:02,270 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:02,270 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784922220_0001_000000, appSubmitTime=1452784922253, launchTime=1452784922253
2016-01-14 15:22:02,278 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:02,282 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:02,283 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:02,283 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:02,287 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:37104
2016-01-14 15:22:02,291 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,294 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784922220_0001_000000, startTime=1452784922294
2016-01-14 15:22:02,294 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:02,295 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:02,349 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:02,350 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:02,350 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001, dagName=[BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,371 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,374 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784922220_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/dag_1452784922220_0001_1.dot
2016-01-14 15:22:02,376 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/dag_1452784922220_0001_1-tez-dag.pb.txt
2016-01-14 15:22:02,406 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784922220_0001_1
2016-01-14 15:22:02,407 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,407 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02 Running Dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,422 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784922220_0001_1, submitTime=1452784922371
2016-01-14 15:22:02,422 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:02,424 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,424 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:02,424 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784922220_0001_1, initTime=1452784922422
2016-01-14 15:22:02,424 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:02,425 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:02,427 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_STARTED]: dagID=dag_1452784922220_0001_1, startTime=1452784922426
2016-01-14 15:22:02,427 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:02,427 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:02,427 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:02,427 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: AFDC1534F21B447DA4FFF2514883B001 : {812ABCEB2A5C4C0D9BF844260CD28C87={InputName=812ABCEB2A5C4C0D9BF844260CD28C87}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:02,427 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 812ABCEB2A5C4C0D9BF844260CD28C87, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:02,428 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]. Starting root input initializers: 1
2016-01-14 15:22:02,429 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:02,429 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,431 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001, dagName=[BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,429 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 812ABCEB2A5C4C0D9BF844260CD28C87 on vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,436 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:02,437 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 812ABCEB2A5C4C0D9BF844260CD28C87 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:02,439 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,441 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,441 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,442 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,442 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CF650B439E834FF6836A619C942F1448, vertexId=vertex_1452784922220_0001_1_01, initRequestedTime=1452784922429, initedTime=1452784922442, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,443 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,444 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,444 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:02,447 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:02,450 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,450 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,451 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:02,451 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,452 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 812ABCEB2A5C4C0D9BF844260CD28C87 on vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,453 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,454 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, vertexId=vertex_1452784922220_0001_1_02, initRequestedTime=1452784922444, initedTime=1452784922452, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,454 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,455 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:02,455 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {812ABCEB2A5C4C0D9BF844260CD28C87=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:02,456 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] parallelism set to 1
2016-01-14 15:22:02,457 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:02,457 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_00, eventCount=1
2016-01-14 15:22:02,458 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784922220_0001, with dag id: dag_1452784922220_0001_1
2016-01-14 15:22:02,459 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,460 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, vertexId=vertex_1452784922220_0001_1_00, initRequestedTime=1452784922427, initedTime=1452784922458, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:02,460 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] with distanceFromRoot: 0
2016-01-14 15:22:02,461 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:02,461 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,464 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,465 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,465 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,465 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,466 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:02,466 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,468 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexName=B1158DCDB6CE43E7A3D082659164EA2A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:02,470 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,472 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,472 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, vertexId=vertex_1452784922220_0001_1_03, initRequestedTime=1452784922461, initedTime=1452784922472, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,473 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,473 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,474 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,475 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_00, startRequestedTime=1452784922473, startedTime=1452784922473
2016-01-14 15:22:02,475 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,475 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,476 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskId=task_1452784922220_0001_1_00_000000, scheduledTime=1452784922475, launchTime=1452784922475
2016-01-14 15:22:02,476 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,476 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_00 for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,476 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,478 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] with distanceFromRoot: 1
2016-01-14 15:22:02,479 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,479 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: AFDC1534F21B447DA4FFF2514883B001 in vertex: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,480 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: CF650B439E834FF6836A619C942F1448 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,480 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_01, startRequestedTime=1452784922479, startedTime=1452784922479
2016-01-14 15:22:02,480 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,480 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,480 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: AFDC1534F21B447DA4FFF2514883B001, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,482 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,482 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_01 for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,486 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] with distanceFromRoot: 2
2016-01-14 15:22:02,486 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,486 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: CF650B439E834FF6836A619C942F1448 in vertex: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,487 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D4F1C81073004F1EA40841A2F7C36F50 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,488 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:02,490 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,490 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_02, startRequestedTime=1452784922486, startedTime=1452784922486
2016-01-14 15:22:02,492 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,493 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:02,494 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:02,495 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000001
2016-01-14 15:22:02,495 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,496 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_02 for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,496 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] with distanceFromRoot: 3
2016-01-14 15:22:02,498 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784922220_0001_00_000001
2016-01-14 15:22:02,498 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:02,498 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,499 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000001, launchTime=1452784922498
2016-01-14 15:22:02,500 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,500 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D4F1C81073004F1EA40841A2F7C36F50 in vertex: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,500 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B1158DCDB6CE43E7A3D082659164EA2A with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,500 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,501 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_03, startRequestedTime=1452784922500, startedTime=1452784922500
2016-01-14 15:22:02,501 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,501 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,510 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_00_000000_0] to container: [container_1452784922220_0001_00_000001]
2016-01-14 15:22:02,511 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,511 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000001 given task: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,511 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,511 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_00_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,512 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,512 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,512 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,512 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,512 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,512 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,513 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,513 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: AFDC1534F21B447DA4FFF2514883B001, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,513 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,513 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,513 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,513 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, startTime=1452784922511, containerId=container_1452784922220_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:02,513 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,514 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,515 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:02,514 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,515 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,515 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,517 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:02,518 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,518 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,525 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,525 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,525 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,526 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:02,526 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,526 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,527 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,527 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,529 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,529 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,530 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,534 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, ordinal: 0
2016-01-14 15:22:02,534 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, mem on start (mb), free: 134, total: 570, max: 672
2016-01-14 15:22:02,534 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,534 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,535 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,535 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,535 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,535 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:02,536 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:02,536 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,536 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,536 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:02,536 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 812ABCEB2A5C4C0D9BF844260CD28C87, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:02,537 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, CF650B439E834FF6836A619C942F1448, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:02,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 812ABCEB2A5C4C0D9BF844260CD28C87 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,537 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:02,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,538 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,538 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,538 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,538 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:02,539 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@5b7e588b
2016-01-14 15:22:02,540 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@5b7e588b
2016-01-14 15:22:02,540 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:02,541 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:02,541 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,542 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,542 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,542 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,543 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6fd85a65; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6516d640; comparator=cascading.tuple.hadoop.util.TupleComparator@3e4c3799; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,544 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,544 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,544 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,580 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:02,581 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:02,581 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:02,581 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:02,581 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:02,582 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,582 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:02,582 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,586 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,586 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, mem on close (mb), free: 161, total: 580, max: 672
2016-01-14 15:22:02,587 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,587 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,587 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:02,587 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:02,590 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,594 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,595 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_00, eventCount=1
2016-01-14 15:22:02,596 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, startTime=1452784922511, finishTime=1452784922595, timeTaken=84, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:02,596 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,596 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,596 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,596 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,598 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskId=task_1452784922220_0001_1_00_000000, startTime=1452784922511, finishTime=1452784922596, timeTaken=85, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:02,598 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,599 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] : 1
2016-01-14 15:22:02,599 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,599 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:02,599 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:02,600 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,600 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,600 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,602 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, vertexId=vertex_1452784922220_0001_1_00, initRequestedTime=1452784922427, initedTime=1452784922458, startRequestedTime=1452784922473, startedTime=1452784922473, finishTime=1452784922599, timeTaken=126, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784922511, firstTasksToStart=[ task_1452784922220_0001_1_00_000000 ], lastTaskFinishTime=1452784922595, lastTasksToFinish=[ task_1452784922220_0001_1_00_000000 ], minTaskDuration=84, maxTaskDuration=84, avgTaskDuration=84.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_00_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,603 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,603 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,603 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,605 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000001, stoppedTime=1452784922605, exitStatus=0
2016-01-14 15:22:02,605 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,606 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] attempt: attempt_1452784922220_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,606 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,607 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,608 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,608 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,609 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,609 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,609 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=CF650B439E834FF6836A619C942F1448, taskId=task_1452784922220_0001_1_01_000000, scheduledTime=1452784922609, launchTime=1452784922609
2016-01-14 15:22:02,610 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,610 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,610 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,611 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: CF650B439E834FF6836A619C942F1448, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,611 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,611 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,611 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,612 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,612 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000001, Exiting
2016-01-14 15:22:02,612 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,612 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,619 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000002
2016-01-14 15:22:02,620 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,623 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784922220_0001_00_000002
2016-01-14 15:22:02,624 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,629 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,629 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000001 completed successfully
2016-01-14 15:22:02,629 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,629 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000002, launchTime=1452784922629
2016-01-14 15:22:02,630 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,630 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,632 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,632 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_01_000000_0] to container: [container_1452784922220_0001_00_000002]
2016-01-14 15:22:02,633 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,634 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_01_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,634 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CF650B439E834FF6836A619C942F1448, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, startTime=1452784922633, containerId=container_1452784922220_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:02,634 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,634 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,635 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,637 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000002 given task: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,637 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,638 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,638 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,638 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,638 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,638 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,638 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,639 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,639 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: CF650B439E834FF6836A619C942F1448, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,639 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,641 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,642 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,642 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,642 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,645 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,646 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,647 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,648 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,650 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,650 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,650 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,647 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,650 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,650 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,651 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,664 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CF650B439E834FF6836A619C942F1448, ordinal: 1
2016-01-14 15:22:02,664 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF650B439E834FF6836A619C942F1448, mem on start (mb), free: 154, total: 580, max: 672
2016-01-14 15:22:02,664 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,664 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,664 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,664 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,664 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,665 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,667 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,667 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,668 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:02,668 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D4F1C81073004F1EA40841A2F7C36F50, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:02,668 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, AFDC1534F21B447DA4FFF2514883B001, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:02,668 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,669 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: AFDC1534F21B447DA4FFF2514883B001 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,669 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,669 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,670 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,671 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,672 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,672 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: AFDC1534F21B447DA4FFF2514883B001 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,672 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:02,672 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,673 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,674 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,674 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,674 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,674 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_00_000000_0_10027, runDuration: 65666, ]
2016-01-14 15:22:02,677 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_00_000000_0_10027] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784922677, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:02,678 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,678 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 15:22:02,678 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:02,679 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [AFDC1534F21B447DA4FFF2514883B001] #7 in 3ms
2016-01-14 15:22:02,679 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [AFDC1534F21B447DA4FFF2514883B001] #7, status:false, isInterrupted:false
2016-01-14 15:22:02,679 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,679 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,680 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,680 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,681 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:02,681 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,681 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,681 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:02,681 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,680 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@2f0d2baf; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@9af064b; comparator=cascading.tuple.hadoop.util.TupleComparator@41c62ae4; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,684 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,685 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,685 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,682 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,693 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:02,693 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:02,694 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:02,694 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:02,694 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:02,694 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 8AEE460591F24356B3FFAF9C3C396535, for 1 inputs
2016-01-14 15:22:02,695 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:02,695 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:02,695 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CF650B439E834FF6836A619C942F1448, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,696 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@13d730fc; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@14d0fa91
2016-01-14 15:22:02,724 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,724 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF650B439E834FF6836A619C942F1448, mem on close (mb), free: 87, total: 580, max: 672
2016-01-14 15:22:02,724 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,724 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,725 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,725 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,725 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,725 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02,725 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,725 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,725 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:02,726 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:02,738 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,743 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,744 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_01, eventCount=1
2016-01-14 15:22:02,745 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, startTime=1452784922633, finishTime=1452784922745, timeTaken=112, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 15:22:02,745 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,746 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,745 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-01-14 15:22:02,747 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,747 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:02,748 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,748 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,746 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,749 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,749 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, taskId=task_1452784922220_0001_1_01_000000, startTime=1452784922633, finishTime=1452784922748, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 15:22:02,749 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,750 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] : 1
2016-01-14 15:22:02,750 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,751 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,751 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, vertexId=vertex_1452784922220_0001_1_01, initRequestedTime=1452784922429, initedTime=1452784922442, startRequestedTime=1452784922479, startedTime=1452784922479, finishTime=1452784922750, timeTaken=271, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1452784922633, firstTasksToStart=[ task_1452784922220_0001_1_01_000000 ], lastTaskFinishTime=1452784922745, lastTasksToFinish=[ task_1452784922220_0001_1_01_000000 ], minTaskDuration=112, maxTaskDuration=112, avgTaskDuration=112.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_01_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,752 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,752 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,752 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000002, stoppedTime=1452784922752, exitStatus=0
2016-01-14 15:22:02,752 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,752 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,753 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] attempt: attempt_1452784922220_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,753 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,753 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,753 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,753 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,754 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskId=task_1452784922220_0001_1_02_000000, scheduledTime=1452784922754, launchTime=1452784922754
2016-01-14 15:22:02,754 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,754 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,754 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,755 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,755 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D4F1C81073004F1EA40841A2F7C36F50, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,755 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,756 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,756 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000003
2016-01-14 15:22:02,757 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,759 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784922220_0001_00_000003
2016-01-14 15:22:02,759 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - TezTask metrics system already initialized!
2016-01-14 15:22:02,760 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,760 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,760 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,760 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,760 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000002, Exiting
2016-01-14 15:22:02,760 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,761 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,764 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,764 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000002 completed successfully
2016-01-14 15:22:02,764 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,764 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000003, launchTime=1452784922760
2016-01-14 15:22:02,765 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,766 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_02_000000_0] to container: [container_1452784922220_0001_00_000003]
2016-01-14 15:22:02,766 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,767 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000003 given task: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,767 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,767 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_02_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,768 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,769 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, startTime=1452784922767, containerId=container_1452784922220_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:02,769 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,769 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,770 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,770 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,770 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,770 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,770 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,771 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,771 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,771 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D4F1C81073004F1EA40841A2F7C36F50, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,771 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,778 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,778 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,778 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,780 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,781 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,782 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,783 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,783 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,783 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,784 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,785 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,785 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,786 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,786 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,781 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:02,788 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,788 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,789 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,789 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,799 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, ordinal: 2
2016-01-14 15:22:02,799 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, mem on start (mb), free: 81, total: 580, max: 672
2016-01-14 15:22:02,799 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,799 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,800 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,801 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,801 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,801 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,802 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:02,802 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B1158DCDB6CE43E7A3D082659164EA2A, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:02,802 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CF650B439E834FF6836A619C942F1448, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:02,802 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,803 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CF650B439E834FF6836A619C942F1448 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,803 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,803 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,804 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,807 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,807 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,809 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: CF650B439E834FF6836A619C942F1448 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,809 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:02,809 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,810 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,811 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,812 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,812 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,814 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_01_000000_0_10030, runDuration: 92362, ]
2016-01-14 15:22:02,815 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:22:02,816 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,817 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,817 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_01_000000_0_10030] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784922817, TimeTaken=2, Rate=3.59 MB/s
2016-01-14 15:22:02,817 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,817 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:02,817 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [CF650B439E834FF6836A619C942F1448] #8 in 3ms
2016-01-14 15:22:02,817 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,821 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@164657cc; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@642e1d80; comparator=cascading.tuple.hadoop.util.TupleComparator@f92f1e2; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,822 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [CF650B439E834FF6836A619C942F1448] #8, status:false, isInterrupted:false
2016-01-14 15:22:02,822 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,824 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,825 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:02,825 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,825 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,825 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:02,826 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,826 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,826 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,835 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,844 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:02,844 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:02,844 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:02,844 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:02,845 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:02,846 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 6CA27EF2C8F0401681041459C024470F, for 1 inputs
2016-01-14 15:22:02,847 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,847 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@120a2a74; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@59692eec
2016-01-14 15:22:02,867 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,867 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, mem on close (mb), free: 201, total: 580, max: 672
2016-01-14 15:22:02,867 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,868 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,868 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,868 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,868 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,869 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02,869 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,869 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,869 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 15:22:02,869 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:02,874 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,880 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,882 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_02, eventCount=1
2016-01-14 15:22:02,882 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 15:22:02,883 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,883 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:02,884 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,884 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, startTime=1452784922767, finishTime=1452784922883, timeTaken=116, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:22:02,884 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,885 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,885 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,885 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,885 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,886 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskId=task_1452784922220_0001_1_02_000000, startTime=1452784922767, finishTime=1452784922885, timeTaken=118, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:22:02,886 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,886 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] : 1
2016-01-14 15:22:02,886 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,887 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,888 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, vertexId=vertex_1452784922220_0001_1_02, initRequestedTime=1452784922444, initedTime=1452784922452, startRequestedTime=1452784922486, startedTime=1452784922486, finishTime=1452784922886, timeTaken=400, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452784922767, firstTasksToStart=[ task_1452784922220_0001_1_02_000000 ], lastTaskFinishTime=1452784922883, lastTasksToFinish=[ task_1452784922220_0001_1_02_000000 ], minTaskDuration=116, maxTaskDuration=116, avgTaskDuration=116.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_02_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,888 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,888 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,889 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000003, stoppedTime=1452784922888, exitStatus=0
2016-01-14 15:22:02,889 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,889 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,889 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] attempt: attempt_1452784922220_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,889 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,889 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,890 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,890 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,891 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,891 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskId=task_1452784922220_0001_1_03_000000, scheduledTime=1452784922890, launchTime=1452784922890
2016-01-14 15:22:02,891 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,891 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,892 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,892 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B1158DCDB6CE43E7A3D082659164EA2A, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:02,893 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,892 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,893 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,893 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,894 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000004
2016-01-14 15:22:02,894 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000003, Exiting
2016-01-14 15:22:02,895 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,895 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,895 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,896 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,896 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000003 completed successfully
2016-01-14 15:22:02,896 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,897 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784922220_0001_00_000004
2016-01-14 15:22:02,898 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,900 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000004, launchTime=1452784922900
2016-01-14 15:22:02,900 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,900 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,901 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,902 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_03_000000_0] to container: [container_1452784922220_0001_00_000004]
2016-01-14 15:22:02,902 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,903 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000004 given task: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,903 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_03_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,903 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,903 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,903 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, startTime=1452784922903, containerId=container_1452784922220_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,904 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,904 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,904 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,904 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,904 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,905 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,905 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,905 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,905 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,905 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B1158DCDB6CE43E7A3D082659164EA2A, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:02,905 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,906 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,908 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:02,908 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,909 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,911 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,911 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:02,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,913 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,913 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,913 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,913 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,914 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,914 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,915 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:02,915 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,915 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,928 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:02,928 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,929 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, ordinal: 3
2016-01-14 15:22:02,929 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, mem on start (mb), free: 194, total: 580, max: 672
2016-01-14 15:22:02,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,930 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,930 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,930 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:02,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:02,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:02,932 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D4F1C81073004F1EA40841A2F7C36F50, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:02,932 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 8F3D3AC0D03D441AAB2B30771DE4FED8, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:02,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D4F1C81073004F1EA40841A2F7C36F50 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,933 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,933 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,933 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D4F1C81073004F1EA40841A2F7C36F50 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,934 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:02,934 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,934 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,934 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,935 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,936 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,936 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_02_000000_0_10033, runDuration: 94648, ]
2016-01-14 15:22:02,939 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_02_000000_0_10033] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452784922939, TimeTaken=2, Rate=2.44 MB/s
2016-01-14 15:22:02,939 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,939 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:02,939 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D4F1C81073004F1EA40841A2F7C36F50] #9 in 2ms
2016-01-14 15:22:02,939 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D4F1C81073004F1EA40841A2F7C36F50] #9, status:false, isInterrupted:false
2016-01-14 15:22:02,940 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 15:22:02,942 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,942 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 15:22:02,943 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,943 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,943 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,944 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,944 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 15:22:02,944 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 15:22:02,944 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 654C236BC9EE44D88920076030074B14, for 1 inputs
2016-01-14 15:22:02,945 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,945 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,945 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,946 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@5ac92449; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4b6d632f
2016-01-14 15:22:02,956 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:02,956 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:02,957 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,957 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784922220_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:02,958 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,958 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, mem on close (mb), free: 191, total: 580, max: 672
2016-01-14 15:22:02,958 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,959 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,959 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,959 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,960 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,960 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,961 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,962 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:02,962 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,963 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:02,963 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,963 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, startTime=1452784922903, finishTime=1452784922962, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:02,963 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,963 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,964 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,964 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,964 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,964 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskId=task_1452784922220_0001_1_03_000000, startTime=1452784922903, finishTime=1452784922964, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:02,965 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,965 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] : 1
2016-01-14 15:22:02,965 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,966 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, vertexId=vertex_1452784922220_0001_1_03, initRequestedTime=1452784922461, initedTime=1452784922472, startRequestedTime=1452784922500, startedTime=1452784922500, finishTime=1452784922965, timeTaken=465, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784922903, firstTasksToStart=[ task_1452784922220_0001_1_03_000000 ], lastTaskFinishTime=1452784922962, lastTasksToFinish=[ task_1452784922220_0001_1_03_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_03_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,966 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,966 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,967 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,969 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000004, stoppedTime=1452784922969, exitStatus=0
2016-01-14 15:22:02,969 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,969 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,969 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,969 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,970 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784922220_0001_1
2016-01-14 15:22:02,970 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 8F3D3AC0D03D441AAB2B30771DE4FED8 for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,974 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,974 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,974 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000004, Exiting
2016-01-14 15:22:02,974 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,974 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,975 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,975 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000004 completed successfully
2016-01-14 15:22:02,980 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,980 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,981 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,987 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784922220_0001_1, queueSize=0
2016-01-14 15:22:02,988 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:02,989 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784922220_0001_1, startTime=1452784922426, finishTime=1452784922981, timeTaken=555, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=31, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2433744896, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139691070, Process_Duration=65, Process_End_Time=5811139691135, Read_Duration=13, Tuples_Read=303, Tuples_Written=317, Write_Duration=16, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:02,989 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784922220_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:02,990 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:02,991 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784922220_0001_00_000004
2016-01-14 15:22:02,992 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02 Completed Dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,995 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784922220_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:02,996 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784922220_0001_1_post
2016-01-14 15:22:02,996 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,997 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:03,001 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001
2016-01-14 15:22:03,003 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:03,003 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:03,003 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:03,003 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:03,003 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:03,004 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:03,004 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:03,005 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:03,005 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:03,005 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:03,006 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:03,007 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:03,007 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:03,014 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001
2016-01-14 15:22:03,014 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:00.816
2016-01-14 15:22:03,092 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 15:22:03,123 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:03,124 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:03,200 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.050
2016-01-14 15:22:03,220 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.070
2016-01-14 15:22:03,221 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:03,221 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:03,221 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:03,242 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:03,264 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-01-14 15:22:03,403 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:22:03,404 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,404 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 15:22:03,404 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 15:22:03,405 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 15:22:03,405 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 15:22:03,406 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,406 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:03,418 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:22:03,423 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:03,424 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:03,424 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:03,433 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001 doesn't exist and is created
2016-01-14 15:22:03,448 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:03,448 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:03,449 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001
2016-01-14 15:22:03,453 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784923423_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:03,464 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:03,465 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:03,466 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:03,467 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:03,467 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir
2016-01-14 15:22:03,467 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/history.txt.appattempt_1452784923423_0001_000000, maxErrors=10
2016-01-14 15:22:03,467 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:03,468 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784923423_0001_000000, appSubmitTime=1452784923452, launchTime=1452784923452
2016-01-14 15:22:03,472 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:03,473 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:03,474 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:03,476 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:03,478 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:57866
2016-01-14 15:22:03,480 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:03,482 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784923423_0001_000000, startTime=1452784923482
2016-01-14 15:22:03,482 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:03,482 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:03,549 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:03,549 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:03,549 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001, dagName=[C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,566 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,568 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784923423_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/dag_1452784923423_0001_1.dot
2016-01-14 15:22:03,570 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/dag_1452784923423_0001_1-tez-dag.pb.txt
2016-01-14 15:22:03,610 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784923423_0001_1
2016-01-14 15:22:03,610 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,610 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03 Running Dag: dag_1452784923423_0001_1
2016-01-14 15:22:03,623 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784923423_0001_1, submitTime=1452784923566
2016-01-14 15:22:03,623 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:03,625 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,625 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:03,625 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784923423_0001_1, initTime=1452784923623
2016-01-14 15:22:03,625 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:03,626 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:03,627 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_STARTED]: dagID=dag_1452784923423_0001_1, startTime=1452784923627
2016-01-14 15:22:03,627 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:03,627 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:03,627 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:03,627 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: CDC852A364E64A83B81E0ED2B2C93103 : {333A4BC1528E493D9A03C012238A7AEC={InputName=333A4BC1528E493D9A03C012238A7AEC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:03,627 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 333A4BC1528E493D9A03C012238A7AEC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:03,628 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]. Starting root input initializers: 1
2016-01-14 15:22:03,629 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:03,629 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,630 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001, dagName=[C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,633 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 333A4BC1528E493D9A03C012238A7AEC on vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,634 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,634 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,635 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,635 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,636 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B969A35B407643359B20905A4B079125, vertexId=vertex_1452784923423_0001_1_01, initRequestedTime=1452784923629, initedTime=1452784923635, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,636 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,636 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,641 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:03,641 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 333A4BC1528E493D9A03C012238A7AEC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:03,642 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:03,643 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,643 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:03,644 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,644 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,645 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,645 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, vertexId=vertex_1452784923423_0001_1_02, initRequestedTime=1452784923636, initedTime=1452784923644, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,645 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,645 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,649 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,649 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:03,651 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,652 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784923423_0001, with dag id: dag_1452784923423_0001_1
2016-01-14 15:22:03,650 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:03,652 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=66B5A676F67342598BB0CC462F2FC64C, vertexName=C55DC0153409441F8FC29EAB74C3F340, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:03,658 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,657 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 333A4BC1528E493D9A03C012238A7AEC on vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,659 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, vertexId=vertex_1452784923423_0001_1_03, initRequestedTime=1452784923645, initedTime=1452784923659, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,660 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,659 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,660 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:03,660 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {333A4BC1528E493D9A03C012238A7AEC=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:03,661 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] parallelism set to 1
2016-01-14 15:22:03,663 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:03,664 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_00, eventCount=1
2016-01-14 15:22:03,665 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, vertexId=vertex_1452784923423_0001_1_00, initRequestedTime=1452784923627, initedTime=1452784923665, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:03,665 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] with distanceFromRoot: 0
2016-01-14 15:22:03,666 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:03,665 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,666 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,667 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,667 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,667 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_00, startRequestedTime=1452784923666, startedTime=1452784923666
2016-01-14 15:22:03,668 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,668 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,668 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskId=task_1452784923423_0001_1_00_000000, scheduledTime=1452784923668, launchTime=1452784923668
2016-01-14 15:22:03,669 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,669 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_00 for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,669 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] with distanceFromRoot: 1
2016-01-14 15:22:03,670 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,670 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: CDC852A364E64A83B81E0ED2B2C93103 in vertex: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,670 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B969A35B407643359B20905A4B079125 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,671 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_01, startRequestedTime=1452784923670, startedTime=1452784923670
2016-01-14 15:22:03,671 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,672 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,672 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: CDC852A364E64A83B81E0ED2B2C93103, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,673 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,673 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_01 for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,673 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] with distanceFromRoot: 2
2016-01-14 15:22:03,674 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,674 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B969A35B407643359B20905A4B079125 in vertex: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,674 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:03,674 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: FACEDF332B814CD28C9FB990F959C4C3 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,675 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,676 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_02, startRequestedTime=1452784923674, startedTime=1452784923674
2016-01-14 15:22:03,676 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,676 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:03,677 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:03,677 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000001
2016-01-14 15:22:03,678 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,678 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_02 for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,678 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] with distanceFromRoot: 3
2016-01-14 15:22:03,681 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784923423_0001_00_000001
2016-01-14 15:22:03,681 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:03,682 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,682 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000001, launchTime=1452784923682
2016-01-14 15:22:03,682 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: FACEDF332B814CD28C9FB990F959C4C3 in vertex: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,682 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,683 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C55DC0153409441F8FC29EAB74C3F340 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,683 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_03, startRequestedTime=1452784923682, startedTime=1452784923682
2016-01-14 15:22:03,683 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,684 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,684 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,693 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_00_000000_0] to container: [container_1452784923423_0001_00_000001]
2016-01-14 15:22:03,694 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,694 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000001 given task: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,694 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,694 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_00_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,695 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,695 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,695 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,695 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, startTime=1452784923694, containerId=container_1452784923423_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:03,695 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,695 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,695 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,696 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,696 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,696 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,696 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,697 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: CDC852A364E64A83B81E0ED2B2C93103, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,697 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,698 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,700 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,700 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,699 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:03,702 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,702 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,703 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:03,705 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,703 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,706 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,706 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,707 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,708 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,708 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,708 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,710 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,712 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:03,712 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,712 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,714 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, ordinal: 0
2016-01-14 15:22:03,714 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, mem on start (mb), free: 63, total: 580, max: 672
2016-01-14 15:22:03,714 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,714 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,715 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,715 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,715 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,716 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:03,716 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:03,716 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,717 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,717 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:03,718 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 333A4BC1528E493D9A03C012238A7AEC, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:03,718 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B969A35B407643359B20905A4B079125, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:03,718 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,718 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 333A4BC1528E493D9A03C012238A7AEC being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,719 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,720 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,719 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,721 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:03,721 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,722 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,722 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,725 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:03,726 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@7b34c936
2016-01-14 15:22:03,726 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,726 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,727 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,727 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,727 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@7b34c936
2016-01-14 15:22:03,727 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:03,727 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:03,728 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@699dad28; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6ed35343; comparator=cascading.tuple.hadoop.util.TupleComparator@73ff8c8c; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,728 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,728 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,729 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,753 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:03,754 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:03,754 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:03,754 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:03,754 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:03,755 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,755 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:03,756 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,760 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,760 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, mem on close (mb), free: 160, total: 580, max: 672
2016-01-14 15:22:03,760 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,761 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,761 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:03,761 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:03,764 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,768 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,768 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:03,769 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:03,769 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,769 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_00, eventCount=1
2016-01-14 15:22:03,769 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,770 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,770 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, startTime=1452784923694, finishTime=1452784923770, timeTaken=76, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:03,770 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,770 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,770 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,771 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,771 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskId=task_1452784923423_0001_1_00_000000, startTime=1452784923694, finishTime=1452784923770, timeTaken=76, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:03,771 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,771 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] : 1
2016-01-14 15:22:03,772 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,773 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:03,773 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, vertexId=vertex_1452784923423_0001_1_00, initRequestedTime=1452784923627, initedTime=1452784923665, startRequestedTime=1452784923666, startedTime=1452784923666, finishTime=1452784923772, timeTaken=106, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784923694, firstTasksToStart=[ task_1452784923423_0001_1_00_000000 ], lastTaskFinishTime=1452784923770, lastTasksToFinish=[ task_1452784923423_0001_1_00_000000 ], minTaskDuration=76, maxTaskDuration=76, avgTaskDuration=76.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_00_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:03,774 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:03,774 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:03,775 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] attempt: attempt_1452784923423_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:03,776 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,776 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,776 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000001, stoppedTime=1452784923776, exitStatus=0
2016-01-14 15:22:03,777 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:03,776 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:03,778 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:03,779 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,779 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=B969A35B407643359B20905A4B079125, taskId=task_1452784923423_0001_1_01_000000, scheduledTime=1452784923778, launchTime=1452784923778
2016-01-14 15:22:03,779 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,780 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:03,780 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,780 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:03,780 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:03,781 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:03,781 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: B969A35B407643359B20905A4B079125, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,781 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000001, Exiting
2016-01-14 15:22:03,781 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,781 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,781 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,783 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,783 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:03,784 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000001 completed successfully
2016-01-14 15:22:03,784 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000002
2016-01-14 15:22:03,784 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,784 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:03,786 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784923423_0001_00_000002
2016-01-14 15:22:03,787 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:03,789 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,789 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000002, launchTime=1452784923789
2016-01-14 15:22:03,790 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,789 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,794 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_01_000000_0] to container: [container_1452784923423_0001_00_000002]
2016-01-14 15:22:03,794 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,796 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000002 given task: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,796 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,796 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_01_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,797 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B969A35B407643359B20905A4B079125, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, startTime=1452784923796, containerId=container_1452784923423_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:03,797 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,797 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,797 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,797 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,797 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,797 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,798 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,798 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,798 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,798 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,798 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: B969A35B407643359B20905A4B079125, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,798 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,800 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:03,800 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,800 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,801 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:03,802 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,802 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,803 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,803 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,804 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,804 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,805 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,805 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,806 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,806 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,811 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:03,811 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:03,811 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,813 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B969A35B407643359B20905A4B079125, ordinal: 1
2016-01-14 15:22:03,814 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B969A35B407643359B20905A4B079125, mem on start (mb), free: 153, total: 580, max: 672
2016-01-14 15:22:03,814 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,814 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,814 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,814 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,814 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,815 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,815 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,815 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:03,815 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,815 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:03,816 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, FACEDF332B814CD28C9FB990F959C4C3, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:03,816 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CDC852A364E64A83B81E0ED2B2C93103, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:03,816 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,817 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CDC852A364E64A83B81E0ED2B2C93103 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,817 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,817 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,817 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,823 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,823 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:03,824 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: CDC852A364E64A83B81E0ED2B2C93103 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:03,824 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:03,824 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:03,824 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:03,826 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,827 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,828 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,830 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_00_000000_0_10038, runDuration: 60176, ]
2016-01-14 15:22:03,831 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 15:22:03,832 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,832 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,832 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_00_000000_0_10038] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784923832, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:03,833 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,833 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,833 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:03,834 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@5955509e; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@701fe2e3; comparator=cascading.tuple.hadoop.util.TupleComparator@3d68de97; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,834 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,834 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,835 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,836 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [CDC852A364E64A83B81E0ED2B2C93103] #10 in 6ms
2016-01-14 15:22:03,837 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [CDC852A364E64A83B81E0ED2B2C93103] #10, status:false, isInterrupted:false
2016-01-14 15:22:03,844 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:03,844 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:03,844 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:03,845 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:03,845 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:03,845 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:03,844 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:03,846 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,846 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:03,846 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:03,847 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] D4ECE2BAEFAB4B4F80B2DA6C6AC43097, for 1 inputs
2016-01-14 15:22:03,847 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,847 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:03,847 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:03,848 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:03,848 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B969A35B407643359B20905A4B079125, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,849 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1f277ec9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@59f4ce19
2016-01-14 15:22:03,870 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,870 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B969A35B407643359B20905A4B079125, mem on close (mb), free: 87, total: 580, max: 672
2016-01-14 15:22:03,871 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,871 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:03,871 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:03,871 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,871 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:03,872 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:03,872 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,872 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,872 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:03,873 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:03,882 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,885 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,886 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_01, eventCount=1
2016-01-14 15:22:03,886 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4]]
2016-01-14 15:22:03,886 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,887 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, startTime=1452784923796, finishTime=1452784923886, timeTaken=90, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4
2016-01-14 15:22:03,887 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,887 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,887 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,888 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,888 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, taskId=task_1452784923423_0001_1_01_000000, startTime=1452784923796, finishTime=1452784923887, timeTaken=91, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4
2016-01-14 15:22:03,888 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,889 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:03,889 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,889 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] : 1
2016-01-14 15:22:03,889 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,890 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,890 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:03,891 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, vertexId=vertex_1452784923423_0001_1_01, initRequestedTime=1452784923629, initedTime=1452784923635, startRequestedTime=1452784923670, startedTime=1452784923670, finishTime=1452784923890, timeTaken=220, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4, vertexStats=firstTaskStartTime=1452784923796, firstTasksToStart=[ task_1452784923423_0001_1_01_000000 ], lastTaskFinishTime=1452784923886, lastTasksToFinish=[ task_1452784923423_0001_1_01_000000 ], minTaskDuration=90, maxTaskDuration=90, avgTaskDuration=90.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_01_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:03,891 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:03,891 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:03,891 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000002, stoppedTime=1452784923891, exitStatus=0
2016-01-14 15:22:03,891 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:03,891 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] attempt: attempt_1452784923423_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:03,892 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:03,892 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,892 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,892 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:03,892 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:03,892 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskId=task_1452784923423_0001_1_02_000000, scheduledTime=1452784923892, launchTime=1452784923892
2016-01-14 15:22:03,892 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,893 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,895 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,895 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:03,896 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: FACEDF332B814CD28C9FB990F959C4C3, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,896 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,897 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,897 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000003
2016-01-14 15:22:03,897 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,899 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784923423_0001_00_000003
2016-01-14 15:22:03,899 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - TezTask metrics system already initialized!
2016-01-14 15:22:03,900 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,900 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000003, launchTime=1452784923900
2016-01-14 15:22:03,900 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,900 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:03,900 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:03,901 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000002, Exiting
2016-01-14 15:22:03,901 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,901 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,901 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000003
2016-01-14 15:22:03,905 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:03,907 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000002 completed successfully
2016-01-14 15:22:03,907 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:03,907 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_02_000000_0] to container: [container_1452784923423_0001_00_000003]
2016-01-14 15:22:03,908 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,908 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000003 given task: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,908 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,909 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_02_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:03,909 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,910 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, startTime=1452784923909, containerId=container_1452784923423_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:03,911 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,911 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,911 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,911 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,911 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,911 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,912 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: FACEDF332B814CD28C9FB990F959C4C3, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,910 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,912 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,911 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,912 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,913 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,913 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,913 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,914 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,914 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:03,915 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:03,915 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,916 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,916 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,917 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,917 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,917 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,918 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:03,919 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:03,919 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,919 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,920 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,920 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,920 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,929 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, ordinal: 2
2016-01-14 15:22:03,929 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, mem on start (mb), free: 82, total: 580, max: 672
2016-01-14 15:22:03,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,930 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,930 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:03,930 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,931 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:03,931 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B969A35B407643359B20905A4B079125, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:03,931 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C55DC0153409441F8FC29EAB74C3F340, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:03,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B969A35B407643359B20905A4B079125 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,932 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,933 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,933 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,934 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:03,934 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B969A35B407643359B20905A4B079125 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:03,934 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:03,934 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:03,935 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:03,937 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,938 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,938 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,939 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_01_000000_0_10042, runDuration: 80035, ]
2016-01-14 15:22:03,941 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:22:03,942 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_01_000000_0_10042] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784923942, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:22:03,942 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,943 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:03,942 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,943 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [B969A35B407643359B20905A4B079125] #11 in 2ms
2016-01-14 15:22:03,943 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:03,944 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:03,944 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@46cb23b0; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@42941c1f; comparator=cascading.tuple.hadoop.util.TupleComparator@9c2595f; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,944 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,944 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,945 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,954 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B969A35B407643359B20905A4B079125] #11, status:false, isInterrupted:false
2016-01-14 15:22:03,961 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:03,961 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:03,961 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:03,962 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:03,962 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:03,962 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:03,963 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:03,963 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,963 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:03,963 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2797C146692A4F6D93EB14D62DF5DF9E, for 1 inputs
2016-01-14 15:22:03,964 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:03,964 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,964 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:03,964 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,965 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@213f379; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6bd58dff
2016-01-14 15:22:03,988 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,988 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, mem on close (mb), free: 183, total: 564, max: 672
2016-01-14 15:22:03,988 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,988 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:03,989 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:03,989 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,989 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:03,989 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:03,989 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,989 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,990 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 15:22:03,990 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:03,993 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,997 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,997 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_02, eventCount=1
2016-01-14 15:22:03,997 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 15:22:03,998 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:03,998 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,998 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,998 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, startTime=1452784923909, finishTime=1452784923998, timeTaken=89, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:03,998 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000003
2016-01-14 15:22:03,998 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,999 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,999 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,999 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,999 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskId=task_1452784923423_0001_1_02_000000, startTime=1452784923909, finishTime=1452784923999, timeTaken=90, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:03,999 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,999 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] : 1
2016-01-14 15:22:04,000 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:04,000 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:04,000 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, vertexId=vertex_1452784923423_0001_1_02, initRequestedTime=1452784923636, initedTime=1452784923644, startRequestedTime=1452784923674, startedTime=1452784923674, finishTime=1452784924000, timeTaken=326, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452784923909, firstTasksToStart=[ task_1452784923423_0001_1_02_000000 ], lastTaskFinishTime=1452784923998, lastTasksToFinish=[ task_1452784923423_0001_1_02_000000 ], minTaskDuration=89, maxTaskDuration=89, avgTaskDuration=89.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_02_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:04,001 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:04,001 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:04,001 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000003, stoppedTime=1452784924001, exitStatus=0
2016-01-14 15:22:04,001 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:04,001 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:04,001 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] attempt: attempt_1452784923423_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:04,002 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:04,002 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,002 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:04,002 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:04,002 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskId=task_1452784923423_0001_1_03_000000, scheduledTime=1452784924002, launchTime=1452784924002
2016-01-14 15:22:04,002 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:04,003 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:04,004 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,004 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:04,004 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C55DC0153409441F8FC29EAB74C3F340, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:04,004 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:04,004 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:04,006 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000004
2016-01-14 15:22:04,006 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:04,007 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452784923423_0001_00_000004
2016-01-14 15:22:04,008 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:04,009 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:04,009 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:04,009 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000003, Exiting
2016-01-14 15:22:04,009 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000003
2016-01-14 15:22:04,009 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000003
2016-01-14 15:22:04,011 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:04,011 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000004, launchTime=1452784924011
2016-01-14 15:22:04,011 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:04,011 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000003 completed successfully
2016-01-14 15:22:04,011 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:04,011 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:04,011 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,012 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_03_000000_0] to container: [container_1452784923423_0001_00_000004]
2016-01-14 15:22:04,012 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:04,013 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_03_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:04,013 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, startTime=1452784924013, containerId=container_1452784923423_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:04,013 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:04,013 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:04,013 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:04,014 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000004 given task: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04 Starting to run new task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,014 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:04,014 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:04,015 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:04,015 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:04,015 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:04,015 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C55DC0153409441F8FC29EAB74C3F340, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:04,015 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:04,016 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,016 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:04,017 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,017 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:04,019 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:04,019 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,019 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:04,019 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,021 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:04,021 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:04,021 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,021 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,022 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:04,023 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:04,023 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:04,024 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,024 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:04,024 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,034 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, ordinal: 3
2016-01-14 15:22:04,035 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, mem on start (mb), free: 177, total: 564, max: 672
2016-01-14 15:22:04,035 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,035 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:04,035 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:04,039 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:04,039 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,039 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:04,039 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:04,040 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:04,040 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:04,040 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:04,041 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:04,041 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:04,041 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, FACEDF332B814CD28C9FB990F959C4C3, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:04,041 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 66B5A676F67342598BB0CC462F2FC64C, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:04,042 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:04,042 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: FACEDF332B814CD28C9FB990F959C4C3 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:04,042 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,042 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:04,043 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:04,043 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:04,044 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:04,045 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: FACEDF332B814CD28C9FB990F959C4C3 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:04,045 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:04,046 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:04,046 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:04,047 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,048 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:04,049 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,050 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_02_000000_0_10045, runDuration: 77934, ]
2016-01-14 15:22:04,052 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 15:22:04,052 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_02_000000_0_10045] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452784924052, TimeTaken=1, Rate=4.89 MB/s
2016-01-14 15:22:04,052 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,053 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:04,053 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,054 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 15:22:04,054 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] A3A4232218854981AC93E364A39014EB, for 1 inputs
2016-01-14 15:22:04,053 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:04,054 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [FACEDF332B814CD28C9FB990F959C4C3] #12 in 3ms
2016-01-14 15:22:04,054 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [FACEDF332B814CD28C9FB990F959C4C3] #12, status:false, isInterrupted:false
2016-01-14 15:22:04,056 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:04,057 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 15:22:04,057 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:04,057 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:04,057 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 15:22:04,057 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,057 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:04,058 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, all 1 inputs ready in: 00:00:00.004
2016-01-14 15:22:04,058 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@69fc0ce9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@22b3cb5b
2016-01-14 15:22:04,067 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:04,067 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:04,068 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,068 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784923423_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:04,069 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,069 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, mem on close (mb), free: 174, total: 564, max: 672
2016-01-14 15:22:04,069 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,070 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:04,070 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:04,070 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:04,070 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:04,070 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:04 Completed running task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,072 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:04,073 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, startTime=1452784924013, finishTime=1452784924073, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:04,073 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:04,074 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,074 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:04,074 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskId=task_1452784923423_0001_1_03_000000, startTime=1452784924013, finishTime=1452784924074, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:04,075 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:04,075 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:04,074 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:04,075 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] : 1
2016-01-14 15:22:04,076 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,077 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, vertexId=vertex_1452784923423_0001_1_03, initRequestedTime=1452784923645, initedTime=1452784923659, startRequestedTime=1452784923682, startedTime=1452784923682, finishTime=1452784924076, timeTaken=394, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784924013, firstTasksToStart=[ task_1452784923423_0001_1_03_000000 ], lastTaskFinishTime=1452784924073, lastTasksToFinish=[ task_1452784923423_0001_1_03_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_03_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:04,077 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:04,078 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:04,078 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000004, stoppedTime=1452784924078, exitStatus=0
2016-01-14 15:22:04,078 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:04,078 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:04,078 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784923423_0001_1
2016-01-14 15:22:04,079 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784923423_0001_1
2016-01-14 15:22:04,079 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:04,079 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:04,079 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 66B5A676F67342598BB0CC462F2FC64C for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,078 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:04,079 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:04,081 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:04,082 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 15:22:04,082 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,083 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,083 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:04,083 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:04,083 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000004, Exiting
2016-01-14 15:22:04,083 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,083 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,084 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:04,084 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000004 completed successfully
2016-01-14 15:22:04,092 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:04,096 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784923423_0001_1, queueSize=0
2016-01-14 15:22:04,096 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:04,097 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784923423_0001_1, startTime=1452784923627, finishTime=1452784924093, timeTaken=466, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=18, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2400190464, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=27, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139695626, Process_Duration=61, Process_End_Time=5811139695687, Read_Duration=15, Tuples_Read=303, Tuples_Written=317, Write_Duration=7, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:04,098 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784923423_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:04,098 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:04,098 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784923423_0001_00_000004
2016-01-14 15:22:04,098 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:04 Completed Dag: dag_1452784923423_0001_1
2016-01-14 15:22:04,100 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784923423_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:04,100 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784923423_0001_1_post
2016-01-14 15:22:04,100 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,101 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:04,106 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001
2016-01-14 15:22:04,108 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:04,108 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:04,108 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:04,108 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:04,108 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:04,109 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:04,110 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:04,110 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:04,110 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:04,111 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:04,111 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:04,112 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:04,112 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:04,120 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001
2016-01-14 15:22:04,121 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:00.717
2016-01-14 15:22:04,191 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short

Standard error

2016-01-14 15:21:58 Running Dag: dag_1452784917401_0001_1
2016-01-14 15:21:58 Starting to run new task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:22:00 Starting to run new task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00 Completed running task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00 Completed Dag: dag_1452784917401_0001_1
2016-01-14 15:22:01 Running Dag: dag_1452784920806_0001_1
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01 Completed Dag: dag_1452784920806_0001_1
2016-01-14 15:22:02 Running Dag: dag_1452784922220_0001_1
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02 Completed Dag: dag_1452784922220_0001_1
2016-01-14 15:22:03 Running Dag: dag_1452784923423_0001_1
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:04 Starting to run new task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04 Completed running task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04 Completed Dag: dag_1452784923423_0001_1