2016-01-14 15:21:55,144 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(195)) - found 1 test platforms from classpath
2016-01-14 15:21:55,178 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(244)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-01-14 15:21:55,214 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-01-14 15:21:55,603 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-14 15:21:55,968 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(73)) - platform property overrides:
2016-01-14 15:21:56,136 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:21:56,138 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:21:56,217 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: 86ED42454B414342AD844F975AE040CD
2016-01-14 15:21:56,515 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.160
2016-01-14 15:21:56,560 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.206
2016-01-14 15:21:56,562 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:21:56,562 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:21:56,563 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:21:56,693 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:21:56,911 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 15:21:57,187 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.0.4
2016-01-14 15:21:57,189 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:21:57,190 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:57,191 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 15:21:57,191 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 15:21:57,191 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 15:21:57,191 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 15:21:57,192 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-01-14 15:21:57,193 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:21:57,213 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:21:57,395 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:21:57,402 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:21:57,402 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 15:21:57,447 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001 doesn't exist and is created
2016-01-14 15:21:57,520 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:21:57,521 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,522 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001
2016-01-14 15:21:57,553 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784917401_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:21:57,605 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:21:57,621 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-01-14 15:21:57,621 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-01-14 15:21:57,622 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,626 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:21:57,649 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:21:57,653 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:21:57,656 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir
2016-01-14 15:21:57,657 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/history.txt.appattempt_1452784917401_0001_000000, maxErrors=10
2016-01-14 15:21:57,657 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:21:57,660 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784917401_0001_000000, appSubmitTime=1452784917539, launchTime=1452784917552
2016-01-14 15:21:57,665 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:21:57,666 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:21:57,670 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:21:57,670 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:21:57,722 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 15:21:57,722 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 15:21:57,722 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,728 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:56013
2016-01-14 15:21:57,770 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:21:57,823 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 15:21:57,823 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 15:21:57,823 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:21:57,845 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:21:57,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784917401_0001_000000, startTime=1452784917844
2016-01-14 15:21:57,845 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:21:57,923 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:21:57,940 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:21:57,941 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001, dagName=[CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,163 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,264 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784917401_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/dag_1452784917401_0001_1.dot
2016-01-14 15:21:58,268 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001/localmode-log-dir/dag_1452784917401_0001_1-tez-dag.pb.txt
2016-01-14 15:21:58,408 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784917401_0001_1
2016-01-14 15:21:58,408 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:58,409 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58 Running Dag: dag_1452784917401_0001_1
2016-01-14 15:21:58,459 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:21:58,459 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784917401_0001_1, submitTime=1452784918163
2016-01-14 15:21:58,490 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:58,500 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:21:58,503 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:21:58,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784917401_0001_1, initTime=1452784918461
2016-01-14 15:21:58,504 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:21:58,507 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:21:58,507 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_STARTED]: dagID=dag_1452784917401_0001_1, startTime=1452784918506
2016-01-14 15:21:58,507 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:21:58,508 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:21:58,509 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E41B0DA6047647CD86110E309795E83D : {E2A47943811E44E4B9DEB0E1C823D43E={InputName=E2A47943811E44E4B9DEB0E1C823D43E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:21:58,509 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: E2A47943811E44E4B9DEB0E1C823D43E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:21:58,509 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,519 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,519 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,521 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001, dagName=[CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 15:21:58,524 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]. Starting root input initializers: 1
2016-01-14 15:21:58,529 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: E2A47943811E44E4B9DEB0E1C823D43E on vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,540 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:21:58,543 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,552 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784917401_0001, with dag id: dag_1452784917401_0001_1
2016-01-14 15:21:58,563 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,564 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,567 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:21:58,569 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input E2A47943811E44E4B9DEB0E1C823D43E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:21:58,570 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:21:58,570 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:21:58,583 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:58,586 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A07D163EA427438690C0188ECA117939, vertexId=vertex_1452784917401_0001_1_01, initRequestedTime=1452784918541, initedTime=1452784918585, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,587 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,591 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,628 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,629 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,629 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:58,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, vertexId=vertex_1452784917401_0001_1_02, initRequestedTime=1452784918591, initedTime=1452784918630, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,630 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,631 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,632 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,635 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:21:58,640 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: E2A47943811E44E4B9DEB0E1C823D43E on vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,652 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:21:58,652 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,653 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,653 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,654 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:21:58,657 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,668 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=8E92A936D8E747F4AE5E78A4F1908DF1, vertexName=D317F954092243EE989F4AA89C4E0B92, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:21:58,676 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=8E92A936D8E747F4AE5E78A4F1908DF1, vertexId=vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:21:58,678 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D317F954092243EE989F4AA89C4E0B92, vertexId=vertex_1452784917401_0001_1_03, initRequestedTime=1452784918632, initedTime=1452784918678, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:21:58,680 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:21:58,681 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:21:58,682 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {E2A47943811E44E4B9DEB0E1C823D43E=forAllWorkUnits=true, update=[1]}
2016-01-14 15:21:58,683 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] parallelism set to 1
2016-01-14 15:21:58,683 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:21:58,692 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_00, eventCount=1
2016-01-14 15:21:58,693 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:21:58,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E41B0DA6047647CD86110E309795E83D, vertexId=vertex_1452784917401_0001_1_00, initRequestedTime=1452784918509, initedTime=1452784918693, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:21:58,695 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] with distanceFromRoot: 0
2016-01-14 15:21:58,695 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:21:58,697 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:58,697 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,699 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_00, startRequestedTime=1452784918696, startedTime=1452784918696
2016-01-14 15:21:58,702 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,739 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:58,741 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskId=task_1452784917401_0001_1_00_000000, scheduledTime=1452784918739, launchTime=1452784918739
2016-01-14 15:21:58,742 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:58,744 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_00 for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,744 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] with distanceFromRoot: 1
2016-01-14 15:21:58,746 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:58,751 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E41B0DA6047647CD86110E309795E83D in vertex: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,754 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A07D163EA427438690C0188ECA117939 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,755 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_01, startRequestedTime=1452784918746, startedTime=1452784918746
2016-01-14 15:21:58,755 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,755 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,765 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E41B0DA6047647CD86110E309795E83D, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:58,767 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:58,767 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_01 for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,768 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] with distanceFromRoot: 2
2016-01-14 15:21:58,768 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:58,769 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A07D163EA427438690C0188ECA117939 in vertex: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:58,769 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E5DC0B0707E8449C9745B6C3C1E14658 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,771 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,772 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_02, startRequestedTime=1452784918768, startedTime=1452784918768
2016-01-14 15:21:58,772 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,773 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784917401_0001_1_02 for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] numStartedSources: 1 numSources: 1
2016-01-14 15:21:58,773 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] with distanceFromRoot: 3
2016-01-14 15:21:58,774 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E5DC0B0707E8449C9745B6C3C1E14658 in vertex: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:58,774 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D317F954092243EE989F4AA89C4E0B92 with 1 source tasks and 1 pending tasks
2016-01-14 15:21:58,776 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:21:58,777 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784917401_0001_1_03, startRequestedTime=1452784918774, startedTime=1452784918774
2016-01-14 15:21:58,777 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:21:58,820 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:21:58,825 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:21:58,825 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:21:58,830 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000001
2016-01-14 15:21:58,830 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:58,837 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784917401_0001_00_000001
2016-01-14 15:21:58,837 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:21:58,845 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:58,845 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:58,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000001, launchTime=1452784918845
2016-01-14 15:21:58,848 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000001
2016-01-14 15:21:58,852 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_00_000000_0] to container: [container_1452784917401_0001_00_000001]
2016-01-14 15:21:58,852 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:58,854 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_00_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:21:58,855 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000001 given task: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,855 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,856 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, startTime=1452784918853, containerId=container_1452784917401_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:21:58,856 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:58,857 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:58,858 INFO counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters: GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-01-14 15:21:58 Starting to run new task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,862 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,863 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:58,863 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:58,864 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:58,865 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:58,865 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:58,866 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:58,894 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:58,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E41B0DA6047647CD86110E309795E83D, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:58,897 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:58,901 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,901 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:58,902 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E2A47943811E44E4B9DEB0E1C823D43E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:21:58,907 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:21:58,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:58,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:58,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:58,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,916 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,923 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:58,923 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,924 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:58,929 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:58,932 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:21:58,933 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,933 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,957 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E41B0DA6047647CD86110E309795E83D, ordinal: 0
2016-01-14 15:21:58,958 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E41B0DA6047647CD86110E309795E83D, mem on start (mb), free: 223, total: 281, max: 672
2016-01-14 15:21:58,958 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:58,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:58,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:58,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:58,960 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:58,967 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:21:58,967 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:21:58,967 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:58,968 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:58,968 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:21:58,969 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E2A47943811E44E4B9DEB0E1C823D43E, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:21:58,969 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A07D163EA427438690C0188ECA117939, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:21:58,969 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:58,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,970 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:21:58,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:58,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E2A47943811E44E4B9DEB0E1C823D43E being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:58,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:58,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:58,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:58,973 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:58,975 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:21:58,984 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@68f6db2f
2016-01-14 15:21:58,997 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@68f6db2f
2016-01-14 15:21:58,998 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:21:58,998 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:21:59,015 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:59,049 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: E2A47943811E44E4B9DEB0E1C823D43E
2016-01-14 15:21:59,051 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,051 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,069 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@5620e5a8; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@9c5a4c5; comparator=cascading.tuple.hadoop.util.TupleComparator@ec48686; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,072 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,074 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,075 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,186 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:21:59,187 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:21:59,188 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:21:59,189 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:21:59,190 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:21:59,198 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:21:59,198 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:21:59,201 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E41B0DA6047647CD86110E309795E83D, all 1 inputs ready in: 00:00:00.002
2016-01-14 15:21:59,214 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,214 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E41B0DA6047647CD86110E309795E83D, mem on close (mb), free: 111, total: 281, max: 672
2016-01-14 15:21:59,215 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,216 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,216 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:21:59,217 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:21:59,237 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,260 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,261 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_00, eventCount=1
2016-01-14 15:21:59,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:21:59,267 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskAttemptId=attempt_1452784917401_0001_1_00_000000_0, startTime=1452784918853, finishTime=1452784919262, timeTaken=409, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:21:59,279 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:21:59,279 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,279 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,283 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,284 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,286 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_00_000000_0
2016-01-14 15:21:59,289 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, taskId=task_1452784917401_0001_1_00_000000, startTime=1452784918853, finishTime=1452784919288, timeTaken=435, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:21:59,296 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,296 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,297 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] : 1
2016-01-14 15:21:59,298 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:21:59,319 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=E41B0DA6047647CD86110E309795E83D, vertexId=vertex_1452784917401_0001_1_00, initRequestedTime=1452784918509, initedTime=1452784918693, startRequestedTime=1452784918696, startedTime=1452784918696, finishTime=1452784919298, timeTaken=602, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919202, Process_Duration=12, Process_End_Time=1452784919214, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784918853, firstTasksToStart=[ task_1452784917401_0001_1_00_000000 ], lastTaskFinishTime=1452784919262, lastTasksToFinish=[ task_1452784917401_0001_1_00_000000 ], minTaskDuration=409, maxTaskDuration=409, avgTaskDuration=409.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_00_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,331 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,333 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,334 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000001, stoppedTime=1452784919333, exitStatus=0
2016-01-14 15:21:59,335 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,335 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] attempt: attempt_1452784917401_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,335 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,336 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,338 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:21:59,338 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:21:59,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:59,339 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=A07D163EA427438690C0188ECA117939, taskId=task_1452784917401_0001_1_01_000000, scheduledTime=1452784919338, launchTime=1452784919338
2016-01-14 15:21:59,339 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:59,339 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,340 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:21:59,340 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A07D163EA427438690C0188ECA117939, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,341 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:59,341 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:59,344 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000002
2016-01-14 15:21:59,344 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:59,346 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:21:59,346 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000001: 63 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:21:59,346 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000001, Exiting
2016-01-14 15:21:59,347 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,347 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000001
2016-01-14 15:21:59,353 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:21:59,353 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000001 completed successfully
2016-01-14 15:21:59,354 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:21:59,355 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784917401_0001_00_000002
2016-01-14 15:21:59,361 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:21:59,366 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:59,370 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:59,370 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000002, launchTime=1452784919366
2016-01-14 15:21:59,374 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,375 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_01_000000_0] to container: [container_1452784917401_0001_00_000002]
2016-01-14 15:21:59,376 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:59,376 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000002 given task: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,377 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,377 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_01_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:21:59,377 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:59,377 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A07D163EA427438690C0188ECA117939, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, startTime=1452784919376, containerId=container_1452784917401_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:21:59,378 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:59,378 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,379 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,379 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,385 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:59,385 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:59,386 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:59,386 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:59,386 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:59,389 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A07D163EA427438690C0188ECA117939, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,389 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:59,390 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,392 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:59,392 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,395 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E41B0DA6047647CD86110E309795E83D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:21:59,395 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,400 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:21:59,405 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:59,409 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:59,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:59,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,409 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:59,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,425 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:59,426 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,440 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:21:59,441 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:21:59,441 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,446 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A07D163EA427438690C0188ECA117939, ordinal: 1
2016-01-14 15:21:59,447 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A07D163EA427438690C0188ECA117939, mem on start (mb), free: 99, total: 281, max: 672
2016-01-14 15:21:59,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:59,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:59,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:59,447 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:59,448 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:59,448 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:21:59,450 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:59,451 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:21:59,451 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E5DC0B0707E8449C9745B6C3C1E14658, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:21:59,451 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E41B0DA6047647CD86110E309795E83D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:21:59,451 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:59,452 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,458 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E41B0DA6047647CD86110E309795E83D being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:59,458 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:59,460 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:59,467 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,468 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:21:59,472 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E41B0DA6047647CD86110E309795E83D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:21:59,474 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:21:59,480 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:21:59,484 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:21:59,487 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,488 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:59,492 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,494 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_00_000000_0_10003, runDuration: 324320, ]
2016-01-14 15:21:59,499 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 15:21:59,505 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_00_000000_0_10003] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784919505, TimeTaken=3, Rate=0.35 MB/s
2016-01-14 15:21:59,508 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,509 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:21:59,513 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E41B0DA6047647CD86110E309795E83D] #1 in 12ms
2016-01-14 15:21:59,514 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: CC67493F06964370956C01CE4B630E7C
2016-01-14 15:21:59,514 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,513 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E41B0DA6047647CD86110E309795E83D] #1, status:false, isInterrupted:false
2016-01-14 15:21:59,515 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:21:59,517 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:21:59,518 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,525 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:21:59,525 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,534 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@9aa1ae3; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@7ecb3209; comparator=cascading.tuple.hadoop.util.TupleComparator@4e422de7; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,534 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,537 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,537 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,540 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:21:59,541 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,605 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:21:59,606 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:21:59,606 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:21:59,607 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:21:59,607 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:21:59,607 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:21:59,608 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] CC67493F06964370956C01CE4B630E7C, for 1 inputs
2016-01-14 15:21:59,617 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:21:59,617 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:21:59,625 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A07D163EA427438690C0188ECA117939, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:21:59,628 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@253217dd; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7bbcc13e
2016-01-14 15:21:59,667 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,667 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A07D163EA427438690C0188ECA117939, mem on close (mb), free: 91, total: 345, max: 672
2016-01-14 15:21:59,667 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E41B0DA6047647CD86110E309795E83D
2016-01-14 15:21:59,668 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:21:59,668 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:21:59,668 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,669 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:21:59,670 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:21:59,671 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,671 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,671 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:21:59,671 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:21:59,703 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,707 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_01, eventCount=1
2016-01-14 15:21:59,709 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12]]
2016-01-14 15:21:59,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, taskAttemptId=attempt_1452784917401_0001_1_01_000000_0, startTime=1452784919376, finishTime=1452784919709, timeTaken=333, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-01-14 15:21:59,714 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,714 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_01_000000_0
2016-01-14 15:21:59,716 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,714 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:21:59,716 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,717 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,718 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, taskId=task_1452784917401_0001_1_01_000000, startTime=1452784919376, finishTime=1452784919716, timeTaken=340, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-01-14 15:21:59,718 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,718 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] : 1
2016-01-14 15:21:59,719 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:21:59,719 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,721 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=A07D163EA427438690C0188ECA117939, vertexId=vertex_1452784917401_0001_1_01, initRequestedTime=1452784918541, initedTime=1452784918585, startRequestedTime=1452784918746, startedTime=1452784918746, finishTime=1452784919719, timeTaken=973, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=42, MERGE_PHASE_TIME=73, FIRST_EVENT_RECEIVED=33, LAST_EVENT_RECEIVED=33, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919625, Process_Duration=42, Process_End_Time=1452784919667, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=12, vertexStats=firstTaskStartTime=1452784919376, firstTasksToStart=[ task_1452784917401_0001_1_01_000000 ], lastTaskFinishTime=1452784919709, lastTasksToFinish=[ task_1452784917401_0001_1_01_000000 ], minTaskDuration=333, maxTaskDuration=333, avgTaskDuration=333.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_01_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,729 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,731 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,732 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000002, stoppedTime=1452784919732, exitStatus=0
2016-01-14 15:21:59,733 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:21:59,733 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,734 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:21:59,734 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000002, Exiting
2016-01-14 15:21:59,734 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,734 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000002
2016-01-14 15:21:59,734 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,735 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] attempt: attempt_1452784917401_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,735 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:21:59,736 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,736 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:21:59,737 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:21:59,738 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:21:59,738 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:21:59,739 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000002 completed successfully
2016-01-14 15:21:59,738 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskId=task_1452784917401_0001_1_02_000000, scheduledTime=1452784919738, launchTime=1452784919738
2016-01-14 15:21:59,740 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:21:59,741 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:21:59,742 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,743 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:21:59,745 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E5DC0B0707E8449C9745B6C3C1E14658, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,745 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:21:59,746 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:21:59,748 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000003
2016-01-14 15:21:59,748 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:21:59,751 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784917401_0001_00_000003
2016-01-14 15:21:59,752 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:21:59,754 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:21:59,754 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000003, launchTime=1452784919754
2016-01-14 15:21:59,754 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:21:59,755 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000003
2016-01-14 15:21:59,756 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_02_000000_0] to container: [container_1452784917401_0001_00_000003]
2016-01-14 15:21:59,756 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:21:59,757 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000003 given task: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,757 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,757 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_02_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:21:59,758 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:21:59 Starting to run new task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,758 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,758 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,759 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:21:59,759 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:21:59,759 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:21:59,759 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:21:59,759 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:21:59,760 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E5DC0B0707E8449C9745B6C3C1E14658, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:21:59,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, startTime=1452784919757, containerId=container_1452784917401_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:21:59,760 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:21:59,761 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:21:59,761 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:21:59,763 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:21:59,764 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:21:59,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A07D163EA427438690C0188ECA117939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:21:59,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D317F954092243EE989F4AA89C4E0B92, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:21:59,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:21:59,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,767 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:21:59,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,771 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:21:59,771 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,771 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:21:59,772 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:21:59,772 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:21:59,772 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:21:59,773 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,773 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,787 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, ordinal: 2
2016-01-14 15:21:59,787 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, mem on start (mb), free: 84, total: 345, max: 672
2016-01-14 15:21:59,787 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:21:59,788 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:21:59,788 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:21:59,788 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:21:59,789 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:21:59,789 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:21:59,789 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:21:59,790 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:21:59,790 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:21:59,791 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D317F954092243EE989F4AA89C4E0B92, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:21:59,791 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A07D163EA427438690C0188ECA117939, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:21:59,791 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:21:59,792 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A07D163EA427438690C0188ECA117939 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:21:59,792 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,794 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:21:59,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:21:59,797 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,798 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:21:59,798 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A07D163EA427438690C0188ECA117939 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:21:59,799 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:21:59,799 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:21:59,799 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:21:59,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:21:59,802 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,805 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_01_000000_0_10006, runDuration: 292878, ]
2016-01-14 15:21:59,808 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_01_000000_0_10006] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784919808, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:21:59,808 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,809 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:21:59,811 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A07D163EA427438690C0188ECA117939] #2 in 5ms
2016-01-14 15:21:59,811 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A07D163EA427438690C0188ECA117939] #2, status:false, isInterrupted:false
2016-01-14 15:21:59,811 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:21:59,814 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:21:59,814 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:21:59,814 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,815 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:21:59,816 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:21:59,816 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,819 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:21:59,821 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 4AC4B50191CE4F079527F017D3591EC2
2016-01-14 15:21:59,825 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:21:59,826 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:21:59,827 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@46423a2d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@25f98464; comparator=cascading.tuple.hadoop.util.TupleComparator@2062d4b6; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:21:59,828 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:21:59,828 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:21:59,829 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:21:59,918 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:21:59,919 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:21:59,919 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:21:59,920 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:21:59,920 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:21:59,921 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 4AC4B50191CE4F079527F017D3591EC2, for 1 inputs
2016-01-14 15:21:59,922 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:21:59,923 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@75789cb0; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@41772310
2016-01-14 15:21:59,958 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,958 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5DC0B0707E8449C9745B6C3C1E14658, mem on close (mb), free: 79, total: 408, max: 672
2016-01-14 15:21:59,958 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A07D163EA427438690C0188ECA117939
2016-01-14 15:21:59,959 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:21:59,960 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:21:59,961 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:21:59,961 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:21:59,961 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:21:59,962 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:21:59,962 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:21:59,963 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 15:21:59,963 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:21:59,973 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:21:59 Completed running task attempt: attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,979 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:21:59,980 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 15:21:59,980 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784917401_0001_1_02, eventCount=1
2016-01-14 15:21:59,981 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:21:59,982 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskAttemptId=attempt_1452784917401_0001_1_02_000000_0, startTime=1452784919757, finishTime=1452784919981, timeTaken=224, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:21:59,983 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:21:59,983 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_02_000000_0
2016-01-14 15:21:59,983 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:21:59,985 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:21:59,986 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:21:59,987 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:21:59,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, taskId=task_1452784917401_0001_1_02_000000, startTime=1452784919757, finishTime=1452784919983, timeTaken=226, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:21:59,990 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:21:59,990 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] : 1
2016-01-14 15:21:59,991 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:21:59,992 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:21:59,990 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000003
2016-01-14 15:21:59,996 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=E5DC0B0707E8449C9745B6C3C1E14658, vertexId=vertex_1452784917401_0001_1_02, initRequestedTime=1452784918591, initedTime=1452784918630, startRequestedTime=1452784918768, startedTime=1452784918768, finishTime=1452784919991, timeTaken=1223, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784919922, Process_Duration=36, Process_End_Time=1452784919958, Read_Duration=15, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452784919757, firstTasksToStart=[ task_1452784917401_0001_1_02_000000 ], lastTaskFinishTime=1452784919981, lastTasksToFinish=[ task_1452784917401_0001_1_02_000000 ], minTaskDuration=224, maxTaskDuration=224, avgTaskDuration=224.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_02_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:21:59,996 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:21:59,996 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:21:59,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000003, stoppedTime=1452784919997, exitStatus=0
2016-01-14 15:21:59,997 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:21:59,998 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:21:59,999 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] attempt: attempt_1452784917401_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:21:59,999 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D317F954092243EE989F4AA89C4E0B92
2016-01-14 15:22:00,000 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,000 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:00,001 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784917401_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:00,001 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:00,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_STARTED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskId=task_1452784917401_0001_1_03_000000, scheduledTime=1452784920001, launchTime=1452784920001
2016-01-14 15:22:00,002 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:00,002 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,003 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:00,003 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: D317F954092243EE989F4AA89C4E0B92, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:00,003 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:00,004 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:00,004 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:00,005 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:00,005 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000003, Exiting
2016-01-14 15:22:00,005 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000003
2016-01-14 15:22:00,005 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000003
2016-01-14 15:22:00,006 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:00,007 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000003 completed successfully
2016-01-14 15:22:00,008 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784917401_0001_00_000004
2016-01-14 15:22:00,008 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:00,008 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:00,011 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784917401_0001_00_000004
2016-01-14 15:22:00,012 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:00,016 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:00,016 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784917401_0001_00_000004, launchTime=1452784920016
2016-01-14 15:22:00,016 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:00,017 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,018 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784917401_0001_1_03_000000_0] to container: [container_1452784917401_0001_00_000004]
2016-01-14 15:22:00,018 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:00,018 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784917401_0001_00_000004 given task: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,018 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00 Starting to run new task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,019 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,019 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,019 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:00,019 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:00,019 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:00,019 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:00,020 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:00,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9] euclidean/(1/1) ...osite/euclidean/composite, VertexName: D317F954092243EE989F4AA89C4E0B92, VertexParallelism: 1, TaskAttemptID:attempt_1452784917401_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:00,020 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:00,021 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,023 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:00,023 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,024 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784917401_0001_1_03_000000_0] started. Is using containerId: [container_1452784917401_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:00,025 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E5DC0B0707E8449C9745B6C3C1E14658, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:00,026 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:00,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,026 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, startTime=1452784920018, containerId=container_1452784917401_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784917401_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:00,032 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:00,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:00,031 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:00,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=8E92A936D8E747F4AE5E78A4F1908DF1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:00,027 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784917401_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:00,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:00,037 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,037 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:00,037 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,039 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:00,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,042 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:00,043 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:00,043 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,045 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,060 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D317F954092243EE989F4AA89C4E0B92, ordinal: 3
2016-01-14 15:22:00,061 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D317F954092243EE989F4AA89C4E0B92, mem on start (mb), free: 73, total: 408, max: 672
2016-01-14 15:22:00,061 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:00,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:00,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:00,072 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:00,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,074 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:00,074 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:00,075 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:00,078 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:00,078 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:00,078 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:00,078 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:00,079 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 8E92A936D8E747F4AE5E78A4F1908DF1, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:00,079 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E5DC0B0707E8449C9745B6C3C1E14658, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:00,080 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:00,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E5DC0B0707E8449C9745B6C3C1E14658 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:00,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,082 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:00,082 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:00,082 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E5DC0B0707E8449C9745B6C3C1E14658 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:00,083 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:00,083 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:00,083 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:00,084 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:00,085 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:00,085 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,086 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:00,087 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,089 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784917401_0001_1_02_000000_0_10009, runDuration: 207486, ]
2016-01-14 15:22:00,092 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-01-14 15:22:00,092 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784917401_0001_1_02_000000_0_10009] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452784920092, TimeTaken=2, Rate=2.07 MB/s
2016-01-14 15:22:00,093 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,093 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:00,094 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E5DC0B0707E8449C9745B6C3C1E14658] #3 in 4ms
2016-01-14 15:22:00,094 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E5DC0B0707E8449C9745B6C3C1E14658] #3, status:false, isInterrupted:false
2016-01-14 15:22:00,094 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: 1F5BA0AA0DD54EC78DF32C09F8E9E519
2016-01-14 15:22:00,094 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: 8E92A936D8E747F4AE5E78A4F1908DF1
2016-01-14 15:22:00,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:00,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 15:22:00,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:00,096 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:00,097 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 15:22:00,097 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,097 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:00,098 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 15:22:00,099 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 1F5BA0AA0DD54EC78DF32C09F8E9E519, for 1 inputs
2016-01-14 15:22:00,100 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D317F954092243EE989F4AA89C4E0B92, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:00,103 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6aff5bd4; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@22e5f22d
2016-01-14 15:22:00,117 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:00,117 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:00,118 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,119 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784917401_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:00,120 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,121 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D317F954092243EE989F4AA89C4E0B92, mem on close (mb), free: 69, total: 408, max: 672
2016-01-14 15:22:00,121 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E5DC0B0707E8449C9745B6C3C1E14658
2016-01-14 15:22:00,122 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:00,123 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:00,123 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:00,124 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:00,124 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:00 Completed running task attempt: attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,126 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:00,127 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:00,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskAttemptId=attempt_1452784917401_0001_1_03_000000_0, startTime=1452784920018, finishTime=1452784920127, timeTaken=109, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:00,130 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784917401_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:00,130 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784917401_0001_1_03_000000_0
2016-01-14 15:22:00,130 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:00,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:00,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:TASK_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, taskId=task_1452784917401_0001_1_03_000000, startTime=1452784920018, finishTime=1452784920130, timeTaken=112, status=SUCCEEDED, successfulAttemptID=attempt_1452784917401_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:00,133 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784917401_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:00,133 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] : 1
2016-01-14 15:22:00,133 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,132 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:00,133 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:00,135 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:VERTEX_FINISHED]: vertexName=D317F954092243EE989F4AA89C4E0B92, vertexId=vertex_1452784917401_0001_1_03, initRequestedTime=1452784918632, initedTime=1452784918678, startRequestedTime=1452784918774, startedTime=1452784918774, finishTime=1452784920133, timeTaken=1359, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784920100, Process_Duration=20, Process_End_Time=1452784920120, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784920018, firstTasksToStart=[ task_1452784917401_0001_1_03_000000 ], lastTaskFinishTime=1452784920127, lastTasksToFinish=[ task_1452784917401_0001_1_03_000000 ], minTaskDuration=109, maxTaskDuration=109, avgTaskDuration=109.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784917401_0001_1_03_000000 ], longestDurationTasks=[ task_1452784917401_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:00,136 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:00,137 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:00,137 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784917401_0001_00_000004, stoppedTime=1452784920137, exitStatus=0
2016-01-14 15:22:00,138 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:00,138 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:00,138 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784917401_0001_1
2016-01-14 15:22:00,136 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,140 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784917401_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:00,137 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:00,143 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:00,144 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784917401_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:00,144 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784917401_0001_00_000004, Exiting
2016-01-14 15:22:00,144 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,144 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784917401_0001_00_000004
2016-01-14 15:22:00,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784917401_0001_1
2016-01-14 15:22:00,145 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784917401_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:00,147 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_02 [E5DC0B0707E8449C9745B6C3C1E14658]
2016-01-14 15:22:00,147 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784917401_0001_00_000004 completed successfully
2016-01-14 15:22:00,147 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 8E92A936D8E747F4AE5E78A4F1908DF1 for vertex: vertex_1452784917401_0001_1_03 [D317F954092243EE989F4AA89C4E0B92]
2016-01-14 15:22:00,162 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_00 [E41B0DA6047647CD86110E309795E83D]
2016-01-14 15:22:00,162 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784917401_0001_1_01 [A07D163EA427438690C0188ECA117939]
2016-01-14 15:22:00,172 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784917401_0001_1, queueSize=0
2016-01-14 15:22:00,172 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:00,173 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784917401_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784917401_0001_1, startTime=1452784918506, finishTime=1452784920163, timeTaken=1657, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14984, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1513619456, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=65, MERGE_PHASE_TIME=106, FIRST_EVENT_RECEIVED=49, LAST_EVENT_RECEIVED=49, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139678849, Process_Duration=110, Process_End_Time=5811139678959, Read_Duration=23, Tuples_Read=303, Tuples_Written=317, Write_Duration=22, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:00,176 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784917401_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:00,176 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784917401_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:00,176 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784917401_0001_00_000004
2016-01-14 15:22:00,176 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784917401_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:00 Completed Dag: dag_1452784917401_0001_1
2016-01-14 15:22:00,179 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784917401_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:00,179 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784917401_0001_1_post
2016-01-14 15:22:00,180 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:00,181 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:00,187 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452784917401_0001
2016-01-14 15:22:00,192 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:00,192 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:00,192 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:00,192 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:00,193 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:00,193 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:00,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:00,196 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:00,197 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:00,197 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:00,198 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:00,199 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:00,199 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:00,208 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/CEF3EE050722414F9689E5A221A24E8E/C300DCBFAA8249C395DFA00EBCD3B2D9/.tez/application_1452784917401_0001
2016-01-14 15:22:00,210 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:03.020
2016-01-14 15:22:00,346 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 15:22:00,386 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:00,386 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:00,487 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.069
2016-01-14 15:22:00,534 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.115
2016-01-14 15:22:00,535 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:00,536 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:00,536 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:00,560 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:00,590 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 15:22:00,784 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting
2016-01-14 15:22:00,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:00,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 15:22:00,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] parallel execution of steps is enabled: false
2016-01-14 15:22:00,786 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executing total steps: 1
2016-01-14 15:22:00,786 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] allocating management threads: 1
2016-01-14 15:22:00,786 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,787 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:00,801 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tez session mode enabled: true
2016-01-14 15:22:00,806 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:00,806 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:00,807 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:00,817 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001 doesn't exist and is created
2016-01-14 15:22:00,838 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:00,838 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:00,840 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001
2016-01-14 15:22:00,844 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784920806_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:00,859 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:00,860 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:00,863 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:00,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:00,864 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir
2016-01-14 15:22:00,864 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/history.txt.appattempt_1452784920806_0001_000000, maxErrors=10
2016-01-14 15:22:00,864 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:00,865 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784920806_0001_000000, appSubmitTime=1452784920843, launchTime=1452784920843
2016-01-14 15:22:00,871 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:00,875 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:00,877 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:00,877 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:00,881 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:41215
2016-01-14 15:22:00,884 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:00,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784920806_0001_000000, startTime=1452784920888
2016-01-14 15:22:00,888 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:00,888 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:00,939 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:00,940 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:00,940 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001, dagName=[A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,963 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:00,966 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784920806_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/dag_1452784920806_0001_1.dot
2016-01-14 15:22:00,967 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001/localmode-log-dir/dag_1452784920806_0001_1-tez-dag.pb.txt
2016-01-14 15:22:01,004 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784920806_0001_1
2016-01-14 15:22:01,004 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,004 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:01 Running Dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,020 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784920806_0001_1, submitTime=1452784920963
2016-01-14 15:22:01,020 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:01,022 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,022 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:01,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784920806_0001_1, initTime=1452784921020
2016-01-14 15:22:01,024 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:01,025 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:01,026 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_STARTED]: dagID=dag_1452784920806_0001_1, startTime=1452784921025
2016-01-14 15:22:01,026 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:01,026 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:01,026 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: C58271B0BFDE40AEA813831159DCF987 : {CB71003DE96947C99F8E2532DE037A51={InputName=CB71003DE96947C99F8E2532DE037A51}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:01,026 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: CB71003DE96947C99F8E2532DE037A51, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:01,026 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,027 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:01,027 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]. Starting root input initializers: 1
2016-01-14 15:22:01,029 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:01,029 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,029 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001, dagName=[A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 15:22:01,031 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: CB71003DE96947C99F8E2532DE037A51 on vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,037 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:01,038 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input CB71003DE96947C99F8E2532DE037A51 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:01,038 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,038 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,041 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,042 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:01,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, vertexId=vertex_1452784920806_0001_1_01, initRequestedTime=1452784921029, initedTime=1452784921042, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,042 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,043 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,043 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,045 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:01,048 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:01,048 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,048 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,048 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,049 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: CB71003DE96947C99F8E2532DE037A51 on vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,049 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D4533A9745A740C398F52479AA7C29A1, vertexId=vertex_1452784920806_0001_1_02, initRequestedTime=1452784921043, initedTime=1452784921049, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,050 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,050 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:01,051 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {CB71003DE96947C99F8E2532DE037A51=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:01,051 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] parallelism set to 1
2016-01-14 15:22:01,051 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:01,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_00, eventCount=1
2016-01-14 15:22:01,055 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,055 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C58271B0BFDE40AEA813831159DCF987, vertexId=vertex_1452784920806_0001_1_00, initRequestedTime=1452784921026, initedTime=1452784921055, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:01,055 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] with distanceFromRoot: 0
2016-01-14 15:22:01,056 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:01,056 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,065 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] submitted tez dag to app master: application_1452784920806_0001, with dag id: dag_1452784920806_0001_1
2016-01-14 15:22:01,067 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:01,067 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,068 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:01,069 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,071 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=AA5BC7EE4A07438280C31731A446F7FB, vertexName=B383C94B7B3947F0A2BD91A29DE0815D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:01,071 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=AA5BC7EE4A07438280C31731A446F7FB, vertexId=vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, vertexId=vertex_1452784920806_0001_1_03, initRequestedTime=1452784921056, initedTime=1452784921073, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:01,073 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:01,073 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:01,074 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,074 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,075 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_00, startRequestedTime=1452784921074, startedTime=1452784921074
2016-01-14 15:22:01,075 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,076 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskId=task_1452784920806_0001_1_00_000000, scheduledTime=1452784921077, launchTime=1452784921077
2016-01-14 15:22:01,077 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,077 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,078 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_00 for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,079 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] with distanceFromRoot: 1
2016-01-14 15:22:01,079 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,080 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C58271B0BFDE40AEA813831159DCF987 in vertex: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,080 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A25E8F44D00D4AD7A993BB5C20FE0E40 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,080 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_01, startRequestedTime=1452784921080, startedTime=1452784921080
2016-01-14 15:22:01,081 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,081 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: C58271B0BFDE40AEA813831159DCF987, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,082 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,082 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_01 for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,082 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] with distanceFromRoot: 2
2016-01-14 15:22:01,082 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,083 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A25E8F44D00D4AD7A993BB5C20FE0E40 in vertex: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,083 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:01,083 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D4533A9745A740C398F52479AA7C29A1 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,084 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,084 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_02, startRequestedTime=1452784921083, startedTime=1452784921083
2016-01-14 15:22:01,084 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,085 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:01,086 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:01,086 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000001
2016-01-14 15:22:01,087 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,087 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784920806_0001_1_02 for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] numStartedSources: 1 numSources: 1
2016-01-14 15:22:01,087 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] with distanceFromRoot: 3
2016-01-14 15:22:01,091 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784920806_0001_00_000001
2016-01-14 15:22:01,091 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:01,093 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D4533A9745A740C398F52479AA7C29A1 in vertex: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,093 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B383C94B7B3947F0A2BD91A29DE0815D with 1 source tasks and 1 pending tasks
2016-01-14 15:22:01,093 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784920806_0001_1_03, startRequestedTime=1452784921092, startedTime=1452784921092
2016-01-14 15:22:01,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000001, launchTime=1452784921093
2016-01-14 15:22:01,093 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:01,093 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:01,094 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,096 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,097 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_00_000000_0] to container: [container_1452784920806_0001_00_000001]
2016-01-14 15:22:01,097 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,098 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000001 given task: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,098 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,098 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_00_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,099 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,099 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, startTime=1452784921098, containerId=container_1452784920806_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:01,099 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,099 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,100 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,100 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,100 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,101 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,101 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,101 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,101 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: C58271B0BFDE40AEA813831159DCF987, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,102 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,103 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,105 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:01,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CB71003DE96947C99F8E2532DE037A51, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:01,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,109 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,109 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,110 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,112 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,112 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,139 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,141 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,142 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,144 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:01,145 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,153 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C58271B0BFDE40AEA813831159DCF987, ordinal: 0
2016-01-14 15:22:01,153 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C58271B0BFDE40AEA813831159DCF987, mem on start (mb), free: 260, total: 516, max: 672
2016-01-14 15:22:01,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,154 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,154 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:01,155 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:01,155 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,155 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,155 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:01,155 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CB71003DE96947C99F8E2532DE037A51, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:01,155 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A25E8F44D00D4AD7A993BB5C20FE0E40, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:01,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CB71003DE96947C99F8E2532DE037A51 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,156 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:01,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,157 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,157 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,157 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,157 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:01,159 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@49557cef
2016-01-14 15:22:01,160 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@49557cef
2016-01-14 15:22:01,160 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:01,161 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:01,163 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:01,164 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: CB71003DE96947C99F8E2532DE037A51
2016-01-14 15:22:01,164 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,164 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,166 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@1ce0f06c; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@73c85143; comparator=cascading.tuple.hadoop.util.TupleComparator@644f4117; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,166 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,167 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,167 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,181 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:01,182 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:01,182 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:01,182 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:01,183 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:01,183 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:01,184 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:01,184 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C58271B0BFDE40AEA813831159DCF987, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,189 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,189 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C58271B0BFDE40AEA813831159DCF987, mem on close (mb), free: 158, total: 516, max: 672
2016-01-14 15:22:01,189 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,189 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,190 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:01,190 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:01,194 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,199 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:01,200 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_00, eventCount=1
2016-01-14 15:22:01,201 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:01,202 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,202 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskAttemptId=attempt_1452784920806_0001_1_00_000000_0, startTime=1452784921098, finishTime=1452784921202, timeTaken=104, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:01,204 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,204 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_00_000000_0
2016-01-14 15:22:01,205 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, taskId=task_1452784920806_0001_1_00_000000, startTime=1452784921098, finishTime=1452784921204, timeTaken=106, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:01,205 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,206 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] : 1
2016-01-14 15:22:01,206 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,204 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,205 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,208 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=C58271B0BFDE40AEA813831159DCF987, vertexId=vertex_1452784920806_0001_1_00, initRequestedTime=1452784921026, initedTime=1452784921055, startRequestedTime=1452784921074, startedTime=1452784921074, finishTime=1452784921206, timeTaken=132, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921184, Process_Duration=5, Process_End_Time=1452784921189, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784921098, firstTasksToStart=[ task_1452784920806_0001_1_00_000000 ], lastTaskFinishTime=1452784921202, lastTasksToFinish=[ task_1452784920806_0001_1_00_000000 ], minTaskDuration=104, maxTaskDuration=104, avgTaskDuration=104.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_00_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,209 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,210 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,213 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,214 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] attempt: attempt_1452784920806_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,214 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,214 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,215 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,215 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,215 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000001, stoppedTime=1452784921215, exitStatus=0
2016-01-14 15:22:01,215 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,215 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,218 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,218 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskId=task_1452784920806_0001_1_01_000000, scheduledTime=1452784921218, launchTime=1452784921218
2016-01-14 15:22:01,218 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,218 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,219 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,219 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,219 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: A25E8F44D00D4AD7A993BB5C20FE0E40, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,219 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,219 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,220 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000002
2016-01-14 15:22:01,221 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,225 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784920806_0001_00_000002
2016-01-14 15:22:01,225 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:01,226 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,226 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000002, launchTime=1452784921226
2016-01-14 15:22:01,230 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,231 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_01_000000_0] to container: [container_1452784920806_0001_00_000002]
2016-01-14 15:22:01,231 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,232 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_01_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,232 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, startTime=1452784921232, containerId=container_1452784920806_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:01,233 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,233 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,233 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,233 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000002 given task: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,233 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,234 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,234 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,234 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,234 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,234 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,235 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,235 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,235 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,235 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,236 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000001, Exiting
2016-01-14 15:22:01,236 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,236 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000001
2016-01-14 15:22:01,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: A25E8F44D00D4AD7A993BB5C20FE0E40, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,238 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,238 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000001 completed successfully
2016-01-14 15:22:01,239 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,239 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,240 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,241 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,241 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,243 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,244 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,246 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:01,241 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C58271B0BFDE40AEA813831159DCF987, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,247 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,249 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,250 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,250 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,250 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,247 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,254 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,254 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,255 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,263 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, ordinal: 1
2016-01-14 15:22:01,263 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, mem on start (mb), free: 148, total: 516, max: 672
2016-01-14 15:22:01,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,263 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:01,264 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C58271B0BFDE40AEA813831159DCF987, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:01,265 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D4533A9745A740C398F52479AA7C29A1, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:01,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C58271B0BFDE40AEA813831159DCF987 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,266 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,267 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,267 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C58271B0BFDE40AEA813831159DCF987 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,267 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:01,267 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,268 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,272 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,272 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,273 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,273 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,273 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_00_000000_0_10015, runDuration: 83630, ]
2016-01-14 15:22:01,278 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_00_000000_0_10015] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784921278, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:01,278 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,278 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:01,278 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [C58271B0BFDE40AEA813831159DCF987] #4 in 1ms
2016-01-14 15:22:01,278 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 15:22:01,279 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C58271B0BFDE40AEA813831159DCF987] #4, status:false, isInterrupted:false
2016-01-14 15:22:01,280 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,280 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:01,280 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,281 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,281 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:01,281 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,282 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,282 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 8EF3D351182B4FC9B104498D5B49B7BB
2016-01-14 15:22:01,283 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,283 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,285 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@332803fd; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@81df148; comparator=cascading.tuple.hadoop.util.TupleComparator@65cbb221; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,285 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,285 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,286 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,337 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:01,337 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:01,337 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:01,338 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:01,338 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:01,339 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 8EF3D351182B4FC9B104498D5B49B7BB, for 1 inputs
2016-01-14 15:22:01,340 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:01,340 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:01,341 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,342 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7f814a00; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1218766
2016-01-14 15:22:01,371 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,371 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A25E8F44D00D4AD7A993BB5C20FE0E40, mem on close (mb), free: 81, total: 516, max: 672
2016-01-14 15:22:01,371 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C58271B0BFDE40AEA813831159DCF987
2016-01-14 15:22:01,372 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,372 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,372 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,372 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,373 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01,373 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,373 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,373 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:01,374 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:01,394 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,398 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,400 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_01, eventCount=1
2016-01-14 15:22:01,402 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,404 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10]]
2016-01-14 15:22:01,404 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskAttemptId=attempt_1452784920806_0001_1_01_000000_0, startTime=1452784921232, finishTime=1452784921402, timeTaken=170, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10
2016-01-14 15:22:01,404 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:01,404 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,404 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,405 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_01_000000_0
2016-01-14 15:22:01,406 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,406 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, taskId=task_1452784920806_0001_1_01_000000, startTime=1452784921232, finishTime=1452784921405, timeTaken=173, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10
2016-01-14 15:22:01,406 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,407 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] : 1
2016-01-14 15:22:01,407 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,405 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,405 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,410 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, vertexId=vertex_1452784920806_0001_1_01, initRequestedTime=1452784921029, initedTime=1452784921042, startRequestedTime=1452784921080, startedTime=1452784921080, finishTime=1452784921407, timeTaken=327, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921342, Process_Duration=29, Process_End_Time=1452784921371, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=10, vertexStats=firstTaskStartTime=1452784921232, firstTasksToStart=[ task_1452784920806_0001_1_01_000000 ], lastTaskFinishTime=1452784921402, lastTasksToFinish=[ task_1452784920806_0001_1_01_000000 ], minTaskDuration=170, maxTaskDuration=170, avgTaskDuration=170.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_01_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,410 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,410 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,413 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] attempt: attempt_1452784920806_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,413 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,414 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,414 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,415 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,415 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000002, stoppedTime=1452784921415, exitStatus=0
2016-01-14 15:22:01,416 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,416 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,417 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskId=task_1452784920806_0001_1_02_000000, scheduledTime=1452784921417, launchTime=1452784921417
2016-01-14 15:22:01,417 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,417 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,418 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,418 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,419 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,419 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,419 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,420 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000002, Exiting
2016-01-14 15:22:01,420 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,420 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000002
2016-01-14 15:22:01,420 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: D4533A9745A740C398F52479AA7C29A1, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,420 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,420 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,421 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,421 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000002 completed successfully
2016-01-14 15:22:01,421 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,422 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000003
2016-01-14 15:22:01,423 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,426 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784920806_0001_00_000003
2016-01-14 15:22:01,428 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:01,430 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,430 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,431 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000003, launchTime=1452784921430
2016-01-14 15:22:01,432 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,433 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_02_000000_0] to container: [container_1452784920806_0001_00_000003]
2016-01-14 15:22:01,433 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,434 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000003 given task: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,434 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,435 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_02_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,435 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,435 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, startTime=1452784921434, containerId=container_1452784920806_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:01,436 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,436 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,437 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,437 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,437 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,437 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,438 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,438 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,438 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,438 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: D4533A9745A740C398F52479AA7C29A1, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:01,439 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,441 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,443 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:01,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A25E8F44D00D4AD7A993BB5C20FE0E40, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B383C94B7B3947F0A2BD91A29DE0815D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:01,446 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,446 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,446 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,448 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,448 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,448 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,449 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,449 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:01,449 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,450 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,453 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,464 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D4533A9745A740C398F52479AA7C29A1, ordinal: 2
2016-01-14 15:22:01,464 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4533A9745A740C398F52479AA7C29A1, mem on start (mb), free: 73, total: 516, max: 672
2016-01-14 15:22:01,464 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,465 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,466 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,466 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:01,467 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,467 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:01,467 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A25E8F44D00D4AD7A993BB5C20FE0E40, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:01,468 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B383C94B7B3947F0A2BD91A29DE0815D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:01,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A25E8F44D00D4AD7A993BB5C20FE0E40 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,470 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,470 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,471 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,472 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A25E8F44D00D4AD7A993BB5C20FE0E40 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,472 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:01,472 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,473 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,475 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,475 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,475 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,477 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_01_000000_0_10017, runDuration: 151311, ]
2016-01-14 15:22:01,478 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_01_000000_0_10017] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784921478, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:22:01,479 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,479 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:01,480 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A25E8F44D00D4AD7A993BB5C20FE0E40] #5 in 3ms
2016-01-14 15:22:01,480 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-01-14 15:22:01,480 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A25E8F44D00D4AD7A993BB5C20FE0E40] #5, status:false, isInterrupted:false
2016-01-14 15:22:01,481 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,482 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: CDC24F3386C2421B89475874E56F482D
2016-01-14 15:22:01,482 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,482 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:01,482 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,482 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,484 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,484 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@59d75340; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@479abcd4; comparator=cascading.tuple.hadoop.util.TupleComparator@7fe32705; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:01,486 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:01,486 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:01,487 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,487 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:01,487 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,549 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:01,549 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,550 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:01,550 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:01,550 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:01,550 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:01,551 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CDC24F3386C2421B89475874E56F482D, for 1 inputs
2016-01-14 15:22:01,552 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D4533A9745A740C398F52479AA7C29A1, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,553 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3b3bb7fb; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4dabde1e
2016-01-14 15:22:01,573 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,573 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4533A9745A740C398F52479AA7C29A1, mem on close (mb), free: 70, total: 580, max: 672
2016-01-14 15:22:01,573 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A25E8F44D00D4AD7A993BB5C20FE0E40
2016-01-14 15:22:01,573 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,574 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,574 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,574 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,574 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01,574 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:01,575 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:01,575 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 15:22:01,575 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:01,579 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,584 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,584 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784920806_0001_1_02, eventCount=1
2016-01-14 15:22:01,585 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskAttemptId=attempt_1452784920806_0001_1_02_000000_0, startTime=1452784921434, finishTime=1452784921585, timeTaken=151, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:01,586 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,586 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 15:22:01,587 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,587 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:01,587 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_02_000000_0
2016-01-14 15:22:01,589 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,589 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, taskId=task_1452784920806_0001_1_02_000000, startTime=1452784921434, finishTime=1452784921588, timeTaken=154, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:01,590 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,590 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] : 1
2016-01-14 15:22:01,590 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,588 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=D4533A9745A740C398F52479AA7C29A1, vertexId=vertex_1452784920806_0001_1_02, initRequestedTime=1452784921043, initedTime=1452784921049, startRequestedTime=1452784921083, startedTime=1452784921083, finishTime=1452784921590, timeTaken=507, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921552, Process_Duration=21, Process_End_Time=1452784921573, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452784921434, firstTasksToStart=[ task_1452784920806_0001_1_02_000000 ], lastTaskFinishTime=1452784921585, lastTasksToFinish=[ task_1452784920806_0001_1_02_000000 ], minTaskDuration=151, maxTaskDuration=151, avgTaskDuration=151.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_02_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,593 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,593 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,594 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000003, stoppedTime=1452784921593, exitStatus=0
2016-01-14 15:22:01,594 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,594 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] attempt: attempt_1452784920806_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:01,593 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,594 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,594 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,595 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,594 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B383C94B7B3947F0A2BD91A29DE0815D
2016-01-14 15:22:01,595 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,595 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,596 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784920806_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:01,596 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000003, Exiting
2016-01-14 15:22:01,596 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,596 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_STARTED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskId=task_1452784920806_0001_1_03_000000, scheduledTime=1452784921596, launchTime=1452784921596
2016-01-14 15:22:01,596 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000003
2016-01-14 15:22:01,597 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:01,598 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:01,598 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,598 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000003 completed successfully
2016-01-14 15:22:01,599 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,599 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01,599 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: B383C94B7B3947F0A2BD91A29DE0815D, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:01,599 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:01,600 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:01,602 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784920806_0001_00_000004
2016-01-14 15:22:01,603 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:01,599 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:01,605 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784920806_0001_00_000004
2016-01-14 15:22:01,606 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:01,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784920806_0001_00_000004, launchTime=1452784921608
2016-01-14 15:22:01,608 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:01,608 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:01,608 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,610 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784920806_0001_1_03_000000_0] to container: [container_1452784920806_0001_00_000004]
2016-01-14 15:22:01,610 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:01,611 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784920806_0001_00_000004 given task: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,611 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01 Starting to run new task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,611 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,612 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,612 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:01,612 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:01,612 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:01,612 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:01,612 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:01,613 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784920806_0001_1_03_000000_0] started. Is using containerId: [container_1452784920806_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:01,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729] pearson/(1/1) ...mposite/pearson/composite, VertexName: B383C94B7B3947F0A2BD91A29DE0815D, VertexParallelism: 1, TaskAttemptID:attempt_1452784920806_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:01,613 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, startTime=1452784921612, containerId=container_1452784920806_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784920806_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:01,613 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:01,613 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:01,613 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:01,614 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:01,614 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:01,617 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,616 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784920806_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:01,618 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,618 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=AA5BC7EE4A07438280C31731A446F7FB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:01,620 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:01,620 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,621 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:01,621 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,623 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:01,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D4533A9745A740C398F52479AA7C29A1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:01,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:01,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,631 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:01,632 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:01,632 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,634 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,641 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:01,642 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,649 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, ordinal: 3
2016-01-14 15:22:01,649 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, mem on start (mb), free: 64, total: 580, max: 672
2016-01-14 15:22:01,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:01,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:01,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:01,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:01,651 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:01,652 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:01,652 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:01,653 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:01,653 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:01,654 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:01,654 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, AA5BC7EE4A07438280C31731A446F7FB, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:01,654 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D4533A9745A740C398F52479AA7C29A1, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:01,655 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:01,655 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,656 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:01,656 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:01,657 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D4533A9745A740C398F52479AA7C29A1 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:01,657 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:01,657 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:01,657 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:01,658 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D4533A9745A740C398F52479AA7C29A1 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:01,659 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:01,662 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:01,662 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:01,663 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,663 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784920806_0001_1_02_000000_0_10021, runDuration: 135175, ]
2016-01-14 15:22:01,665 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784920806_0001_1_02_000000_0_10021] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452784921665, TimeTaken=1, Rate=4.14 MB/s
2016-01-14 15:22:01,665 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,666 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:01,666 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D4533A9745A740C398F52479AA7C29A1] #6 in 2ms
2016-01-14 15:22:01,666 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D4533A9745A740C398F52479AA7C29A1] #6, status:false, isInterrupted:false
2016-01-14 15:22:01,667 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:01,667 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 15:22:01,667 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,667 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:01,668 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 15:22:01,668 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,668 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:01,669 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-01-14 15:22:01,670 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: 63CCBAE3A50F4E7A8F917368CBE80CF6
2016-01-14 15:22:01,671 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: AA5BC7EE4A07438280C31731A446F7FB
2016-01-14 15:22:01,671 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 15:22:01,671 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 63CCBAE3A50F4E7A8F917368CBE80CF6, for 1 inputs
2016-01-14 15:22:01,672 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:01,673 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@23b2412a; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@bd43eb5
2016-01-14 15:22:01,692 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:01,692 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:01,693 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,693 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784920806_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:01,694 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,695 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B383C94B7B3947F0A2BD91A29DE0815D, mem on close (mb), free: 60, total: 580, max: 672
2016-01-14 15:22:01,695 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D4533A9745A740C398F52479AA7C29A1
2016-01-14 15:22:01,695 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:01,696 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:01,696 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:01,696 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:01,697 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:01 Completed running task attempt: attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,698 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:01,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskAttemptId=attempt_1452784920806_0001_1_03_000000_0, startTime=1452784921612, finishTime=1452784921699, timeTaken=87, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:01,700 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:01,700 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784920806_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:01,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:01,701 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784920806_0001_1_03_000000_0
2016-01-14 15:22:01,701 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:01,701 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,701 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,701 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:01,702 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:TASK_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, taskId=task_1452784920806_0001_1_03_000000, startTime=1452784921612, finishTime=1452784921701, timeTaken=89, status=SUCCEEDED, successfulAttemptID=attempt_1452784920806_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:01,702 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:01,703 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784920806_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:01,703 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] : 1
2016-01-14 15:22:01,703 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,705 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:01,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:VERTEX_FINISHED]: vertexName=B383C94B7B3947F0A2BD91A29DE0815D, vertexId=vertex_1452784920806_0001_1_03, initRequestedTime=1452784921056, initedTime=1452784921073, startRequestedTime=1452784921092, startedTime=1452784921092, finishTime=1452784921704, timeTaken=612, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784921672, Process_Duration=22, Process_End_Time=1452784921694, Read_Duration=10, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784921612, firstTasksToStart=[ task_1452784920806_0001_1_03_000000 ], lastTaskFinishTime=1452784921699, lastTasksToFinish=[ task_1452784920806_0001_1_03_000000 ], minTaskDuration=87, maxTaskDuration=87, avgTaskDuration=87.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784920806_0001_1_03_000000 ], longestDurationTasks=[ task_1452784920806_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:01,706 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:01,707 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:01,707 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:01,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784920806_0001_00_000004, stoppedTime=1452784921707, exitStatus=0
2016-01-14 15:22:01,708 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:01,708 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:01,708 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784920806_0001_1
2016-01-14 15:22:01,709 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: AA5BC7EE4A07438280C31731A446F7FB for vertex: vertex_1452784920806_0001_1_03 [B383C94B7B3947F0A2BD91A29DE0815D]
2016-01-14 15:22:01,711 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784920806_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:01,712 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784920806_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:01,712 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784920806_0001_00_000004, Exiting
2016-01-14 15:22:01,712 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,712 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784920806_0001_00_000004
2016-01-14 15:22:01,713 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784920806_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:01,713 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784920806_0001_00_000004 completed successfully
2016-01-14 15:22:01,723 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_01 [A25E8F44D00D4AD7A993BB5C20FE0E40]
2016-01-14 15:22:01,723 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_02 [D4533A9745A740C398F52479AA7C29A1]
2016-01-14 15:22:01,723 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784920806_0001_1_00 [C58271B0BFDE40AEA813831159DCF987]
2016-01-14 15:22:01,728 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784920806_0001_1, queueSize=0
2016-01-14 15:22:01,728 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:01,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784920806_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784920806_0001_1, startTime=1452784921025, finishTime=1452784921723, timeTaken=698, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14417, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2299527168, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=42, FIRST_EVENT_RECEIVED=23, LAST_EVENT_RECEIVED=23, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139685750, Process_Duration=77, Process_End_Time=5811139685827, Read_Duration=20, Tuples_Read=303, Tuples_Written=317, Write_Duration=16, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:01,730 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784920806_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:01,731 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784920806_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:01,731 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784920806_0001_00_000004
2016-01-14 15:22:01,732 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784920806_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:01 Completed Dag: dag_1452784920806_0001_1
2016-01-14 15:22:01,733 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784920806_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:01,734 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784920806_0001_1_post
2016-01-14 15:22:01,734 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:01,735 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:01,740 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452784920806_0001
2016-01-14 15:22:01,742 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:01,742 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:01,742 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:01,743 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:01,743 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:01,744 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:01,744 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:01,745 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:01,745 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:01,745 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:01,745 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:01,746 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:01,746 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:01,752 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/A9AA42A0600D4AB383ABCF8ED86D350A/E2159E23F5F445FDBF218D8CB19CF729/.tez/application_1452784920806_0001
2016-01-14 15:22:01,754 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] completed in: 00:00.968
2016-01-14 15:22:01,862 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 15:22:01,893 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:01,894 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:01,975 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.053
2016-01-14 15:22:02,007 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.085
2016-01-14 15:22:02,008 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:02,009 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:02,009 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:02,030 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:02,056 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 15:22:02,198 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:22:02,198 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,199 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 15:22:02,199 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 15:22:02,200 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 15:22:02,200 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 15:22:02,201 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,201 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:02,215 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:22:02,220 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:02,221 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:02,221 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:02,231 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001 doesn't exist and is created
2016-01-14 15:22:02,249 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:02,249 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:02,250 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001
2016-01-14 15:22:02,253 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784922220_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:02,266 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:02,267 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:02,269 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:02,269 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:02,269 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir
2016-01-14 15:22:02,270 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/history.txt.appattempt_1452784922220_0001_000000, maxErrors=10
2016-01-14 15:22:02,270 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:02,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784922220_0001_000000, appSubmitTime=1452784922253, launchTime=1452784922253
2016-01-14 15:22:02,278 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:02,282 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:02,283 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:02,283 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:02,287 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:37104
2016-01-14 15:22:02,291 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784922220_0001_000000, startTime=1452784922294
2016-01-14 15:22:02,294 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:02,295 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:02,349 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:02,350 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:02,350 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001, dagName=[BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,371 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,374 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784922220_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/dag_1452784922220_0001_1.dot
2016-01-14 15:22:02,376 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001/localmode-log-dir/dag_1452784922220_0001_1-tez-dag.pb.txt
2016-01-14 15:22:02,406 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784922220_0001_1
2016-01-14 15:22:02,407 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,407 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02 Running Dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,422 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784922220_0001_1, submitTime=1452784922371
2016-01-14 15:22:02,422 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:02,424 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,424 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:02,424 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784922220_0001_1, initTime=1452784922422
2016-01-14 15:22:02,424 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:02,425 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:02,427 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_STARTED]: dagID=dag_1452784922220_0001_1, startTime=1452784922426
2016-01-14 15:22:02,427 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:02,427 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:02,427 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:02,427 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: AFDC1534F21B447DA4FFF2514883B001 : {812ABCEB2A5C4C0D9BF844260CD28C87={InputName=812ABCEB2A5C4C0D9BF844260CD28C87}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:02,427 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 812ABCEB2A5C4C0D9BF844260CD28C87, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:02,428 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,428 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]. Starting root input initializers: 1
2016-01-14 15:22:02,429 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:02,429 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,431 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001, dagName=[BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 15:22:02,429 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 812ABCEB2A5C4C0D9BF844260CD28C87 on vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,436 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:02,437 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 812ABCEB2A5C4C0D9BF844260CD28C87 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:02,439 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,441 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,441 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,442 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,442 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CF650B439E834FF6836A619C942F1448, vertexId=vertex_1452784922220_0001_1_01, initRequestedTime=1452784922429, initedTime=1452784922442, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,443 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,444 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,444 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:02,447 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:02,450 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,450 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,451 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:02,451 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,452 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 812ABCEB2A5C4C0D9BF844260CD28C87 on vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,453 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,454 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, vertexId=vertex_1452784922220_0001_1_02, initRequestedTime=1452784922444, initedTime=1452784922452, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,454 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,455 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:02,455 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {812ABCEB2A5C4C0D9BF844260CD28C87=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:02,456 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] parallelism set to 1
2016-01-14 15:22:02,457 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:02,457 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_00, eventCount=1
2016-01-14 15:22:02,458 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784922220_0001, with dag id: dag_1452784922220_0001_1
2016-01-14 15:22:02,459 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,460 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, vertexId=vertex_1452784922220_0001_1_00, initRequestedTime=1452784922427, initedTime=1452784922458, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:02,460 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] with distanceFromRoot: 0
2016-01-14 15:22:02,461 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:02,461 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,464 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:02,465 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,465 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,465 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,466 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:02,466 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,468 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexName=B1158DCDB6CE43E7A3D082659164EA2A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:02,470 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=8F3D3AC0D03D441AAB2B30771DE4FED8, vertexId=vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,472 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:02,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, vertexId=vertex_1452784922220_0001_1_03, initRequestedTime=1452784922461, initedTime=1452784922472, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:02,473 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:02,473 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,474 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,475 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_00, startRequestedTime=1452784922473, startedTime=1452784922473
2016-01-14 15:22:02,475 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,475 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,476 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskId=task_1452784922220_0001_1_00_000000, scheduledTime=1452784922475, launchTime=1452784922475
2016-01-14 15:22:02,476 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,476 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_00 for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,476 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,478 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] with distanceFromRoot: 1
2016-01-14 15:22:02,479 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,479 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: AFDC1534F21B447DA4FFF2514883B001 in vertex: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,480 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: CF650B439E834FF6836A619C942F1448 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,480 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_01, startRequestedTime=1452784922479, startedTime=1452784922479
2016-01-14 15:22:02,480 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,480 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,480 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: AFDC1534F21B447DA4FFF2514883B001, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,482 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,482 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_01 for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,486 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] with distanceFromRoot: 2
2016-01-14 15:22:02,486 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,486 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: CF650B439E834FF6836A619C942F1448 in vertex: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,487 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D4F1C81073004F1EA40841A2F7C36F50 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,488 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:02,490 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,490 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_02, startRequestedTime=1452784922486, startedTime=1452784922486
2016-01-14 15:22:02,492 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,493 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:02,494 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:02,495 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000001
2016-01-14 15:22:02,495 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,496 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784922220_0001_1_02 for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] numStartedSources: 1 numSources: 1
2016-01-14 15:22:02,496 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] with distanceFromRoot: 3
2016-01-14 15:22:02,498 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784922220_0001_00_000001
2016-01-14 15:22:02,498 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:02,498 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000001, launchTime=1452784922498
2016-01-14 15:22:02,500 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,500 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D4F1C81073004F1EA40841A2F7C36F50 in vertex: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,500 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B1158DCDB6CE43E7A3D082659164EA2A with 1 source tasks and 1 pending tasks
2016-01-14 15:22:02,500 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:02,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784922220_0001_1_03, startRequestedTime=1452784922500, startedTime=1452784922500
2016-01-14 15:22:02,501 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:02,501 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,510 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_00_000000_0] to container: [container_1452784922220_0001_00_000001]
2016-01-14 15:22:02,511 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,511 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000001 given task: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,511 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,511 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_00_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,512 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,512 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,512 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,512 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,512 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,512 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,513 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,513 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: AFDC1534F21B447DA4FFF2514883B001, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,513 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,513 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,513 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,513 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, startTime=1452784922511, containerId=container_1452784922220_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:02,513 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,514 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,515 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=812ABCEB2A5C4C0D9BF844260CD28C87, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:02,514 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,515 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,515 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,517 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:02,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,518 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,525 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,525 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,525 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,526 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:02,526 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,526 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,527 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,527 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,529 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,529 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,534 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, ordinal: 0
2016-01-14 15:22:02,534 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, mem on start (mb), free: 134, total: 570, max: 672
2016-01-14 15:22:02,534 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,534 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,535 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,535 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,535 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,535 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:02,536 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:02,536 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,536 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,536 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:02,536 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 812ABCEB2A5C4C0D9BF844260CD28C87, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:02,537 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, CF650B439E834FF6836A619C942F1448, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:02,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 812ABCEB2A5C4C0D9BF844260CD28C87 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,537 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:02,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,538 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,538 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:02,539 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@5b7e588b
2016-01-14 15:22:02,540 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@5b7e588b
2016-01-14 15:22:02,540 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:02,541 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:02,541 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,542 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 812ABCEB2A5C4C0D9BF844260CD28C87
2016-01-14 15:22:02,542 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,542 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,543 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6fd85a65; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6516d640; comparator=cascading.tuple.hadoop.util.TupleComparator@3e4c3799; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,544 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,544 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,544 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,580 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:02,581 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:02,581 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:02,581 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:02,581 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:02,582 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:02,582 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:02,582 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,586 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,586 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFDC1534F21B447DA4FFF2514883B001, mem on close (mb), free: 161, total: 580, max: 672
2016-01-14 15:22:02,587 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,587 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,587 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:02,587 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:02,590 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,594 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,595 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_00, eventCount=1
2016-01-14 15:22:02,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskAttemptId=attempt_1452784922220_0001_1_00_000000_0, startTime=1452784922511, finishTime=1452784922595, timeTaken=84, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:02,596 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,596 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,596 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,596 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_00_000000_0
2016-01-14 15:22:02,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, taskId=task_1452784922220_0001_1_00_000000, startTime=1452784922511, finishTime=1452784922596, timeTaken=85, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:02,598 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,599 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] : 1
2016-01-14 15:22:02,599 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,599 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:02,599 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:02,600 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,600 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,600 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,602 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=AFDC1534F21B447DA4FFF2514883B001, vertexId=vertex_1452784922220_0001_1_00, initRequestedTime=1452784922427, initedTime=1452784922458, startRequestedTime=1452784922473, startedTime=1452784922473, finishTime=1452784922599, timeTaken=126, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922583, Process_Duration=3, Process_End_Time=1452784922586, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784922511, firstTasksToStart=[ task_1452784922220_0001_1_00_000000 ], lastTaskFinishTime=1452784922595, lastTasksToFinish=[ task_1452784922220_0001_1_00_000000 ], minTaskDuration=84, maxTaskDuration=84, avgTaskDuration=84.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_00_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,603 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,603 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,603 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,605 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000001, stoppedTime=1452784922605, exitStatus=0
2016-01-14 15:22:02,605 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,606 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] attempt: attempt_1452784922220_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,606 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,607 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,608 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,608 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,609 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,609 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=CF650B439E834FF6836A619C942F1448, taskId=task_1452784922220_0001_1_01_000000, scheduledTime=1452784922609, launchTime=1452784922609
2016-01-14 15:22:02,610 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,610 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,610 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,611 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: CF650B439E834FF6836A619C942F1448, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,611 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,611 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,611 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,612 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,612 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000001, Exiting
2016-01-14 15:22:02,612 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,612 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000001
2016-01-14 15:22:02,619 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000002
2016-01-14 15:22:02,620 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,623 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784922220_0001_00_000002
2016-01-14 15:22:02,624 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,629 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,629 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000001 completed successfully
2016-01-14 15:22:02,629 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,629 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000002, launchTime=1452784922629
2016-01-14 15:22:02,630 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,630 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,632 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,632 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_01_000000_0] to container: [container_1452784922220_0001_00_000002]
2016-01-14 15:22:02,633 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,634 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_01_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,634 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CF650B439E834FF6836A619C942F1448, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, startTime=1452784922633, containerId=container_1452784922220_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:02,634 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,634 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,635 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,637 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000002 given task: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,637 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,638 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,638 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,638 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,638 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,638 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,638 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,639 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,639 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: CF650B439E834FF6836A619C942F1448, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,639 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,641 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,642 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,642 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,642 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=AFDC1534F21B447DA4FFF2514883B001, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,645 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,646 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,648 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,650 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,650 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,647 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,650 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,650 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,664 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CF650B439E834FF6836A619C942F1448, ordinal: 1
2016-01-14 15:22:02,664 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF650B439E834FF6836A619C942F1448, mem on start (mb), free: 154, total: 580, max: 672
2016-01-14 15:22:02,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,664 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,665 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,667 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,667 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,668 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:02,668 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D4F1C81073004F1EA40841A2F7C36F50, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:02,668 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, AFDC1534F21B447DA4FFF2514883B001, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:02,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: AFDC1534F21B447DA4FFF2514883B001 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,671 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,672 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,672 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: AFDC1534F21B447DA4FFF2514883B001 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,672 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:02,672 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,673 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,674 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,674 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,674 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,674 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_00_000000_0_10027, runDuration: 65666, ]
2016-01-14 15:22:02,677 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_00_000000_0_10027] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784922677, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:02,678 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,678 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 15:22:02,678 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:02,679 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [AFDC1534F21B447DA4FFF2514883B001] #7 in 3ms
2016-01-14 15:22:02,679 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [AFDC1534F21B447DA4FFF2514883B001] #7, status:false, isInterrupted:false
2016-01-14 15:22:02,679 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 8AEE460591F24356B3FFAF9C3C396535
2016-01-14 15:22:02,679 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,680 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,680 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,681 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:02,681 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,681 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,681 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:02,681 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,680 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@2f0d2baf; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@9af064b; comparator=cascading.tuple.hadoop.util.TupleComparator@41c62ae4; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,684 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,685 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,685 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,682 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,693 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:02,693 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:02,694 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:02,694 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:02,694 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:02,694 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 8AEE460591F24356B3FFAF9C3C396535, for 1 inputs
2016-01-14 15:22:02,695 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:02,695 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:02,695 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CF650B439E834FF6836A619C942F1448, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,696 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@13d730fc; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@14d0fa91
2016-01-14 15:22:02,724 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,724 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF650B439E834FF6836A619C942F1448, mem on close (mb), free: 87, total: 580, max: 672
2016-01-14 15:22:02,724 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: AFDC1534F21B447DA4FFF2514883B001
2016-01-14 15:22:02,724 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,725 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,725 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,725 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,725 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02,725 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,725 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,725 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:02,726 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:02,738 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,743 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,744 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_01, eventCount=1
2016-01-14 15:22:02,745 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, taskAttemptId=attempt_1452784922220_0001_1_01_000000_0, startTime=1452784922633, finishTime=1452784922745, timeTaken=112, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 15:22:02,745 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,746 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,745 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-01-14 15:22:02,747 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,747 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:02,748 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,748 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,746 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_01_000000_0
2016-01-14 15:22:02,749 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,749 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, taskId=task_1452784922220_0001_1_01_000000, startTime=1452784922633, finishTime=1452784922748, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 15:22:02,749 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,750 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] : 1
2016-01-14 15:22:02,750 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,751 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=CF650B439E834FF6836A619C942F1448, vertexId=vertex_1452784922220_0001_1_01, initRequestedTime=1452784922429, initedTime=1452784922442, startRequestedTime=1452784922479, startedTime=1452784922479, finishTime=1452784922750, timeTaken=271, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608698368, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922695, Process_Duration=29, Process_End_Time=1452784922724, Read_Duration=4, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1452784922633, firstTasksToStart=[ task_1452784922220_0001_1_01_000000 ], lastTaskFinishTime=1452784922745, lastTasksToFinish=[ task_1452784922220_0001_1_01_000000 ], minTaskDuration=112, maxTaskDuration=112, avgTaskDuration=112.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_01_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,752 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,752 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,752 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000002, stoppedTime=1452784922752, exitStatus=0
2016-01-14 15:22:02,752 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,752 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,753 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] attempt: attempt_1452784922220_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,753 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,753 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,753 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,753 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,754 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskId=task_1452784922220_0001_1_02_000000, scheduledTime=1452784922754, launchTime=1452784922754
2016-01-14 15:22:02,754 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,754 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,754 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,755 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,755 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D4F1C81073004F1EA40841A2F7C36F50, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,755 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,756 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,756 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000003
2016-01-14 15:22:02,757 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,759 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784922220_0001_00_000003
2016-01-14 15:22:02,759 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - TezTask metrics system already initialized!
2016-01-14 15:22:02,760 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,760 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,760 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,760 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,760 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000002, Exiting
2016-01-14 15:22:02,760 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,761 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000002
2016-01-14 15:22:02,764 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,764 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000002 completed successfully
2016-01-14 15:22:02,764 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,764 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000003, launchTime=1452784922760
2016-01-14 15:22:02,765 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,766 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_02_000000_0] to container: [container_1452784922220_0001_00_000003]
2016-01-14 15:22:02,766 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,767 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000003 given task: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,767 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,767 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_02_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,768 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,769 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, startTime=1452784922767, containerId=container_1452784922220_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:02,769 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,769 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,770 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,770 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,770 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,770 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,770 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,771 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,771 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,771 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D4F1C81073004F1EA40841A2F7C36F50, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:02,771 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,778 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CF650B439E834FF6836A619C942F1448, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B1158DCDB6CE43E7A3D082659164EA2A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:02,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,785 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:02,785 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,786 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,786 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,781 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:02,788 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,788 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,789 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,789 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,799 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, ordinal: 2
2016-01-14 15:22:02,799 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, mem on start (mb), free: 81, total: 580, max: 672
2016-01-14 15:22:02,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,800 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:02,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,802 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:02,802 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B1158DCDB6CE43E7A3D082659164EA2A, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:02,802 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CF650B439E834FF6836A619C942F1448, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:02,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CF650B439E834FF6836A619C942F1448 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,807 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,807 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,809 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: CF650B439E834FF6836A619C942F1448 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,809 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:02,809 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,810 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,812 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,812 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,814 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_01_000000_0_10030, runDuration: 92362, ]
2016-01-14 15:22:02,815 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:22:02,816 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 6CA27EF2C8F0401681041459C024470F
2016-01-14 15:22:02,817 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,817 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_01_000000_0_10030] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784922817, TimeTaken=2, Rate=3.59 MB/s
2016-01-14 15:22:02,817 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,817 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:02,817 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [CF650B439E834FF6836A619C942F1448] #8 in 3ms
2016-01-14 15:22:02,817 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,821 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@164657cc; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@642e1d80; comparator=cascading.tuple.hadoop.util.TupleComparator@f92f1e2; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:02,822 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [CF650B439E834FF6836A619C942F1448] #8, status:false, isInterrupted:false
2016-01-14 15:22:02,822 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:02,824 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,825 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:02,825 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,825 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,825 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:02,826 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,826 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:02,826 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,835 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,844 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:02,844 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:02,844 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:02,844 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:02,845 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:02,846 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 6CA27EF2C8F0401681041459C024470F, for 1 inputs
2016-01-14 15:22:02,847 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,847 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@120a2a74; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@59692eec
2016-01-14 15:22:02,867 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,867 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F1C81073004F1EA40841A2F7C36F50, mem on close (mb), free: 201, total: 580, max: 672
2016-01-14 15:22:02,867 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: CF650B439E834FF6836A619C942F1448
2016-01-14 15:22:02,868 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,868 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,868 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,868 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,869 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02,869 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:02,869 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:02,869 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 15:22:02,869 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:02,874 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,880 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784922220_0001_1_02, eventCount=1
2016-01-14 15:22:02,882 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 15:22:02,883 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,883 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:02,884 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,884 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskAttemptId=attempt_1452784922220_0001_1_02_000000_0, startTime=1452784922767, finishTime=1452784922883, timeTaken=116, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:22:02,884 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,885 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,885 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,885 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_02_000000_0
2016-01-14 15:22:02,885 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,886 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, taskId=task_1452784922220_0001_1_02_000000, startTime=1452784922767, finishTime=1452784922885, timeTaken=118, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 15:22:02,886 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,886 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] : 1
2016-01-14 15:22:02,886 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,887 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=D4F1C81073004F1EA40841A2F7C36F50, vertexId=vertex_1452784922220_0001_1_02, initRequestedTime=1452784922444, initedTime=1452784922452, startRequestedTime=1452784922486, startedTime=1452784922486, finishTime=1452784922886, timeTaken=400, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922847, Process_Duration=20, Process_End_Time=1452784922867, Read_Duration=5, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452784922767, firstTasksToStart=[ task_1452784922220_0001_1_02_000000 ], lastTaskFinishTime=1452784922883, lastTasksToFinish=[ task_1452784922220_0001_1_02_000000 ], minTaskDuration=116, maxTaskDuration=116, avgTaskDuration=116.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_02_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,888 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,888 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000003, stoppedTime=1452784922888, exitStatus=0
2016-01-14 15:22:02,889 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,889 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,889 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] attempt: attempt_1452784922220_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:02,889 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B1158DCDB6CE43E7A3D082659164EA2A
2016-01-14 15:22:02,889 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,890 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,890 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784922220_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:02,891 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:02,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_STARTED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskId=task_1452784922220_0001_1_03_000000, scheduledTime=1452784922890, launchTime=1452784922890
2016-01-14 15:22:02,891 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:02,891 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,892 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:02,892 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B1158DCDB6CE43E7A3D082659164EA2A, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:02,893 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:02,892 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,893 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:02,893 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,894 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784922220_0001_00_000004
2016-01-14 15:22:02,894 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000003, Exiting
2016-01-14 15:22:02,895 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,895 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000003
2016-01-14 15:22:02,895 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:02,896 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,896 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000003 completed successfully
2016-01-14 15:22:02,896 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02,897 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784922220_0001_00_000004
2016-01-14 15:22:02,898 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:02,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784922220_0001_00_000004, launchTime=1452784922900
2016-01-14 15:22:02,900 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:02,900 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:02,901 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,902 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784922220_0001_1_03_000000_0] to container: [container_1452784922220_0001_00_000004]
2016-01-14 15:22:02,902 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:02,903 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784922220_0001_00_000004 given task: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,903 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784922220_0001_1_03_000000_0] started. Is using containerId: [container_1452784922220_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:02,903 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,903 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:02,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, startTime=1452784922903, containerId=container_1452784922220_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784922220_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:02 Starting to run new task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,904 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,904 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:02,904 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:02,904 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,904 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:02,905 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:02,905 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:02,905 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:02,905 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:02,905 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B1158DCDB6CE43E7A3D082659164EA2A, VertexParallelism: 1, TaskAttemptID:attempt_1452784922220_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:02,905 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:02,906 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,908 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784922220_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:02,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:02,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D4F1C81073004F1EA40841A2F7C36F50, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:02,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:02,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=8F3D3AC0D03D441AAB2B30771DE4FED8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:02,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,913 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:02,913 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:02,913 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:02,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,915 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:02,915 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:02,915 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,928 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:02,928 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,929 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, ordinal: 3
2016-01-14 15:22:02,929 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, mem on start (mb), free: 194, total: 580, max: 672
2016-01-14 15:22:02,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:02,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:02,930 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:02,930 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:02,930 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:02,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:02,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:02,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:02,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:02,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:02,932 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D4F1C81073004F1EA40841A2F7C36F50, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:02,932 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 8F3D3AC0D03D441AAB2B30771DE4FED8, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:02,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:02,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D4F1C81073004F1EA40841A2F7C36F50 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:02,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:02,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:02,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,933 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:02,933 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:02,933 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D4F1C81073004F1EA40841A2F7C36F50 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:02,934 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:02,934 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:02,934 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:02,934 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,935 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:02,936 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,936 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784922220_0001_1_02_000000_0_10033, runDuration: 94648, ]
2016-01-14 15:22:02,939 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784922220_0001_1_02_000000_0_10033] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452784922939, TimeTaken=2, Rate=2.44 MB/s
2016-01-14 15:22:02,939 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,939 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:02,939 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D4F1C81073004F1EA40841A2F7C36F50] #9 in 2ms
2016-01-14 15:22:02,939 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D4F1C81073004F1EA40841A2F7C36F50] #9, status:false, isInterrupted:false
2016-01-14 15:22:02,940 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 15:22:02,942 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:02,942 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 15:22:02,943 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,943 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 654C236BC9EE44D88920076030074B14
2016-01-14 15:22:02,943 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:02,944 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: 8F3D3AC0D03D441AAB2B30771DE4FED8
2016-01-14 15:22:02,944 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 15:22:02,944 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 15:22:02,944 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 654C236BC9EE44D88920076030074B14, for 1 inputs
2016-01-14 15:22:02,945 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,945 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:02,945 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:02,946 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@5ac92449; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4b6d632f
2016-01-14 15:22:02,956 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:02,956 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:02,957 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,957 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784922220_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:02,958 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,958 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B1158DCDB6CE43E7A3D082659164EA2A, mem on close (mb), free: 191, total: 580, max: 672
2016-01-14 15:22:02,958 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D4F1C81073004F1EA40841A2F7C36F50
2016-01-14 15:22:02,959 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:02,959 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:02,959 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:02,960 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:02,960 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:02 Completed running task attempt: attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,961 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:02,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:02,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:02,963 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:02,963 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,963 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskAttemptId=attempt_1452784922220_0001_1_03_000000_0, startTime=1452784922903, finishTime=1452784922962, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:02,963 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,963 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784922220_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:02,964 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:02,964 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784922220_0001_1_03_000000_0
2016-01-14 15:22:02,964 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:02,964 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:TASK_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, taskId=task_1452784922220_0001_1_03_000000, startTime=1452784922903, finishTime=1452784922964, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1452784922220_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:02,965 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784922220_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:02,965 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] : 1
2016-01-14 15:22:02,965 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,966 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:VERTEX_FINISHED]: vertexName=B1158DCDB6CE43E7A3D082659164EA2A, vertexId=vertex_1452784922220_0001_1_03, initRequestedTime=1452784922461, initedTime=1452784922472, startRequestedTime=1452784922500, startedTime=1452784922500, finishTime=1452784922965, timeTaken=465, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784922945, Process_Duration=13, Process_End_Time=1452784922958, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784922903, firstTasksToStart=[ task_1452784922220_0001_1_03_000000 ], lastTaskFinishTime=1452784922962, lastTasksToFinish=[ task_1452784922220_0001_1_03_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784922220_0001_1_03_000000 ], longestDurationTasks=[ task_1452784922220_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:02,966 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:02,966 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:02,967 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:02,969 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784922220_0001_00_000004, stoppedTime=1452784922969, exitStatus=0
2016-01-14 15:22:02,969 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:02,969 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:02,969 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:02,969 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784922220_0001_1
2016-01-14 15:22:02,970 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 8F3D3AC0D03D441AAB2B30771DE4FED8 for vertex: vertex_1452784922220_0001_1_03 [B1158DCDB6CE43E7A3D082659164EA2A]
2016-01-14 15:22:02,974 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784922220_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:02,974 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784922220_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:02,974 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784922220_0001_00_000004, Exiting
2016-01-14 15:22:02,974 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,974 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784922220_0001_00_000004
2016-01-14 15:22:02,975 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784922220_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:02,975 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784922220_0001_00_000004 completed successfully
2016-01-14 15:22:02,980 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_00 [AFDC1534F21B447DA4FFF2514883B001]
2016-01-14 15:22:02,980 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_01 [CF650B439E834FF6836A619C942F1448]
2016-01-14 15:22:02,981 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784922220_0001_1_02 [D4F1C81073004F1EA40841A2F7C36F50]
2016-01-14 15:22:02,987 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784922220_0001_1, queueSize=0
2016-01-14 15:22:02,988 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:02,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784922220_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784922220_0001_1, startTime=1452784922426, finishTime=1452784922981, timeTaken=555, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=31, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2433744896, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139691070, Process_Duration=65, Process_End_Time=5811139691135, Read_Duration=13, Tuples_Read=303, Tuples_Written=317, Write_Duration=16, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:02,989 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784922220_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:02,990 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784922220_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:02,991 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784922220_0001_00_000004
2016-01-14 15:22:02,992 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784922220_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:02 Completed Dag: dag_1452784922220_0001_1
2016-01-14 15:22:02,995 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784922220_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:02,996 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784922220_0001_1_post
2016-01-14 15:22:02,996 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:02,997 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:03,001 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452784922220_0001
2016-01-14 15:22:03,003 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:03,003 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:03,003 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:03,003 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:03,003 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:03,004 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:03,004 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:03,005 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:03,005 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:03,005 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:03,006 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:03,007 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:03,007 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:03,014 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/BC5D476D7F0C47FF8AD5E906572A87F6/A377845BD16A4A27A30E8D415F4BFF3B/.tez/application_1452784922220_0001
2016-01-14 15:22:03,014 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:00.816
2016-01-14 15:22:03,092 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 15:22:03,123 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 15:22:03,124 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 15:22:03,200 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.050
2016-01-14 15:22:03,220 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.070
2016-01-14 15:22:03,221 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:03,221 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 15:22:03,221 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 15:22:03,242 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 15:22:03,264 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-01-14 15:22:03,403 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 15:22:03,404 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,404 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 15:22:03,404 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 15:22:03,405 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 15:22:03,405 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 15:22:03,406 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,406 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 15:22:03,418 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 15:22:03,423 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:03,424 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 15:22:03,424 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 15:22:03,433 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001 doesn't exist and is created
2016-01-14 15:22:03,448 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 15:22:03,448 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 15:22:03,449 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001
2016-01-14 15:22:03,453 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452784923423_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 15:22:03,464 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 15:22:03,465 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 15:22:03,466 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 15:22:03,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 15:22:03,467 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir
2016-01-14 15:22:03,467 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/history.txt.appattempt_1452784923423_0001_000000, maxErrors=10
2016-01-14 15:22:03,467 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 15:22:03,468 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452784923423_0001_000000, appSubmitTime=1452784923452, launchTime=1452784923452
2016-01-14 15:22:03,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 15:22:03,473 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 15:22:03,474 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 15:22:03,476 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 15:22:03,478 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:57866
2016-01-14 15:22:03,480 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 15:22:03,482 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452784923423_0001_000000, startTime=1452784923482
2016-01-14 15:22:03,482 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 15:22:03,482 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 15:22:03,549 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 15:22:03,549 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 15:22:03,549 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001, dagName=[C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,566 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,568 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452784923423_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/dag_1452784923423_0001_1.dot
2016-01-14 15:22:03,570 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001/localmode-log-dir/dag_1452784923423_0001_1-tez-dag.pb.txt
2016-01-14 15:22:03,610 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784923423_0001_1
2016-01-14 15:22:03,610 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,610 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03 Running Dag: dag_1452784923423_0001_1
2016-01-14 15:22:03,623 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452784923423_0001_1, submitTime=1452784923566
2016-01-14 15:22:03,623 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 15:22:03,625 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,625 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 15:22:03,625 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452784923423_0001_1, initTime=1452784923623
2016-01-14 15:22:03,625 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 15:22:03,626 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 15:22:03,627 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_STARTED]: dagID=dag_1452784923423_0001_1, startTime=1452784923627
2016-01-14 15:22:03,627 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 15:22:03,627 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 15:22:03,627 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 15:22:03,627 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: CDC852A364E64A83B81E0ED2B2C93103 : {333A4BC1528E493D9A03C012238A7AEC={InputName=333A4BC1528E493D9A03C012238A7AEC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 15:22:03,627 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 333A4BC1528E493D9A03C012238A7AEC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 15:22:03,628 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,628 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]. Starting root input initializers: 1
2016-01-14 15:22:03,629 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 15:22:03,629 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,630 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001, dagName=[C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 15:22:03,633 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 333A4BC1528E493D9A03C012238A7AEC on vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,634 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,634 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,635 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,635 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B969A35B407643359B20905A4B079125, vertexId=vertex_1452784923423_0001_1_01, initRequestedTime=1452784923629, initedTime=1452784923635, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,636 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,636 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,641 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 15:22:03,641 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 333A4BC1528E493D9A03C012238A7AEC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 15:22:03,642 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 15:22:03,643 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,643 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 15:22:03,644 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,644 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,645 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, vertexId=vertex_1452784923423_0001_1_02, initRequestedTime=1452784923636, initedTime=1452784923644, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,645 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,645 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,649 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 15:22:03,649 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,650 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 15:22:03,651 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,652 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452784923423_0001, with dag id: dag_1452784923423_0001_1
2016-01-14 15:22:03,650 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 15:22:03,652 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=66B5A676F67342598BB0CC462F2FC64C, vertexName=C55DC0153409441F8FC29EAB74C3F340, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 15:22:03,658 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=66B5A676F67342598BB0CC462F2FC64C, vertexId=vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:03,657 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 333A4BC1528E493D9A03C012238A7AEC on vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,659 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, vertexId=vertex_1452784923423_0001_1_03, initRequestedTime=1452784923645, initedTime=1452784923659, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 15:22:03,660 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from NEW to INITED due to event V_INIT
2016-01-14 15:22:03,659 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,660 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:03,660 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {333A4BC1528E493D9A03C012238A7AEC=forAllWorkUnits=true, update=[1]}
2016-01-14 15:22:03,661 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] parallelism set to 1
2016-01-14 15:22:03,663 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 15:22:03,664 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_00, eventCount=1
2016-01-14 15:22:03,665 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, vertexId=vertex_1452784923423_0001_1_00, initRequestedTime=1452784923627, initedTime=1452784923665, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 15:22:03,665 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] with distanceFromRoot: 0
2016-01-14 15:22:03,666 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 15:22:03,665 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 15:22:03,666 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,667 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,667 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,667 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_00, startRequestedTime=1452784923666, startedTime=1452784923666
2016-01-14 15:22:03,668 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,668 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,668 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskId=task_1452784923423_0001_1_00_000000, scheduledTime=1452784923668, launchTime=1452784923668
2016-01-14 15:22:03,669 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,669 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_00 for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,669 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] with distanceFromRoot: 1
2016-01-14 15:22:03,670 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,670 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: CDC852A364E64A83B81E0ED2B2C93103 in vertex: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,670 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B969A35B407643359B20905A4B079125 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,671 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_01, startRequestedTime=1452784923670, startedTime=1452784923670
2016-01-14 15:22:03,671 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,672 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,672 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: CDC852A364E64A83B81E0ED2B2C93103, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,673 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,673 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_01 for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,673 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] with distanceFromRoot: 2
2016-01-14 15:22:03,674 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,674 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B969A35B407643359B20905A4B079125 in vertex: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,674 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 15:22:03,674 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: FACEDF332B814CD28C9FB990F959C4C3 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,675 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,676 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_02, startRequestedTime=1452784923674, startedTime=1452784923674
2016-01-14 15:22:03,676 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,676 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 15:22:03,677 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 15:22:03,677 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000001
2016-01-14 15:22:03,678 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,678 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452784923423_0001_1_02 for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] numStartedSources: 1 numSources: 1
2016-01-14 15:22:03,678 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] with distanceFromRoot: 3
2016-01-14 15:22:03,681 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784923423_0001_00_000001
2016-01-14 15:22:03,681 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - DAGAppMaster metrics system already initialized!
2016-01-14 15:22:03,682 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000001, launchTime=1452784923682
2016-01-14 15:22:03,682 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: FACEDF332B814CD28C9FB990F959C4C3 in vertex: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,682 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,683 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C55DC0153409441F8FC29EAB74C3F340 with 1 source tasks and 1 pending tasks
2016-01-14 15:22:03,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452784923423_0001_1_03, startRequestedTime=1452784923682, startedTime=1452784923682
2016-01-14 15:22:03,683 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 15:22:03,684 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from INITED to RUNNING due to event V_START
2016-01-14 15:22:03,684 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,693 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_00_000000_0] to container: [container_1452784923423_0001_00_000001]
2016-01-14 15:22:03,694 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,694 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000001 given task: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,694 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,694 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_00_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,695 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,695 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,695 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,695 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, startTime=1452784923694, containerId=container_1452784923423_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000001/teamcity, completedLogs=
2016-01-14 15:22:03,695 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,695 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,695 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,696 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,696 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,696 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,696 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,697 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: CDC852A364E64A83B81E0ED2B2C93103, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,697 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,698 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,699 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 15:22:03,702 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,702 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,703 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=333A4BC1528E493D9A03C012238A7AEC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 15:22:03,705 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,703 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,708 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,708 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,708 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,710 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,712 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 15:22:03,712 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,714 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, ordinal: 0
2016-01-14 15:22:03,714 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, mem on start (mb), free: 63, total: 580, max: 672
2016-01-14 15:22:03,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,715 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,716 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 15:22:03,716 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:03,716 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,717 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,717 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 15:22:03,718 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 333A4BC1528E493D9A03C012238A7AEC, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 15:22:03,718 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B969A35B407643359B20905A4B079125, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 15:22:03,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 333A4BC1528E493D9A03C012238A7AEC being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,720 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,721 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:03,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,722 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,725 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 15:22:03,726 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@7b34c936
2016-01-14 15:22:03,726 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,726 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 333A4BC1528E493D9A03C012238A7AEC
2016-01-14 15:22:03,727 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,727 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,727 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@7b34c936
2016-01-14 15:22:03,727 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 15:22:03,727 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 15:22:03,728 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@699dad28; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6ed35343; comparator=cascading.tuple.hadoop.util.TupleComparator@73ff8c8c; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,728 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,728 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,729 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,753 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 15:22:03,754 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 15:22:03,754 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 15:22:03,754 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 15:22:03,754 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 15:22:03,755 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 15:22:03,755 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 15:22:03,756 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,760 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,760 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDC852A364E64A83B81E0ED2B2C93103, mem on close (mb), free: 160, total: 580, max: 672
2016-01-14 15:22:03,760 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,761 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,761 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 15:22:03,761 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 15:22:03,764 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,768 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 15:22:03,769 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:03,769 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,769 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_00, eventCount=1
2016-01-14 15:22:03,769 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,770 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,770 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskAttemptId=attempt_1452784923423_0001_1_00_000000_0, startTime=1452784923694, finishTime=1452784923770, timeTaken=76, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:03,770 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,770 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_00_000000_0
2016-01-14 15:22:03,770 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,771 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, taskId=task_1452784923423_0001_1_00_000000, startTime=1452784923694, finishTime=1452784923770, timeTaken=76, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 15:22:03,771 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,771 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] : 1
2016-01-14 15:22:03,772 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:03,773 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:03,773 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=CDC852A364E64A83B81E0ED2B2C93103, vertexId=vertex_1452784923423_0001_1_00, initRequestedTime=1452784923627, initedTime=1452784923665, startRequestedTime=1452784923666, startedTime=1452784923666, finishTime=1452784923772, timeTaken=106, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923756, Process_Duration=4, Process_End_Time=1452784923760, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452784923694, firstTasksToStart=[ task_1452784923423_0001_1_00_000000 ], lastTaskFinishTime=1452784923770, lastTasksToFinish=[ task_1452784923423_0001_1_00_000000 ], minTaskDuration=76, maxTaskDuration=76, avgTaskDuration=76.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_00_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:03,774 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:03,774 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:03,775 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] attempt: attempt_1452784923423_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:03,776 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,776 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000001, stoppedTime=1452784923776, exitStatus=0
2016-01-14 15:22:03,777 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:03,776 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:03,778 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:03,779 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,779 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=B969A35B407643359B20905A4B079125, taskId=task_1452784923423_0001_1_01_000000, scheduledTime=1452784923778, launchTime=1452784923778
2016-01-14 15:22:03,779 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,780 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 15:22:03,780 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,780 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:03,780 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 15:22:03,781 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:03,781 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: B969A35B407643359B20905A4B079125, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,781 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000001, Exiting
2016-01-14 15:22:03,781 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,781 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000001
2016-01-14 15:22:03,781 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,783 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,783 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:03,784 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000001 completed successfully
2016-01-14 15:22:03,784 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000002
2016-01-14 15:22:03,784 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,784 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:03,786 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784923423_0001_00_000002
2016-01-14 15:22:03,787 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:03,789 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000002, launchTime=1452784923789
2016-01-14 15:22:03,790 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,789 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,794 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_01_000000_0] to container: [container_1452784923423_0001_00_000002]
2016-01-14 15:22:03,794 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,796 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000002 given task: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,796 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,796 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_01_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B969A35B407643359B20905A4B079125, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, startTime=1452784923796, containerId=container_1452784923423_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000002/teamcity, completedLogs=
2016-01-14 15:22:03,797 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,797 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,797 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,797 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,797 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,797 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,798 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,798 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,798 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,798 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: B969A35B407643359B20905A4B079125, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,798 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,800 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 15:22:03,800 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=CDC852A364E64A83B81E0ED2B2C93103, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:03,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,805 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,805 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,806 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,811 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:03,811 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:03,811 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,813 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B969A35B407643359B20905A4B079125, ordinal: 1
2016-01-14 15:22:03,814 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B969A35B407643359B20905A4B079125, mem on start (mb), free: 153, total: 580, max: 672
2016-01-14 15:22:03,814 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,814 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,814 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,814 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,814 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,815 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,815 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,815 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:03,815 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,815 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:03,816 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, FACEDF332B814CD28C9FB990F959C4C3, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:03,816 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, CDC852A364E64A83B81E0ED2B2C93103, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:03,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: CDC852A364E64A83B81E0ED2B2C93103 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,823 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,823 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:03,824 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: CDC852A364E64A83B81E0ED2B2C93103 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:03,824 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:03,824 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:03,824 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:03,826 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,827 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,828 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,830 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_00_000000_0_10038, runDuration: 60176, ]
2016-01-14 15:22:03,831 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 15:22:03,832 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: D4ECE2BAEFAB4B4F80B2DA6C6AC43097
2016-01-14 15:22:03,832 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,832 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_00_000000_0_10038] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452784923832, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 15:22:03,833 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,833 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,833 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:03,834 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@5955509e; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@701fe2e3; comparator=cascading.tuple.hadoop.util.TupleComparator@3d68de97; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,834 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,834 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,835 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,836 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [CDC852A364E64A83B81E0ED2B2C93103] #10 in 6ms
2016-01-14 15:22:03,837 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [CDC852A364E64A83B81E0ED2B2C93103] #10, status:false, isInterrupted:false
2016-01-14 15:22:03,844 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:03,844 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:03,844 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:03,845 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:03,845 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:03,845 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:03,844 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 15:22:03,846 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,846 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:03,846 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 15:22:03,847 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] D4ECE2BAEFAB4B4F80B2DA6C6AC43097, for 1 inputs
2016-01-14 15:22:03,847 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,847 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:03,847 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:03,848 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 15:22:03,848 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B969A35B407643359B20905A4B079125, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,849 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1f277ec9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@59f4ce19
2016-01-14 15:22:03,870 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,870 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B969A35B407643359B20905A4B079125, mem on close (mb), free: 87, total: 580, max: 672
2016-01-14 15:22:03,871 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: CDC852A364E64A83B81E0ED2B2C93103
2016-01-14 15:22:03,871 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:03,871 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:03,871 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,871 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:03,872 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:03,872 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,872 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,872 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 15:22:03,873 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 15:22:03,882 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,885 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,886 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_01, eventCount=1
2016-01-14 15:22:03,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4]]
2016-01-14 15:22:03,886 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, taskAttemptId=attempt_1452784923423_0001_1_01_000000_0, startTime=1452784923796, finishTime=1452784923886, timeTaken=90, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4
2016-01-14 15:22:03,887 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,887 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_01_000000_0
2016-01-14 15:22:03,887 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,888 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, taskId=task_1452784923423_0001_1_01_000000, startTime=1452784923796, finishTime=1452784923887, timeTaken=91, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4
2016-01-14 15:22:03,888 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,889 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:03,889 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,889 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] : 1
2016-01-14 15:22:03,889 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:03,890 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,890 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:03,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=B969A35B407643359B20905A4B079125, vertexId=vertex_1452784923423_0001_1_01, initRequestedTime=1452784923629, initedTime=1452784923635, startRequestedTime=1452784923670, startedTime=1452784923670, finishTime=1452784923890, timeTaken=220, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=608174080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923848, Process_Duration=22, Process_End_Time=1452784923870, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=4, vertexStats=firstTaskStartTime=1452784923796, firstTasksToStart=[ task_1452784923423_0001_1_01_000000 ], lastTaskFinishTime=1452784923886, lastTasksToFinish=[ task_1452784923423_0001_1_01_000000 ], minTaskDuration=90, maxTaskDuration=90, avgTaskDuration=90.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_01_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:03,891 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:03,891 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:03,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000002, stoppedTime=1452784923891, exitStatus=0
2016-01-14 15:22:03,891 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:03,891 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] attempt: attempt_1452784923423_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:03,892 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:03,892 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:03,892 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,892 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:03,892 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 15:22:03,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskId=task_1452784923423_0001_1_02_000000, scheduledTime=1452784923892, launchTime=1452784923892
2016-01-14 15:22:03,892 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:03,893 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:03,895 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:03,895 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:03,896 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: FACEDF332B814CD28C9FB990F959C4C3, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,896 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:03,897 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:03,897 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000003
2016-01-14 15:22:03,897 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:03,899 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784923423_0001_00_000003
2016-01-14 15:22:03,899 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(144)) - TezTask metrics system already initialized!
2016-01-14 15:22:03,900 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:03,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000003, launchTime=1452784923900
2016-01-14 15:22:03,900 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:03,900 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 15:22:03,900 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:03,901 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000002, Exiting
2016-01-14 15:22:03,901 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,901 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000002
2016-01-14 15:22:03,901 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000003
2016-01-14 15:22:03,905 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:03,907 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000002 completed successfully
2016-01-14 15:22:03,907 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:03,907 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_02_000000_0] to container: [container_1452784923423_0001_00_000003]
2016-01-14 15:22:03,908 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:03,908 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000003 given task: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,908 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03 Starting to run new task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,909 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_02_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 15:22:03,909 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, startTime=1452784923909, containerId=container_1452784923423_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000003/teamcity, completedLogs=
2016-01-14 15:22:03,911 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,911 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:03,911 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:03,911 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:03,911 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:03,911 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:03,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: FACEDF332B814CD28C9FB990F959C4C3, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 15:22:03,910 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:03,912 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:03,911 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:03,912 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:03,913 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:03,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B969A35B407643359B20905A4B079125, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:03,915 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 15:22:03,915 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:03,916 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,916 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C55DC0153409441F8FC29EAB74C3F340, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 15:22:03,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:03,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,917 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:03,918 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:03,919 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:03,919 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,919 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 15:22:03,920 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,920 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,920 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:03,929 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, ordinal: 2
2016-01-14 15:22:03,929 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, mem on start (mb), free: 82, total: 580, max: 672
2016-01-14 15:22:03,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:03,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:03,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:03,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:03,930 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:03,930 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:03,930 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 15:22:03,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:03,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 15:22:03,931 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B969A35B407643359B20905A4B079125, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 15:22:03,931 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C55DC0153409441F8FC29EAB74C3F340, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 15:22:03,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:03,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B969A35B407643359B20905A4B079125 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:03,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:03,933 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:03,933 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,934 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:03,934 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B969A35B407643359B20905A4B079125 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:03,934 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 15:22:03,934 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:03,935 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:03,937 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:03,938 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,939 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_01_000000_0_10042, runDuration: 80035, ]
2016-01-14 15:22:03,941 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 15:22:03,942 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_01_000000_0_10042] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452784923942, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 15:22:03,942 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,943 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 15:22:03,942 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 2797C146692A4F6D93EB14D62DF5DF9E
2016-01-14 15:22:03,943 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [B969A35B407643359B20905A4B079125] #11 in 2ms
2016-01-14 15:22:03,943 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:03,944 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:03,944 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@46cb23b0; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@42941c1f; comparator=cascading.tuple.hadoop.util.TupleComparator@9c2595f; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 15:22:03,944 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 15:22:03,944 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 15:22:03,945 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:03,954 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B969A35B407643359B20905A4B079125] #11, status:false, isInterrupted:false
2016-01-14 15:22:03,961 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 15:22:03,961 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 15:22:03,961 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 15:22:03,962 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 15:22:03,962 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 15:22:03,962 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:03,963 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 15:22:03,963 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,963 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:03,963 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2797C146692A4F6D93EB14D62DF5DF9E, for 1 inputs
2016-01-14 15:22:03,964 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 15:22:03,964 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,964 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:03,964 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, all 1 inputs ready in: 00:00:00.000
2016-01-14 15:22:03,965 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@213f379; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6bd58dff
2016-01-14 15:22:03,988 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,988 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FACEDF332B814CD28C9FB990F959C4C3, mem on close (mb), free: 183, total: 564, max: 672
2016-01-14 15:22:03,988 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B969A35B407643359B20905A4B079125
2016-01-14 15:22:03,988 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:03,989 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:03,989 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:03,989 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:03,989 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:03,989 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 15:22:03,989 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 15:22:03,990 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 15:22:03,990 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 15:22:03,993 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 15:22:03 Completed running task attempt: attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,997 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:03,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452784923423_0001_1_02, eventCount=1
2016-01-14 15:22:03,997 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 15:22:03,998 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:03,998 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:03,998 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:03,998 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskAttemptId=attempt_1452784923423_0001_1_02_000000_0, startTime=1452784923909, finishTime=1452784923998, timeTaken=89, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:03,998 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000003
2016-01-14 15:22:03,998 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:03,999 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_02_000000_0
2016-01-14 15:22:03,999 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:03,999 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:03,999 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, taskId=task_1452784923423_0001_1_02_000000, startTime=1452784923909, finishTime=1452784923999, timeTaken=90, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 15:22:03,999 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:03,999 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] : 1
2016-01-14 15:22:04,000 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:04,000 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:04,000 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=FACEDF332B814CD28C9FB990F959C4C3, vertexId=vertex_1452784923423_0001_1_02, initRequestedTime=1452784923636, initedTime=1452784923644, startRequestedTime=1452784923674, startedTime=1452784923674, finishTime=1452784924000, timeTaken=326, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5679, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=7, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=30, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784923964, Process_Duration=24, Process_End_Time=1452784923988, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452784923909, firstTasksToStart=[ task_1452784923423_0001_1_02_000000 ], lastTaskFinishTime=1452784923998, lastTasksToFinish=[ task_1452784923423_0001_1_02_000000 ], minTaskDuration=89, maxTaskDuration=89, avgTaskDuration=89.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_02_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:04,001 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:04,001 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:04,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000003, stoppedTime=1452784924001, exitStatus=0
2016-01-14 15:22:04,001 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:04,001 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:04,001 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] attempt: attempt_1452784923423_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 15:22:04,002 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C55DC0153409441F8FC29EAB74C3F340
2016-01-14 15:22:04,002 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,002 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:04,002 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452784923423_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 15:22:04,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_STARTED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskId=task_1452784923423_0001_1_03_000000, scheduledTime=1452784924002, launchTime=1452784924002
2016-01-14 15:22:04,002 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 15:22:04,003 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 15:22:04,004 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,004 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 15:22:04,004 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C55DC0153409441F8FC29EAB74C3F340, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:04,004 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 15:22:04,004 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 15:22:04,006 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452784923423_0001_00_000004
2016-01-14 15:22:04,006 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 15:22:04,007 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452784923423_0001_00_000004
2016-01-14 15:22:04,008 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 15:22:04,009 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 15:22:04,009 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:04,009 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000003, Exiting
2016-01-14 15:22:04,009 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000003
2016-01-14 15:22:04,009 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000003
2016-01-14 15:22:04,011 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:04,011 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452784923423_0001_00_000004, launchTime=1452784924011
2016-01-14 15:22:04,011 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 15:22:04,011 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000003 completed successfully
2016-01-14 15:22:04,011 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 15:22:04,011 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 15:22:04,011 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,012 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452784923423_0001_1_03_000000_0] to container: [container_1452784923423_0001_00_000004]
2016-01-14 15:22:04,012 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 15:22:04,013 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452784923423_0001_1_03_000000_0] started. Is using containerId: [container_1452784923423_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 15:22:04,013 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, startTime=1452784924013, containerId=container_1452784923423_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452784923423_0001_00_000004/teamcity, completedLogs=
2016-01-14 15:22:04,013 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 15:22:04,013 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 15:22:04,013 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 15:22:04,014 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452784923423_0001_00_000004 given task: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04 Starting to run new task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,014 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,014 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 15:22:04,014 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 15:22:04,015 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 15:22:04,015 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 15:22:04,015 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 15:22:04,015 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C55DC0153409441F8FC29EAB74C3F340, VertexParallelism: 1, TaskAttemptID:attempt_1452784923423_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 15:22:04,015 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 15:22:04,016 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,016 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 15:22:04,017 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,017 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452784923423_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 15:22:04,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=FACEDF332B814CD28C9FB990F959C4C3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 15:22:04,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 15:22:04,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,021 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 15:22:04,021 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 15:22:04,021 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,021 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,022 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 15:22:04,023 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=66B5A676F67342598BB0CC462F2FC64C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 15:22:04,023 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 15:22:04,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,024 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 15:22:04,024 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,034 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, ordinal: 3
2016-01-14 15:22:04,035 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, mem on start (mb), free: 177, total: 564, max: 672
2016-01-14 15:22:04,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 15:22:04,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 15:22:04,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 15:22:04,039 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 15:22:04,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 15:22:04,039 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 15:22:04,040 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 15:22:04,040 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 15:22:04,040 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 15:22:04,041 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 15:22:04,041 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 15:22:04,041 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, FACEDF332B814CD28C9FB990F959C4C3, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 15:22:04,041 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 66B5A676F67342598BB0CC462F2FC64C, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 15:22:04,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 15:22:04,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: FACEDF332B814CD28C9FB990F959C4C3 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 15:22:04,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 15:22:04,043 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 15:22:04,043 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 15:22:04,044 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 15:22:04,045 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: FACEDF332B814CD28C9FB990F959C4C3 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 15:22:04,045 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 15:22:04,046 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 15:22:04,046 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 15:22:04,047 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 15:22:04,049 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,050 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452784923423_0001_1_02_000000_0_10045, runDuration: 77934, ]
2016-01-14 15:22:04,052 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 15:22:04,052 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452784923423_0001_1_02_000000_0_10045] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452784924052, TimeTaken=1, Rate=4.89 MB/s
2016-01-14 15:22:04,052 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,053 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: A3A4232218854981AC93E364A39014EB
2016-01-14 15:22:04,053 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: 66B5A676F67342598BB0CC462F2FC64C
2016-01-14 15:22:04,054 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 15:22:04,054 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] A3A4232218854981AC93E364A39014EB, for 1 inputs
2016-01-14 15:22:04,053 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 15:22:04,054 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [FACEDF332B814CD28C9FB990F959C4C3] #12 in 3ms
2016-01-14 15:22:04,054 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [FACEDF332B814CD28C9FB990F959C4C3] #12, status:false, isInterrupted:false
2016-01-14 15:22:04,056 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 15:22:04,057 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 15:22:04,057 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:04,057 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 15:22:04,057 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 15:22:04,057 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,057 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 15:22:04,058 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, all 1 inputs ready in: 00:00:00.004
2016-01-14 15:22:04,058 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@69fc0ce9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@22b3cb5b
2016-01-14 15:22:04,067 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 15:22:04,067 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 15:22:04,068 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,068 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452784923423_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 15:22:04,069 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,069 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C55DC0153409441F8FC29EAB74C3F340, mem on close (mb), free: 174, total: 564, max: 672
2016-01-14 15:22:04,069 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: FACEDF332B814CD28C9FB990F959C4C3
2016-01-14 15:22:04,070 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 15:22:04,070 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 15:22:04,070 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 15:22:04,070 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 15:22:04,070 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 15:22:04 Completed running task attempt: attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,072 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 15:22:04,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskAttemptId=attempt_1452784923423_0001_1_03_000000_0, startTime=1452784924013, finishTime=1452784924073, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:04,073 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452784923423_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 15:22:04,074 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452784923423_0001_1_03_000000_0
2016-01-14 15:22:04,074 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 15:22:04,074 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:TASK_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, taskId=task_1452784923423_0001_1_03_000000, startTime=1452784924013, finishTime=1452784924074, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1452784923423_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 15:22:04,075 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 15:22:04,075 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452784923423_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 15:22:04,074 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 15:22:04,075 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] : 1
2016-01-14 15:22:04,076 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:VERTEX_FINISHED]: vertexName=C55DC0153409441F8FC29EAB74C3F340, vertexId=vertex_1452784923423_0001_1_03, initRequestedTime=1452784923645, initedTime=1452784923659, startRequestedTime=1452784923682, startedTime=1452784923682, finishTime=1452784924076, timeTaken=394, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452784924013, firstTasksToStart=[ task_1452784923423_0001_1_03_000000 ], lastTaskFinishTime=1452784924073, lastTasksToFinish=[ task_1452784923423_0001_1_03_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452784923423_0001_1_03_000000 ], longestDurationTasks=[ task_1452784923423_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 15:22:04,077 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 15:22:04,078 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 15:22:04,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452784923423_0001_00_000004, stoppedTime=1452784924078, exitStatus=0
2016-01-14 15:22:04,078 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 15:22:04,078 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 15:22:04,078 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452784923423_0001_1
2016-01-14 15:22:04,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452784923423_0001_1
2016-01-14 15:22:04,079 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_00 [CDC852A364E64A83B81E0ED2B2C93103]
2016-01-14 15:22:04,079 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_01 [B969A35B407643359B20905A4B079125]
2016-01-14 15:22:04,079 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 66B5A676F67342598BB0CC462F2FC64C for vertex: vertex_1452784923423_0001_1_03 [C55DC0153409441F8FC29EAB74C3F340]
2016-01-14 15:22:04,078 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=591921152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452784924058, Process_Duration=11, Process_End_Time=1452784924069, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 15:22:04,079 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 15:22:04,081 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 15:22:04,082 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 15:22:04,082 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,083 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,083 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452784923423_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 15:22:04,083 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452784923423_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 15:22:04,083 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452784923423_0001_00_000004, Exiting
2016-01-14 15:22:04,083 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,083 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452784923423_0001_00_000004
2016-01-14 15:22:04,084 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452784923423_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 15:22:04,084 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452784923423_0001_00_000004 completed successfully
2016-01-14 15:22:04,092 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452784923423_0001_1_02 [FACEDF332B814CD28C9FB990F959C4C3]
2016-01-14 15:22:04,096 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452784923423_0001_1, queueSize=0
2016-01-14 15:22:04,096 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 15:22:04,097 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452784923423_0001_1][Event:DAG_FINISHED]: dagId=dag_1452784923423_0001_1, startTime=1452784923627, finishTime=1452784924093, timeTaken=466, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=18, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2400190464, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=27, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811139695626, Process_Duration=61, Process_End_Time=5811139695687, Read_Duration=15, Tuples_Read=303, Tuples_Written=317, Write_Duration=7, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 15:22:04,098 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452784923423_0001_1 finished with state: SUCCEEDED
2016-01-14 15:22:04,098 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452784923423_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 15:22:04,098 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452784923423_0001_00_000004
2016-01-14 15:22:04,098 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452784923423_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 15:22:04 Completed Dag: dag_1452784923423_0001_1
2016-01-14 15:22:04,100 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452784923423_0001_1, dagState=SUCCEEDED
2016-01-14 15:22:04,100 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452784923423_0001_1_post
2016-01-14 15:22:04,100 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 15:22:04,101 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 15:22:04,106 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452784923423_0001
2016-01-14 15:22:04,108 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 15:22:04,108 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 15:22:04,108 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 15:22:04,108 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 15:22:04,108 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 15:22:04,109 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 15:22:04,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 15:22:04,110 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 15:22:04,110 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 15:22:04,111 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 15:22:04,111 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 15:22:04,112 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 15:22:04,112 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 15:22:04,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/tmp/.staging/C2B1EC681BD5451298D31460C4F7091A/C18FD8A6A09B4C189ED01DCB4BD557E8/.tez/application_1452784923423_0001
2016-01-14 15:22:04,121 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:00.717
2016-01-14 15:22:04,191 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.4.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short